In between performance and documentation

This essay was originally published November 2022 as a chapter in the book Documentation as Art, edited by Annet Dekker and Gabriella Giannachi.

In the preservation of digital art documentation is filling the gaps in between manifestations of a piece. When an installation is set up in a gallery, net art is migrated from one server to another, or generally, when an artwork finds itself in a technical environment that is different from the one it was made in or made for, documentation plays a vital role in guiding curators, preservation specialists, and exhibition technicians. This essay proposes that specific types of documentation can become part of an artwork’s manifestation, and that, over time, artworks can be moved into a mixed form that is part full performance and part documentation.

Processes described below are based on a definition of digital objects as ensembles of artifacts and environments from the article Digital Objecthood.n This is put into practice at Rhizome, a digital arts non-profit organization founded in 1996, stewarding more than 2000 born-digital artworks in its ArtBase collection. Additionally, this essay elaborates and extends on ideas presented in the paper Fencing Apparently Infinite Objectsn that for purposes of preservation defines networked objects composed of local and remote out-of-reach artifacts as ‘blurry’.

Components of documentation

The classic practice of documenting digital artworks has two main components. First, technical documentation that describes how to maintain, stage, or even recreate a work. If everything goes according to plan, this type of documentation will contain clear, reproducible instructions on how to align the numerous elements typically required for the performance of a digital artwork. Second, narrative documentation that aims to capture the aspects of the work that are beyond its artefactual existence. This can take the form of artist interviews, art historical classification, videos of installations of the artwork, screenshots, press clippings, etc. The narrative documentation might influence what the technical documentation focuses on and help produce a list of ‘significant properties’ that are meant to receive special attention when a work gets preservation treatment. However, the two documentation components differ fundamentally in that the technical one is mostly concerned with a snapshot state of the work, basically describing and drawing from one occurrence of bringing it from a switched-off to a switched-on state, from artifact to performance. The narrative documentation considers the artwork regardless of its current performative state and can cover multiple instantiations and overarching conceptual aspects. Given the potential variability of digital artworks, the two documentation components can often not be clearly separated, which in turn can lead to confusion about what exactly is meant by preservation, restoration, and documentation, in relation or contrast to the artwork.

When approached consciously, a hybrid of technical and narrative documentation can provide invaluable materials for the preservation of software in its performative state – especially for objects that are, for preservation purposes, defined as having a ‘blurry boundary’, or being completely ‘boundless’. Blurry objects make use of remote resources that can, due to their design, function, or variability and vastness, not be acquired, maintained, or even owned, neither by creators of the artwork nor collecting institutions, and sometimes not by anybody really. Typical examples for blurriness are an artwork’s reliance on accessing products like Google Image Search, a social media feed like Twitter, or in general platforms that allow access to live data, such as stock market tickers or weather information. For instance, while the part of a work that interfaces with such resources has a ‘local’, artefactual representation as performable software, it is impossible to lay hands on Google Image Search and bring it into an artefactual state from which its performance can be reproduced. Instead, the behaviour of Google Image Search in the context of a particular artwork can only be observed and described. Boundless objects are completely located within such remote resources, for example, social media performances or other interventions on platforms that are not under the creator’s control.n

To understand how practices that can be considered documentation are able to support the preservation of performance, it makes sense to distinguish between the performance of objecthood and computer performance. During its execution software produces ‘inner states’ in computer (or emulator) components: bits in memory, storage devices, the CPU, etc., are set or unset. These inner states could potentially be intercepted, imaged, and examined with every tick of the system clock. The changes that could be observed from one tick to next are an expression of computer performance, but are not, on their own, necessarily legible as developments of or interactions between objects, or as objects at all. Only where computer performance touches the ‘outside world’ via input and output devices can a performance of objecthood happen, and hence, legible documentation be created. Forms of documentation that are located in between ‘physical imaging’ and ‘legible’ can be used in the re-enactment of a digital object, shifting the object itself to a manifestation in between full performance and documentation.

For instance, the mouse pointer is the performance of a legible digital object that can be directly manipulated: pixels on a screen are configured to look like an arrow that changes its position according to signals from an input device like a touchpad. The pointer’s computer performance is tightly coupled with its performance of objecthood, because through the pointer’s consistent and unique ability to visibly be moved to every single pixel of a screen with varying velocities, it can be recognised and separated from other things happening in a computer system. As a general means for interaction, the pointer can be applied for all kinds of purposes. For the sake of the argument, the context of navigating the web will be examined, which is pretty simple compared with for instance the role of the mouse in playing a flight simulator. In this web navigation setting, it would not make sense to document every possible inner state and every possible position of the mouse pointer on screen; there are simply too many, and too complex relations that produce changes in between states: the pointer can move from one pixel to another via an infinite number of paths. This is not necessarily the case for a web page the mouse pointer acts on: while it is performed as a ‘page’ – it is shown as visually bound width ‘edges’, displaying certain items (text, images, etc.), and reacting to high level user inputs like mouse events and key presses in a defined manner – many technically independent computer systems, each with their own inner states, are potentially involved in creating such a legible presentation. Each of these computer systems can influence how a web page’s objecthood is performed and perceived, possibly depending on many more factors than in the case of the mouse pointer. Still, documenting the occurrence of a web page loading makes sense as a semantic unit that represents a desirable, legible result of computer performance – quite unlike the infinite and overall homologous inner states of the mouse pointer.

The classic process for documenting the result of the browser reaching out of its computer system requesting remote resources, and from these resources compositing the visible graphical representation would be taking a screenshot, which is easily legible, yet not computable.n In an actual re-performance of the rendering process, a screenshot cannot play any role. A higher level of abstraction is more productive for that purpose: a protocol of the browser requesting and receiving remote resources – in other words, a web archive. Finally, a disk image of the computer system running the browser can be launched in an emulator and connected with a web archive that was specifically created to contain all resources required by the web page.n This ensemble has the capability to reach all possible inner states in any possible order when it comes to the mouse pointer, as its performance is supported by a complete, fully operating computer system. Using the pointer, links on a web page might be activated in a different order in each re-enactment session, but the resources requested by the browser will always be drawn from the same pool contained in a static, finite web archive.

Grading the performability of documentation

How exactly do web archives differ from the live web in terms of performance? As outlined earlier, requesting and receiving remote web resources always involves computing activity of at minimum two computer systems. A web archive abstracts away the computer at the other end of the network and reduces its behaviour to basic matching of requests and responses: if a current request for a web resource is similar enough to a request that was previously observed, the previously observed response will be returned. This is quite similar to other types of documentation that describe under what conditions or at what points in time something is supposed to happen or has happened. Obviously, even rather primitive variability, like a website showing different random elements on each access, or changing the background colour depending on time of day, cannot be expressed with that simple matching model. But this difference to full performance will not be relevant in a wide range of other cases.

If a remote web server exposes a fixed number of resources that are returned without regard to any input apart from a finite number of requests defined as meaningful, that server can be abstracted into a web archive in a lossless manner, effectively becoming identical with its documentation. Of course, this can be true for what is considered an artifact representing the core of an artwork (such as an artist-created website), or supporting artifacts (for instance, general web resources that provide material to be processed by software created by an artist). That a web server accepts only a finite number of request variations is key here: a fixed set of discrete affordances like links, checkboxes, radio buttons, or sliders that, for instance, provide bound views into a database, or a clearly defined continuous input like a map that can be panned and zoomed, resulting in set of finite x-y-z coordinates, offer a potentially vast but still countable amount of input variations.

A free-form text input field submitted to the server, or a random generator executed on the server typically introduce a level of variability, of infinite possible states, that cannot be represented as a web archive.n The preconditions for a web archive acting as a complete stand-in for a full web server performance can appear very limiting when not considering that object-specific web archives that are supposed to support a specific artwork are quite different from mainstream web archives like the Internet Archive’s Wayback Machine that is supposed to capture general, broad snapshots of ‘the whole web’. An object-specific web archive can be constrained so that in effect the preconditions are met, for instance by only taking into account resources revealed after a login process is completed on a website.

If an artwork is processing live web resources, replacing them with web archives obviously changes the artwork’s performance. The more performing artifacts are replaced with static, documentation-like artifacts, the more an artwork moves towards the documentation end of the performance-documentation spectrum. As long as fully performing artifacts are still part of the ensemble, the artwork becomes a hybrid of itself and its documentation. This can be a reasonable strategy for certain preservation, re-enactment, and staging challenges.

Four hybrids

The following examples from my own practice as Rhizome’s preservation director and as a freelance digital art preservation consultant illustrate how digital artworks can be prepared for or brought into a state of mixed performance and documentation, for purposes of stabilisation, preservation, or making current and future exhibitions more manageable. Emulation work was done using Emulation as a Service (EaaS), a preservation framework that manages and orchestrates emulators, containers, and disk images. Web archives were handled using tools from Webrecorder, an open source project that produces software components and end-user tools for advanced web archiving scenarios.n Custom artwork installation settings were achieved with scripted Linux systems.

Stabilising web resources

For the 2019 exhibition The Art Happens Here: Net Art’s Archival Poetics at the New Museum in New York, skinonskinonskin, a web artwork created in 1999 by the artist duo Entropy8Zuper! was presented in a contemporaneous Windows 98 software environment equipped with legacy versions of the Netscape browser as well as Flash, Java, and Director plugins.n The artwork consists of a series of grandiosely designed web pages which the artists gifted each other online and originally had made available behind a paywall. Users are presented with lively and dynamic visuals and sound, in many cases interactive, but the web resources the artwork is performed with have not changed for more than two decades. Making them available to the legacy software environment via a web archive included in the emulated network environment removed the considerable burden of setting up and maintaining a web server or even just an Internet connection. The original paywall access mechanism would have required a fully performing server to execute some authentication logic. Since that was not desired the web archive could be used without any change to the artwork’s performance.

Public search services

The work onewordmovie by Beat Brogle and Philippe Zimmermann, originally created in 2003, outputs animation sequences that are only a few seconds long, with every frame being picked from Google’s Image Search service based on querying a single word users are asked to input (Figs. 14.4 and 14.5). When the artwork was accessioned by Haus der elektronischen Künste Basel (HEK) in 2016,n the preservation risk was evident: in the history of the artwork, the artists had to continuously adapt how the core software communicates with Google’s service, which regularly introduces changes to technical interfaces, usage policy, pricing, and rate limits. Of course, Google could discontinue the image search service completely, or change it so much that it would become impossible to use for the purpose of the artwork; the same applies to competing services such as Bing.

The preservation plan devised for the work was to expand the component that interfaces with Google so it stores the image search results for every user request in a database, including actual image data, image URLs, and timestamps. When no user is active on onewordmovie, the image search is queried automatically with user input the artists recorded during the history of the piece, and with regular dictionary words. Over time, this creates a body of data about images being matched with words at specific points in time. When interfacing with Google is impossible, this database can be queried instead. As soon as the switch from Google Image Search to the database is made, the artwork becomes a hybrid of performance and documentation: the video clips keep being assembled from user submitted words, but, with growing temporal distance to the last database records, less and less current imagery will appear. For some user inputs no results will be available. With Google Image search becoming inaccessible, representing the loss of a major dependency outside of the control of the artists or the collecting institution, onewordmovie will be historicised, but it will never be reduced to pure documentation.

Additionally, the collected data set will be usable by other artworks that rely on Google Image Search in a similar way.

Public news resources

For Hans Haacke’s solo exhibition All Connected, presented 2019/2020 at the New Museum in New York, the artwork News had to be installed. In its first iteration dating back to 1969, a telex printer machine was connected to news agency wires and filled the gallery space with news printouts. With the discontinuation of wire news, the work was redefined in 2008 to pull real-time data from RSS feeds offered on the web by news agencies and daily newspapers and to use dot matrix printers for output. In 2019, only very few websites still provided information in RSS format; the ones that did had stopped publishing full articles and reduced feed content to summaries. The challenge of web publishers to monetise their products has had the field abandon standardised formats and focus on proprietary websites, custom apps, and paywalls. In general, it cannot be assumed that a broad range of news outlets will keep making articles available in a text-only format that is comfortably processed and assembled as required for News. Faced with this situation, a new software was developed that scrapes news headlines and articles directly from publishers’ websites and stores the results in a database. Additionally, all interactions with the outside web are captured in a web archive. While performing on exhibition, News documented itself, in a form that has the potential to be used in a historicised re-enactment of the piece.

Distributed hosting

Olia Lialina’s 2020 net art work Hosted shows a looped animation of the artist swimming, with each of the 70 frames hosted on a different free web service, such as image sharing sites, social media, dating platforms, etc. The structure of the work is intentionally fragile since these free services frequently remove uploaded material for a variety of reasons, change addresses of resources, or require users to regularly perform certain activities to signal that their free account is still active. The artist has to constantly fix the piece and users on the web have to expect different frames from the loop missing at times.

For the 2020 solo exhibitions Best Effort Network at arebyte gallery and Something for Everyone at Espace Multimedia Gantner, a gallery version of the work was produced that on start-up presents a choice in between using live web resources or a web archive representing the last state in which all animation frames were intact. If one of the 70 hosting services is posing issues, curators have the option of showing a backup version of the work while the artist is fixing the piece, or present the live version with missing frames. Since the web archive is running on the same computer the artwork is installed on, it can also serve as a backup for general Internet outage. Again, once free web-hosting services stop existing in general, the web archival snapshots of the artwork’s remote resources will allow for a mixed performance/documentation staging.

Narrowing affordances

The examples presented earlier are dealing with artworks that have blurry boundaries. Making some parts of these works behave like documentation – that is: reducing possible inputs and inner states towards a basic, descriptive one-to-one relation of condition and result – perhaps paradoxically, is a reasonable strategy for stabilising and preserving their performance. To make such preservation actions productive, documentation has to be regarded not as creating a version of an artwork that has each and every performative aspect removed – by producing photos, screenshots, video documentation, or description – but as a careful, gradual restriction of the inner states a computer system or ensemble thereof can be brought into, a reasonable narrowing of available affordances, and replacing aspects of performance that are difficult to manage with more ‘static’ artifacts. The similarity in all discussed cases and examples is that these documentation resources are created by digitally recording observable behaviour in a structured format that can be computationally acted on to integrate it with fully performing parts of an artwork, and that provides enough variability to uphold a sense of the original performance.

Similar techniques have been applied to other forms of digital art that are ‘blurry’ in different ways. For instance, pieces engaging with what is currently defined as virtual reality are often created for fragile devices that are produced by highly competitive corporations with little interest in interoperability and hence involve quick market turnovers with products and software frameworks frequently being replaced with newer versions. Maintaining such a stack of hardware and software beyond its official support time is an enormous undertaking. However, if a virtual reality artwork does not allow for users to manipulate the simulated environment and restricts movement to changing the camera rotation,n documentation in the form of a best-possible resolution linear 360° video would be able to reproduce all the states of the work on future VR devices. Pixel resolution mismatches, colour inaccuracies, and other effects will have to be expected, but might be a reasonable trade-off.

 

From screen essentialism to network traffic essentialism?

It is easy to connect the hybrid performance/documentation approach with ‘screen essentialism’,n a term, coined by Nick Montfort to criticise the reduction of complex computational processes and digital materiality to what is the visual end product displayed on screens. And indeed, the preservation of digital art is often discussed in terms of time-based media. In the ensemble of an artwork, replacing a computational process with documentation of this process is similarly just recording of what is observable, maybe not by a human but by a specialised tool.

Some artists won’t agree to an animation that is computed in real-time yet produces the same linear, time-bound result on each run being rendered as a digital video because they define computation as the focus of the work.n Some artists will refuse to work with object-specific web archives because they define the real-time transmission of network resources or the materiality of network communication as significant for their work. Both positions are absolutely valid. However, in many cases, designing preservation projects around such ‘significant properties’ will lead to screen essentialism much quicker, and through a more painful process: insisting on conditions that are impossible to meet in the future – for instance, receiving data from discontinued commercial services – will cause preservation efforts to fail, artworks declared ‘irreparable’ well before their time, and hence result in full-on documentation – photos, screenshots, video captures, descriptions – as the only future leftovers of any even slightly media-specific or ‘complex’ (a.k.a. performative) digital artwork: ‘Documentation is often all that remains after a work has been shown and experienced’ (LIMA).n Instead, regarding documentation and performance not as absolutes but end points on a scale should lead to thinking about ‘reproducible properties’: every possible inner state of an object needs to be supported by an artifact.n Such an artifact can provide for full performance, as, for instance, a disk image of a web server; or support a reduced subset of the original performance via documentation in a machine-readable and actionable format, as, for instance, a web archive. Using the latter will reduce the amount of possible inner states for the overall ensemble. In many cases, like when the number of meaningful interactions is finite, a reduction of possible inner states is a benefit – for instance, if registering only desired interactions and leaving out errors, the preservation management of a performative system can be radically simplified. In cases where an infinite amount of inner states were supported in the original setting, there is a trade-off when using this reduced artifact: preservation is still simpler, and in case of blurry object boundaries made possible in the first place, but the complete breadth of inner states will not be reproducible. Yet, however noticeably or unnoticeably reduced, the inner states supported by artifacts will remain reproducible in the future.