Tackle Climate Change Through Art: A case study

This paper describes the creative process behind the creation of an audiovisual installation. Titled The Great Acceleration (2021), this installation is built upon scientific data simulating the retreat of the Ross Ice Shelf from 2015 to 2100. The dataset is the result of a computer simulation based on a scientific model provided by researchers from the Laboratory for Sciences of Climate and Environment (LSCE). The paper details the methodology used to offer a sensitive experience of climate change through the analysis of a large amount of data and the creation of generative audiovisual compositions. It describes the artistic approach and how the dataset related to the Ross Ice Shelf activity has been processed in order to create visual compositions and shape sounds. The paper also explains the relationship between these images and sounds and how they react to external outputs such as the presence and movement of visitors within an exhibition space.


INTRODUCTION
This paper describes the creative process behind the creation of an audiovisual installation that has obtained the financial support of the Diagonale Paris-Saclay through its call for projects Expérimentation 2020. Titled The Great Acceleration (2021), this installation is built upon scientific data evoking the uncertain future of Antarctica's melting ice. The selected dataset relates more precisely to the transfiguration of the largest ice shelf in Antarctica: The Ross Ice Shelf from 2015 to 2100. The dataset is the result of a computer simulation based on a scientific model provided by researchers from the LSCE laboratory: a laboratory that studies past present and future climate from the analysis of natural archives and the development of numerical models.
The paper details the methodology used to offer a sensitive experience of climate change through the creation of an interactive installation. This methodology is built upon the analysis of a large amount of data and is used to create generative audiovisual compositions that can be seen at this address: http://vimeo.com/608419914/c27388c024. First, the artistic approach and the processing of the dataset related to the activity of the Ross Ice Shelf is defined. Then, we explain how images and sounds have been created and animated while using this dataset as raw and refined materials. Finally, we discuss the dynamic relationship between these images and sounds and how they react to external outputs such as the presence and movement of visitors within an exhibition space.

The musical work
The musical part of the installation is based on a work of mixed music for cello and electroacoustic (9'16) by Xavier Hautbois composed on the theme of melting ice. In accordance with the slowness of the whole installation, the general atmosphere of the musical piece has a static and timeless aspect, despite the moments of tension carried by the instrument which plays the main role: the cello.
The soundtrack that carries the music is produced by virtual instruments. It is in perpetual and slow development.
Two recognisable melodic instruments emerge from it: the cello, playing the dominant role, and the celesta, staying back, which adorns the song of the cello in an almost concertante writing.
A light granular texture accompanies this ensemble in an autonomous way. This texture is built from natural sounds recalling flowing waters as well as ice cracking and breaking up. Although the sound background presents more or less strong tonal reminiscences, the cello melody is expressed in atonal and contemporary expressions.
Formally, the musical piece is made up of eight well-defined sequences. In each one of them, the cello exposes one or more modes of play: These bowing techniques create musical variations which affect the movements of the particle systems used to depict the evolution of the Ross Ice Shelf activity.

Data selection
The installation The Great Acceleration is built upon a dataset provided by researchers from the LSCE working on past and future interactions between climate and ice sheets.
The dataset is an output of the GRISLI ice sheet model: a model able to reproduce expected grounding line advances during glacial periods and subsequent retreats during terminations with reasonable glacial-interglacial ice volume changes (Quiquet et al. 2018). The dataset represents with the help of a Cartesian coordinate system the possible retreat of the Ross Ice Shelf from 2015 to 2100.
The first step to develop an installation based on this data and the movements it contains was to give it a default representation by creating a program able to display the evolution of the Ross Ice Shelf activity. These first attempts of visualisation were used to verify that the data has been handled properly by being able to retrieve an accurate representation of the Ross Ice Shelf that could be compared to visualisations used by climate scientists.
These first images of the Ross Ice Shelf activity were also used to identify a specific parameter, an iconic variable that is used as the main material to create images and sound related to the Great Acceleration: a concept that depicts the dramatic acceleration in human enterprise and its impacts on the Earth system over the last two centuries (Steffen et al. 2015).
Amongst the many parameters available, ice surface velocity has been selected due to its significant increase over the observed period of time. The increased velocity of glaciers and ice sheets due to climate change is one of the main causes of sea level rise. Ice velocity represents one of the most important parameters researchers used to study ice sheet and glacier dynamics. Rates of ice movement vary from one glacier to another and change with depth. For example, glacial motion gets smaller on small glaciers or in the centre of ice sheets.
In this case only the ice sheet surface velocity has been taken into account. The internal and sliding velocities of the Ross Ice Shelf have not been selected to shape the audiovisual compositions presented in this paper.

Data translation
The data selection process was followed by its treatment. Ice velocities were used to define the outlines of a constantly evolving territory. Ice surface velocities were originally related to a specific position on a Cartesian grid. Together, they draw the movements animating a territory of 16 km by 16 km which represents the Ross Ice Shelf. As such, they have been first used to create grayscale images of ice velocities at the ice shelf surface. Then, these digital images have been processed using blob detection methods in order to identify different regions inside each of them ( Figure 1). This computational process serves a speculative inquiry as any process of speculative computing (Drucker & Nowviskie 2004). It aims to produce a useful aesthetic provocation that is the result of a subjective interpretation enacted by an interpreter and predicting the disappearance of the Ross Ice Shelf.
The regions identified show 5 different levels of intensity related to the ice shelf movement. They allow us to follow the evolution of the Ross Barrier and to observe the acceleration of Antarctic Ice Sheet mass loss over a long period of time. These regions are mainly defined by their curves, surface, position and the level of intensity they are associated with. Their curves represent the main element used to create the datascape of the installation The Great Acceleration.

THE CREATION OF GENERATIVE AUDIOVISUAL COMPOSITIONS
The lists of 2D vectors that define each region are at the core of the images that depict the activity of the Ross Ice Shelf. These images are built using multiple particle systems that redraw the curves of each region over time in a 3D environment ( Figure  3). The third dimension of the 3D space is used to separate the regions between them based on the intensity of the movements they are representing.
With the colour code used to draw these lines, this spatial distancing recreates figurative representations of the Ross Ice Shelf that allows their viewers to observe the retreat of the ice barrier.
The curves of each region are used to modulate the sound composition using IanniX: a graphical open-source sequencer. They become velocity curves modifying in real-time musical parameters. Each curve is assigned to a specific sound source depending on its velocity level. Celesta is assigned to curve level 1, the harmonic background to levels 2 and 3, while the granular texture is assigned to levels 4 and 5. The perimeter of each curve influences the loudness of the sound assigned to it. Therefore, the granular texture becomes more present as the curves of levels 4 and 5 appear and grow stronger over time. The sound generator which controls this progression sets noticeable variations in the balance between voices. In the sequencer IanniX, a playhead slides on every velocity curve ( Figure 2). Every curve is being played in a loop at a constant speed. The shape of these curves acts on band-pass filters in order to give an iridescent and changing colour to the sounds they are associated with. Depending on the angle of rotation of their playhead, the sound quality takes on darker or, on the contrary, brighter tones. Each velocity curve has its own length and shape. This characteristic permits the creation of complex filter overlays over time.
The whole installation tends to point out ecological issues through the lenses of an objective observer. The retreat of the Ross Ice Shelf is presented as a slow process without much emphasis on ice calving or the rise of sea level. The installation tends to show the melting of ice shelves as the result of an unstoppable force difficult to see to the naked eye. The installation builds a representation of this long process by giving it a figure that slowly emerges through time over a period of approximately 10 minutes covering 85 simulated years (Figure 3).

AUDIO-REACTIVE PARTICLE SYSTEMS
One of the key elements of the installation The Great Acceleration was to create a close relationship between a visual and a musical expression using custom written programs. The goal was to design a generative system able to simultaneously create images and sounds from the same refined data. In this case, "sounds" represent modulated versions of audio samples combined together inside a sound composition made of height musical sequences and linked to audio track settings. On the other hand, "images" represent animated visual compositions built mainly using particle systems. Images and sounds are shaped by the same ice velocity data. They also interact in many ways. Each mode of playing of the cello has a singular influence on the particle systems used to express the Ross Ice Shelf activity. This interaction between images and sounds is built upon a communication between two entities: a sound generator and an image generator exchanging information. The sound generator provides in real-time information about the number of the musical sequence being played, the amplitude of the sounds as well as the current mode of playing.
These last two pieces of information are used together to modify the strength of forces applied to particle systems in order to recreate well-defined movements inside a visual composition related not only to a specific mode of playing but also to sound intensity. The visual compositions respond to the sound composition by taking into account the transition from one mode of playing to another and the strength with which they are introduced and then played. The movements of the visual compositions tend to follow the rhythm being suggested by the current audio playing mode.
Short sounds introduce, depending on their intensity and related playing mode, almost invisible Brownian noise inside a visual composition or, on the contrary, sudden movements able to vigorously displace entire particle systems. Another example: the legato sounds of the cello related to the first playing mode create slow theatrical movements inside a visual composition that affect almost every particle by forcing them to follow a path specified by a force field. These new trajectories are defined by taking into account the actual position of each particle. They have the effect of momentarily blurring the lines used to represent the Ross Barrier.

Destructured musical form
The musical form is "destructured" (Bootz & Hautbois 2020). Its processing is not the destruction of a preconceived form or its negation, but an extension of a formal, harmonic or acoustic work. A destructured musical form reveals sonic details which did not appear in the original work and which are changing its characteristics, much like a magnifying glass focusing on fragments of a drawing under observation.
The musical score is controlled by the sound generator which gives life to musical aspects of the work not present in its original and closed form: details underlined by the reorganisation of the musical form, by the application of filters activated in real-time or by the balance between voices.
In this case, the music explores new sound configurations of the initial work in order to replay endlessly, but always in a different way, through interactivity, the inexorable process of ice shelves retreating.
The destructuring of the musical form is orchestrated by performing different actions: The differentiated treatment of the four sound sources extracted from the initial piece. (ii) The application of filters and spatialisation effects based on the reading of velocity curves. (iii) The interaction of the public with the installation. (iv) The reorganisation of the musical form itself according to arithmetical constraints.

Arithmetic constraints
The original musical composition lasts 9 minutes and 16 seconds. As a multimedia installation theoretically has no end, the musical form had to be rethought in order to take into account repetitions of sequences, while avoiding this repetition being too frequent. As part of the installation, the musical composition is reorganised each time the cycle of eight sequences comes to an end.
These new forms of arrangement are not randomly chosen. They follow arithmetic constraints that allow 15 different combinations while maintaining a sense of musical order. Considering the musical sequences numbered from 1 to 8, the following rule has been applied: each sequence n can only be followed by the following sequence n+1 in the initial series (which constitutes the order of succession of the musical work) or else by the sequence located five steps forwards in the series n+5 (Table 1).
These new forms of arrangement between audio sequences don't affect the way the visual compositions are organised: The visual compositions follow a fixed timeline. They start by drawing a landscape that depicts the Ross Ice Shelf in 2015 and slowly evolve through the years until reaching 2100 that marks the end of the simulation period.
Nevertheless, each new cycle is not identical to the previous one. Each cycle of musical sequences being different from the previous one, the visual sequences, which react to the music, evolve each time in a different way, while being generated from the same velocity data linked to a specific year.
The arithmetic constraints applied to reorder the audio sequences at the end of each cycle permit to always end a cycle with the audio sequences 4 or 8. This example shows how the rules that associate musical sequences with visual ones are defining a non-unilinear structure which allows building audiovisual momentum and not random chaos.

Interactions between visual compositions
Interactions and interactivity play an important role in the arrangement of visual compositions and evolution of the musical form. The main visual compositions are generated in real-time from velocity data related to a specific year and respond to different modes of playing. They represent individual images of the datascape drawn to show the evolution of the Ross Barrier using particle systems ( Figure 3). This datascape is presented through the eyes of two different cameras. The first camera is used to display a panoramic view of the curves and revolves slowly around them. It works in tandem with a second virtual camera which is used to target specific curves of the datascape from another point of view. This camera is closer to the curves and moves towards them. It provides fragmentary and animated views from above. The visual sequences created using these two different cameras last between 10 and 15 seconds. They can be presented one after the other but can also be automatically swapped with three other types of visual sequences in relation to the musical sequence being played and its attributes. These three other types of visual sequences represent: Icebergs that have broken off an ice shelf.
(ii) Meltwater. (iii) Filtered views of the dataspace. They are built on knowledge acquired while working on audio-reactive point clouds (Di Bartolo 2019). The transition from one visual sequence to another is related to sound events. For example, when the cello ceases to be audible, another type of visual composition can appear. This composition represents icebergs moving slowly towards the camera (Figure 4). While this representation is still made of particles, it offers a counterpoint to the visual compositions displaying the datascape. Its stability contrasts with the slow and long movements generally applied to the datascape according to a specific musical playing mode. The third type of visual compositions gives birth to abstract forms. It has been created to generate meltwater representation from the inside of an ice shelf ( Figure 5). The virtual camera used to capture this scene is inside the particle system used to define the visual composition. A turbulence field constrains the particle movements in order to give shape to meltwater slowly going through sheets of ice.
While these last two types of visual compositions are not directly created from ice surface velocity data (unlike the datascape) they evolve with it. The size of icebergs decreases over time while the movement of meltwater is accelerating throughout the entire simulation period. The shrinking size of icebergs shows they are doomed to fade. Icebergs act as contemporary vanitas. They remind us of the transience of life (Remaud 2020).

Experiencing interactivity
The fourth and last type of visual compositions is only being displayed in interaction with the viewers. Several motion detectors are used to monitor the movements of the public inside the exhibition space. The data outputted by the different sensors is processed to assign their current location to four different zones of equal area. It allows the viewers to interact with the datascape based on its location. When the presence of a visitor is being detected, in one of the four zones, for the first time of a simulation year, the fourth type of visual composition is used to filter the datascape by slowly redrawing its largest region while masking every other visual element ( Figure 6). This interactive behaviour creates a mapping between the simulation space and the exhibition space. The location of each sound in the quadraphonic space (defined by the installation speakers) is established according to the position of the velocity curves and the position of their respective playhead in the sequencer IanniX. The public is therefore surrounded by sounds, each having its own trajectory in space.
This fourth type of animated visual composition is, as the datascape, responsive to the musical composition and its different mode of playing. It also shows a progression throughout the simulated period using progressive colour changes.
The audio composition responds to the appearance of this fourth type of visual. The display of a single region is used to mute all the sounds which are not related to the selected region in the sound generator. This filtering process allows the viewers to listen closely to a single velocity curve. It produces a zoom effect by focusing on a musical detail of the electroacoustic piece which participates in the controlled and indeterminate destructuring of the audio work.