Moving Data: Artistic tendencies in visualising human and non-human movement

How does data ‘move’? How can we both feel data moving and feel movement through data visualisation? As more of our media become data-based and driven, we need to ask: how is movement being registered through data visualisation? We need to inquire in to how specific practices of visualizing data – both analytical and artistic – constrain or help us explore all kinds of movement, including that initiated both by human actions and nonhuman forces. This paper proposes that artistic approaches using ‘movement data’ within experimental animation, including our own artistic work pull made in 2017, as well as choreographic and experimental cartographic visualisation practices register a ‘feeling’ for and of moving data. The use of ‘movement data’ – typically geographical x,y coordinates accompanied by timestamp data – has been an strong focus of big data visual analytic research in to geospatial movement of human, animal and inanimate objects tracked via GPS devices. However, the overall conceptual approach to movement comes from data science influenced by a mathematical (and implicitly Euclidean geometrical) framework. Here movement is conceived as series of discrete positions occupied by moving objects, connected through the sequential unfolding of time. Space, time and object then give rise to discrete data sets that can be analysed and visualized either separately or together through various techniques. However, this approach runs into many classic conceptual problems, such as how to account for movement between time instances, and how to represent collective movement heterogeneously as the differential of the relations across all this data. We argue that these problems pertain to questions of how to register the way movement moves (changes) and how data participates in such movement moving. We will explore approaches to moving data via our own artistic practice-based research developed in the audiovisual installation pull. Here we took x,y coordinate and timestamp data derived from a cinematographer’s movements while shooting underwater wave sequences over several hours. This data was re-animated using 3D visualization and fluid simulation techniques. We will also look at examples from choreographic visualization (William Forsythe’s Synchronous Objects, 2010). We will propose that by focusing on movement as a field of relations, artistic approaches to visualising movement data might enable a feeling of (data) moving to register.


INTRODUCTION
How does data 'move'? How can we both feel data moving and feel movement through visualisation techniques? How might specific visualisation practicesboth analytical and artisticconstrain or help us explore all kinds of movement, including that initiated by both human and nonhuman forces? This paper proposes that artistic approaches using 'movement data' within experimental animation, experimental high-speed cinematography and some choreographic data visualisations, register a 'feeling' for and of moving data and image. On the other hand, visual analytical methods that help locate 'patterns' in movement data might not be helpful for registering the processes that comprise movement as it actually occurs. Instead, such approaches are interested primarily in movement that has taken place and in then analysing and mining those occurrences for their implicit or buried significance.
Following a line of practice-based thinking that comes out of radical choreographic theory (Forsythe et al. 2009;Manning, 2009) and an understanding of the moving image inflected by Henri Bergson's philosophy, we will suggest that the felt registration of movement requires aesthetic strategies that go beyond visual representation and analytics. What, then, do we mean by a felt registration of movement that is conveyed imagistically? There are many media technologies for visually recording and capturing movement, running the gamut from traditional cinema through to motion capture and more; each has entire histories and techniques for representing and conveying motion and hence we are wary to not impossibly broaden the scope of our thinking here. We will need to put aside the many debates in cinema studies, for example, concerning the successes and failures of moving image's capacities to actually work with movement (for example, Deleuze1989;1986) However, we do want to immediately clarify that we are not concerned with whether data visualisation of movement affects us emotionally but rather how the movement of data, and visual techniques that record and present movement generally, might be felt in their very moving. Another way to put this would be to first ask: how does time, changing in onscreen space, feel, given that it is actually impossible to see time? And following from this: what can data and moving image aesthetically offer in registering affectively 'moments' of time's imperceptible movements? We believe that artistic efforts to work with movement data can bring a sensibility attuned to the relationality of movement and time to visualisation. We also suggestin our consideration of both our practice and in collaborative work that deploys data visualisation to explore the choreographic practices of William Forsythethat conventional movement data sets must be carefully reconfigured in their design and visualisation if they are to afford a felt registration of movement. Instead, we will argue that this felt registration can be aesthetically composed for by creating work that itself sets up relational conditions among its visualisation techniques and between other medial and/or nonmedial elements. These might include sonic, spatial, collaborative, corporeal and sculptural entities, among others. It is the differences among these entities' 'movement paths', speeds and intensities that combine and juxtapose to generate an emergent sensing of, as Erin Manning puts it, 'movement moving ' (2012, 14). Furthermore, contemporary visualisation technologies and techniques have their own specific capacities (and limitations) for marking the passing of time and these must be taken in to account when artistically composing for these relational conditions among elements so as to enable movement's felt registration. We will later elaborate upon how two techniques can be developed within, first, cinematic and then, second, data visualisation using this relational sensibility. We discuss our own audiovisual installation work pull (Barker and Munster 2017) as a realisation and exploration of such techniques.
Finally, we will gesture toward other examples such as William Forsythe's Synchronous Objects (2010) to indicate how other artists draw out a feeling for of movement in its very unfolding using data visualisation, albeit in vastly different artistic practices.

MOVEMENT DATA, FROM THE PERSPECTIVE OF VISUAL ANALYTICS... AND ITS LIMITATIONS
The use of 'movement data'typically geographical x,y coordinates accompanied by timestamp datahas been an strong focus of big data visual analytic research in to geospatial movement of human, animal and inanimate objects tracked via GPS or positioning enabled devices.  (Fraser et al. 2016).
One of the main reasons for developing a visual analytics approach to often-massive movement data sets has been to enable representation of data as, for example, aggregated spatial flows. Data visualisation in many historical accounts itself begins with Minard's famous 1869 aggregation of troop movement statistics using Sankey flow conventions. Visualisation has, then, long been deployed for its capacity to draw the eye to, for example, intensifications of multiple and aggregated movement pathways within an image area (Andrienko et al. 2013, 10ff). But --as visual analytical research itself is only too aware (for example, Andrienko et al. 2013, 2)movement data is a form of sampling, which extracts discrete information from activity or action that is continuous; that is, movement. As a result, the process of collecting or sensing data points to create a movement data set is equally a process of selection in which the movement between timestamped data points, however miniscule, will have been omitted from the set.
The data analytic approach to movement derives implicitly from Euclidean geometrical and linear/sequential visual and ontological frameworks. This seems reasonable in many ways, given that most tasks to be solved for movement in relation to, say, computer vision require data rendering within the x,y,z coordinate spaces of 2 and 3D visual computation. Here movement is conceived as series of discrete positions occupied by moving objects, connected via the sequential unfolding of time. Space, time and moving object, then, give rise to discrete data that can be analysed and visualised either separately or together via various techniques. However, this approach runs into many classic philosophical problems: how to account for movement between time instances; and how to represent collective movement as heterogeneous rather than simply aggregated. These problems lie at the core of how to use visual means to register the way movement moves; the way it simultaneously continues and changes as it unfolds as a process.

PROCESS-BASED APPROACHES TO MOVEMENT
We will touch here on an entire field of thinking that has recently revived these issues via a reconsideration of the process-based philosophy of not only Bergson but also Alfred North Whitehead and William James. What we are particularly concerned with is the different way process-based philosophy approaches movement, which could be very loosely characterized as: first, a preoccupation with transition and the production of continuity in any temporal direction over the reduction to position and sequence; and, second, with acknowledging the nonlinear multiplicity of any movement in process, over and above its causal attribution to either a subject or object at a discrete point in time. It is not that there are no discrete moments and that everything is constantly in flux -a somewhat confused criticism sometimes levelled at process-based thought. Instead, experience is considered as made up of complexes of actual occasions, to use Whitehead's terminology (1978, 22ff), whose nature is only ever given as each occasion becomes or is in the process of unfolding what it is. One can only ever know how each actual occasion will come to be what is, as it is in process and in relation ('prehending') all other occasions in the 'complex' (nexus) in which it (and others) are unfolding.
To give a concrete example, the choreography of throngs of people walking through a city street could easily be positionally captured and correlated with its timestamps. Out of such a dataset, movement paths could be mapped and movement trajectories analysed. But what about that common and yet awkward experience of two walkers approaching each other from opposite directions, abruptly stopping less than a metre apart and, then facing each other, attempting to shuffle against each other's potential onward movement so as to continue their own path through the urban surrounds? How might one visually present such a set of relational entanglements in which, somehow, the differing flows of movement are both cut and yet continuing to attempt to form? How is this felt interruption and indeed transition toward continuing movement actually playing out in this minor choreographic event? In even such a simple activity as two people trying to walk on the same piece of pavement at the same time the multiple unfoldings and refoldings back on to each other of movements create a specific space between and surrounding the two walkers and a duration that sometimes feels as if there is no escape. What indeed needs to be presented, in order to evoke, the mild frustration, embarrassment and sometimes humour of such everyday occurrencesalong with the ways in which the forward movement of each walker will have both come to this and re-emerge out of itis a sense of composing for the shifting space-time between the walkers walking, emerging in the actual event. How, then, might it be possible to compose for such spaces using computational visual techniques?

CINEMATICALLY VISUALISING A FEELING OF MOVEMENT
The multi-channel audiovisual installation pull focuses on one momentary experience (a nexus of actual occasions), asking us to sense its lived time as extraordinary. This experience is the moment a large wave forms and you dive deep it as it breaks above. In this instant, everything changes: there is a calmness, silence and energy, transforming the sensing body in this submerged moment. In pull this is drawn out infinitely, suggesting not a discrete time or space that could be simply mapped, but rather a nexus of occasions simultaneously prehending both their pasts (the moments that came before, such as diving under) and forever apprehending their future: how will the wave form; when will it crest and crash? The installation presents the experience of submergence from different perspectivesthe filmic one, shot underneath a wave; and the capture of that experience as movement data. Without narrative and designed as two seamless replays of that moment, the visual dimension of the work is highly cinematic. The composition pull's audiovisual exploration of time and embodied perception uses water as a force outside humans' short 'moment' in geological time.
Underpinning the work is a challenge to the moving image's relationship to movement and temporality. Cinema's movement is generally understood in relation to frames per second (fps): both the speed of the camera's recording of a moment; and the speed that successive frames pass through a projector or digital playback system. This understanding of the moving image assumes that a series of discrete momentsframesare held together as movement for the viewer through these various systems of playback. Here, it is the arc of moving images over a period of time that sets up the premise for movement to occur across the timespan of the film; that is, its total number of frames per second then calculated as the length and duration of the time it takes to watch the film unfold. The piece pull challenges this logic through a series of interconnected techniques. These techniquesextreme slow motion, forward-reverse motion, and seamless loopingoperate as a perceptual machine to 'unhinge' the work from the progressive movement of discrete frames so as to enter a space where frames 'bleed' and linearity loses its grip on temporality.
Bergson's theory of duration informs our understanding of time, movement and experience in this work. For Bergson, time needed to be thought aside from the spatialised linear flows of beginning, middle and end. Such vectors imply that timeand hence our experience of it and in itis something that can be precisely mapped. To measure time is to measure intervals across a specified distance. Yet our experience of time, as we have previously argued, along with various technologies and techniques for measuring time such as timestamps, comprises transitions and intervals between the intervals. Something is left out that is always 'in' time as experience: time measured is not time felt. As Bergson said, "In between these simultaneities anything you like may happen" (1946, pp.10-11). Rather than the precise measurement of timewhat Bergson calls a quantitative multiplicityhe offers us duration: a qualitative multiplicity where time can only be understood as what is experienced, what is felt as passing, changing (Bergson 1960, p.105). Here, movement, understood as what changes in time, cannot be expressed as sequential motion but rather is felt in experience.
Here, things could be moving with respect to their past (memories), their present, whatever is actually unfolding, and their future, or multiple potential unfoldings all at once. The duration of an event moves as freely backward as it does forward. Most instances of reversing footage in the history of cinema operate more as a gimmick or device to create a visual cue for a flashback, although there are also other examples of the use of reverse editing in cinema (see Tohline 2015).
In pull, reverse motion of the wave breaking became a powerful effect that, when insinuated subtly in to the cinematic channel in the editing process, thwarted the vector of forward progressive motion. Instead, time and the wave's movement both ebbs and flows, like the force of the water. These altered directions are not evident when watching: the shift from forward motion to reverse works seamlessly in conjunction with the endless looped structure of the piece. Such looping ensures cinema's desire for temporal ordering via beginning and end is thwarted: it places the viewer in a changing yet strangely familiar landscape. Such a technique become a way to consider the experience of time/events durationally: these do not occur as abrupt shifts for even the most pressing recollections still position themselves relatively seamlessly into the flow of experience. To recall is equally a part of the present.
Most crucial to considering the wave breaking moment as Bergsonian qualitative multiplicity in pull was our use of extreme slow-motion cinematography. The wave sequence was shot at 2000 frames per second (fps)and slowed down even further in post-production.
Played back at 25 fps, things ordinarily invisible are amplified to surreal proportions: details of water density; currents moving in multiple directions; air bubbles and micro-organisms.
In working directly with vastly altered frame rates, the intention was to generate an experience of time appearing to 'almost stand still'. The focus is on bodily experiences of movement and duration in this moment. The aim is not to record a moment in time, but rather to be aware of time as the matter of experience.
High speed cameras are used in a variety of biomechanically led scientific, medical and sporting applications (Dalton 2002) And while this means we can now forensically analyse the rich detail offered by these super-fast frame rates we can also learn a different cinematography by feeling in to the force of movement through the image, rather than scraping to see its traces. Such scraping is never satisfied with even the most minute detailswe can never see all that is unseeable because there will always be some 'quantity' leftover to divide and measure. Here we come to the heart of Bergson's critique of the illusion of motion in which time and movement are made to coincide as the divisible line that underlies both (Bergson 2005, p.191). The paradox -famously known as Zeno's arrow of time is that the more we see the subdivisions, the more the subdivisions keep coming. The subdividable line, rendered here as the increasing quantity of image data captured by increasing quanta of frame rates, never attains a state of total division or absolute visibility. since this very approach always leaves over a potential remainder to be subdivided or perceived. Instead, to feel imperceptibility, we require different techniques in the creation and stitching together of moving images that are sustained by movement and also create it.

TOWARD DATA THAT MOVES, FEELING ITS MOVEMENT
In pull, cinematically we strove to explore how to register the way movement moves yet how can data participate in such movement moving? For this element of the project we took the x,y coordinates and timestamp data derived from a cinematographer while shooting the underwater wave sequences over several hours. Wearing a GPS enabled watch designed specifically for tracking the motion of surfers, we collected data on the cinematographer's movementsvelocity and positionas he moved through the water.
It was crucial to consider the data not as a representation of movement conceived as series of discrete positions occupied by a moving object (the cinematographer's body), re connected through interpolating each moment between the timestamp data captured as seconds. Here, space, time and object then give rise to discrete data sets that can be analysed and visualized either separately or together through various techniques.

Figure 3: Cinematographer Chris Bryan under a wave
and wearing the GPS enabled watch capturing movement data.

Figure 4: Screenshot of the motion path produced by x,y coordinates of the cinematographer's geospatial location correlated with timestamp data over a 3-hour period taken from the visual interface on the GPS enabled watch.
Thus, like the filmic event of the wave forming and breaking, data had to participate in the event of moving, and hence it was not possible to simply animate an object moving along the motion path created by the dataset. As Bergson reminds us, rather than thinking about the line, we must consider the action itself (1946, pp.164-165). In pull, doing this required creating two kinds of movement within the animation scene. The first was the movement of the fluid simulation, which was created using a series of parameters taken from the fluid mechanics of reef breaks. The actual motion path created by the x,y and timestamp of the cinematographer's data while shooting was animated as a negative space moving through and against the fluid. This path was set up as a collider in relation to the fluid sim so that the forces of the fluid against the continuity of the motion path constantly formed and reformed in relation with each other. Rather than looking at a representation of a movement path, the negative space appears like a bubble that both has its own momentum but is also being formed as it moves in relation to the moving fluid sim. The choice of 'negative space' was important since it suggests something whose form is impliedregistered implicitly rather than spatially given.
The moving data here gains its traction from a set of occasions that are all actual, yet not necessarily visualised as such. The movement of a (human) body bobbing around subject to nonhuman forces and yet also intent on a specific activity is felt at the nexus of a set of relations, forms and forces that concresce; that is, come together in their multiple and qualitative changings in time, each also in relation with the other(s). Movement data gains (its) duration: 'A duration is a complete locus of actual occasions in 'unison of becoming' or in 'concrescent unison' (Whitehead 1978, p.320).
We have so far presented a different approach to not only using cinematic and movement data artistically but to also considering what movement might be when thought visually with the help of process-based philosophy. We want to now gesture toward some other artistic approaches that rigorously reconfigure the relations between movement, data and the visual, suggesting that a different aesthetic tendency might be emerging from these practices. We will turn first to the project that grew out of the choreographic and dance work of William Forsythe's Company and culminated in the large project Synchronous Objects (2010). Although we will concentrate on only one aspect of this projectthe use of multivariate datasets taken from recordings of Forsythe dance works to create visualisations of the choreographic thinking operative in this movementit is important to also say a little about Forsythe's overall practice.
His motivation (Forsythe 2010) for working on the Synchronous Objects project came from a desire to provide a visual resource that might respond to and provoke the question: what does physical thinking look like? The 'data visualisations', which in myriad ways drew out complex patterns in the movement of a videoed dance piece, One Flat Thing reproduced (Forsythe 2003), were responses to that question. They were not attempts to represent the movement of the dancers but rather to open a conversation about other modalities for relating to and engaging with the felt or embodied knowledge that dancers enact as they dance.
Forsythe and his dancers already work in unconventional ways with movement, since their approach is not to learn dance positions, routines or sequences but rather to create fields of movement interrelations that are always open to change every time they are performed (see Manning 2009). They are, then, concerned with a conception and practice of movement in which positions and routine cede to change and the deployment of rigorously practiced techniques of improvisation. The emphasis is on movement as it both arises out of relational gesture and through its own moving or, as we have characterised this above, change.
In the Synchronous Objects project, Forsythe collaborated with visual designers, statisticians, dance theorists as well as his ensemble to create a series of visual objects from, among other quantitative and qualitative data, a large dataset captured from One Flat Thing reproduced and from various other video recordings of rehearsals for this work. Importantly, the statisticians and designers took up the challenge of Forsythe's own choreography to understand movement. In their own research to emerge from the project, the data designers state that they adopted Forsythe's definition of dance as connective space between: dancers, sets of choreographies and their dynamic interplay. It is what happens between positions not in the holding of and vector toward the next position. (Alquvist et al. 2010, p.549).
Interestingly this then required a rethinking of movement data as not simply delineated solely by spatial and timestamp data but rather of the necessity of engaging space and time themselves as dynamically interrelated. Traditional movement data sets were still used including: x, y and z coordinates (since the movement was itself a recording of dancers along depth axes as well), and 4ms timestamps for all coordinates. Yet the designers had to think about how these datasets would dynamically relate in the visual objects produced. In some of the visual objects generated, a new technique was used that the designers call 'linked brushing' (Alquvist et al. 2010). In different visual objects, spatial data, time and other attribute data (dancer's name and so on) call each other up through a user interactively 'brushing', as if caressing, an object. This results in changes to the visual object and the generation of complex overlapping visual trajectories in which movement paths criss-cross to create vectoral topological spaces (for example, Synchronous Objects 2010b).
These trajectories no longer represent where a dancer or even the dance moves in space throughout time but rather unfold the felt complex of the relations moving the dance as an entire event, while it was being danced.
Although using quite different visualisation methods, this relational approach has similarities to the way we approached movement and movement data in pull. Both projects work with movement, space and time as processually brought in to being; all emerging in their actual unfoldings through complex series of interrelating and dynamic entities. Importantly these entities are also immanent to these processes. For Forsythe, the dancer is only an entity in relationin counterpoint with other dancers, with the space in which the dance takes place, with the actual performance taking place and with the techniques all bring to the ensemble. The question for visualisation then becomes: how can visual composition engage with movement as this dynamic durational topology? This is not simply a question for artistic visualisation but might also have approaches to offer a range of difficult visualisation tasks such as situational awareness for computer vision in moving autonomous entities.
It is important to distinguish this kind of visualisation approach from using a technology such as motion capture to demonstrate how a dancer or performer is moving or to even create an aesthetic visual depiction of this. There is important research being undertaken within this area in, for example, projects such as Thecla Schiphorst's Moving Stories (2012-2015. However, there is a tendency within even experimental motion capture visualisation to continue to think of movement as that which necessarily proceeds from a moving subjectgenerally the body captured by the motion sensors as genesis of action. Having already set up the origin of movement by locating this in a subject, it is then difficult to understand movement as arising out of a field of relations whose time and space are also generated in the very moving set apace by this field. Nonetheless, there is nothing to prevent future artistic work that explores movement as processually emergent from being carried out via a wide variety of visualisation practices and technologies.
But where we might concludewith a provocation for work yet to be undertaken rather than to summarise what has already been discussed here is with the plea that movement not be locked down by data or by any form of visualisation. If movement datasets are indeed large this is not simply because movement takes place over time hence producing many data records. Nor is it that movement needs to be broken down preciselyin a kind of post-Muybridgean trajectoryin order to reveal the hidden depths that the now high-speed moving image might offer. While both forms of visualisation incur large amounts of data, the reduction of movement to the 'patterns' within this data or to locating the 'invisible' will never resolve its rhythm or potential trajectories. Movement, as a form emerging in duration, will always gesture to its elsewheres: the movement just preceding this one, setting the next one up; the movement now unfolding toward a yet un-imaged future.
To watch thoughtful contemporary dance; to see crowds negotiate a 7-way intersection; to look at a wave cresting and crashing from above or below; all of these ask us to feel movement emerging in the middle. Let's see if we can also visualise from this imperceptible space of the in-between where movement multifariously unfolds.