The Impossibility of Control: Real-time Negotiations with the Heart

We present a practice-based research project, which examines temporal relationships between physiological processes with human movement and mediated environments. Ongoing research will provide a foundation for the creation of performances in which real-time heart rate data from contemporary dancers is used to generate click tracks for live musicians. The complex constructions of time in this project necessitate novel visualization strategies throughout the research, development, creation, and production process, including: time domain display of data with rich and facile interaction using bokeh, visualization of biometric data as a form of biofeedback, and novel representations of time in music notation


INTRODUCTION
In this paper, we discuss our practice-based research regarding temporal relationships between biological processes with human movement and mediated environments.We are investigating ways in which to cultivate an oscillation of awareness and distribution of agency between bodies, sensors, and network.Our focus is not on the sonification of biosignals, nor in the use of bio-data to control attributes of other media such as music and lighting.Rather, the intervention of biosensors in our choreographic and compositional process is intended to complicate linear notions of cause and effect, as well as internal and external processes of self, other, and environment.
Ongoing research will provide a foundation for the creation of performance and installation projects in which real-time heart rate data from contemporary dancers is used to generate click-tracks for live musicians.
Equipped with a wireless, electrocardiogram (ECG) unit, each dancer will perform choreography that is designed to generate intentional arcs in his/her heart activity over time, through a combination of movement, breath, visualizations, human interactions, and environmental stimuli.Each musician will listen to a live, highly variable ECG click track from one dancer, to guide the tempo at which s/he plays from the prepared score.The dancers and musicians will negotiate with one another, as wel as with the digital computing system responsible for electronic sound, ways in which to navigate between time as prescribed by the musical score, versus time as realized in the live event, versus time as perceived and felt in the body.
The complex constructions of time in this project necessitate the development of software tools for the representation and quantitative analysis of relationships across biological and computational processes during performance.Essential tools include beat-detection on an ECG waveform, as well as sonification and visualization of this data in a variety of ways.The software is intended to produce not only musical results, but to aid the musicians, dancers, and directors during the rehearsal and performance process, increasing understandings of the relationship between action and environment with physiological processes.
The paper will proceed with a contextual overview of past and current applications of biosensors in medicine, music, and performance practices, followed by an in depth explanation of our project and the software tools being developed for the analysis and visualization of real-time streams of biosensor data during performance.

Biosensors in Medicine
In the book "Biomedical Signals and Sensors 1", Eugenijus Kaniusas states that: "within the scope of biomedical signals and sensors, a biosignal can be defined as a description of a physiological phenomenon, irrespective of the nature of this description" (2).Dating back to Hippocrates of Cos (circa 460 BC-377 BC), the Greek physician regarded as the father of medicine, diagnosis was based on the patient's verbal account of symptoms, as well as on the assessment of biosignals by means of visual inspection, palpation, percussion (striking the body with short, sharp taps), and auscultation (listening to the sounds of the body) (Kaniusas 7).
The gradual development of biosensing tools in medicine responded to the need for greater objectivity in evaluation of biosignals, including storage, analysis, classification, and comparison beyond the moment of inspection (Kaniusas 12).An early attempt at the analysis of biosignals involved musical notation in which "…the height of the note could be used for a qualitative coding of biosignals, [while] the rhythm of the successive notes could be used for a quantitative coding."(Kaniusas 13).Throughout the 19 th and 20 th century, technical tools for recording the bioelectrical activity of the body continued to develop, including the introduction of the electrocardiogram (ECG), a device for measuring the electrical activity of the heart.The first-ever human electrocardiogram was recorded by British physiologist Augustus Waller in 1987, followed by the invention of the 3-lead, clinical ECG by Dutch physiologist Dr. Willem Einthoven, implemented in hospitals during the early 1900s (AlGhatrif and Lindsay 2).

Biosensors in Music
The integration of biomedical sensors in music gained popularity during the 1960s, with emphasis on the sonification of biosignals.Key figures in the development of early biosignal-driven music include Alvin Lucier who premiered the first ever production with live electroencephalogram (EEG) feedback from the brain, titled Music for Solo Performer (1965); Richard Teitelbaum, a member of the Rome-based group Musica Electronica Viva (MEV); and David Rosenboom, founder of the Laboratory of Experimental Aesthetics at York University in Toronto (Arslan et al.).Rosenboom's seminal publications: "Biofeedback and the Arts" and "Extended Musical Interface with the Human Nervous System" outline critical experiments, as well as conceptual and technical issues related to performance with biosensors, including what he refers to as the "emergence of the biofeedback paradigm" (12)(13)(14)(15)(16)(17)(18)(19)(20)(21)(22)(23)1997) (Arslan et al.; Ortiz et al.).

Biosensors in Performance and Dance
Critical intersections of biosensors in performance have evolved alongside socio-political and philosophical attitudes towards the body.Early investigations are prevalent in the work of French physiologist (and influential cinematographer) Étienne-Jules Marey(1830Marey( -1904)), who expressed severe distrust of sensory observations of the body, and endeavored to reveal the natural body through a combination of mechanical sensors, transducers, and inscriptions that measured the "…invisible movements of blood, breath and neuro-muscular oscillations" (Salter 222).During the 20 th century to present day, investigations of the internal state of the body have permeated the work of numerous performance artists such as the Vienna Actionists, Stelarc, and Orlan who turn to technologies ranging from chemical to surgical interventions to challenge notions of biologically determined, normative bodies.The influential writings by feminist, cultural theorists such as Haraway and Hayles have inspired renderings of hybrid, cyborg, and posthuman bodies which blur boundaries between "…the natural and the artificial, biological and computational" .
From the 1960s to present day, research and creation regarding the integration of biosensors in contemporary dance and music has evolved through collaborations between choreographers and composers with researchers in computer science, electrical engineering, physiology, psychology, phenomenology, Brain-Computer-Interfaces (BCI), Artificial Intelligence (AI), cognitive science, and more, with support from academic and private institutions internationally.Presently, research and creation in this filed continues to thrive at centers such as IRCAM, CNMAT, SARC, MIT's Media Lab, Hexagram, and more.

PROJECT OVERVIEW
This project began with our shared vision to create an evening-length production for music and dance, in which heart rate data from twelve contemporary dancers is used to generate real-time click-tracks for twelve corresponding musicians.In this scenario, the choreography must be designed to initiate intentional arcs in the heart activity of each dancer over time, based on a temporal pathway defined in the musical score.Each musician, click track in ear, must negotiate between the prescribed, polytemporal score, versus the actual heart rate of a dancer during the live event.What has drawn us to this project is that it is, essentially, impossible.The absurdity of relying on heart activity, which is an autonomic and highly variable temporal process in the body, as and underlying clock -in fact multiple cardiac clocks -for a musical composition, will create severe uncertainty during performance.The relationships between the passage of time as felt in the bodies of performers and observers, versus time as realized in the event, versus time as prescribed in the musical score will be extremely complex.Any semblance of control in this situation will be highly distributed between the many human and nonhuman performers at play, and require an oscillation of awareness in order to flow between processes which are at once internal and external, voluntary and involuntary, and biological and computational.
How and where can we begin to choreograph and compose in this scenario?The deeper we move into this project, the more we realize the necessity to develop training practices with the performers, as well as software tools for the visualization and analysis of the temporal relationships between embodied and environmental processes underway.Further, we are drawing from research in diverse disciplines, including studies from physiology, psychology, cognitive science, cardiology, neuroscience, computer science, BCI, AI, and more.Topics emerging from this research include interoception, empathy, entrainment, embodied cognition, enaction, and temporal perception.The intervention of biosensors in this practice-based research is reconfiguring and reformulating the questions, methods, and materials being generated in our choreographic and compositional process.

What is being measured and visualized?
As described in the book "Biomedical Signals and Sensors 1", "the heart is a muscular organ whose rhythmic contractions force blood to circulate" in the human body.Heartbeats mark "the repeating pattern of contraction and relaxation of heart muscles" due to electrical activity.The peaks and waves shown on an electrocardiogram (ECG) describe the electrical excitation of the muscles of the heart, that is, the depolarization and repolarization of the atria and ventricles.The electrical events in the heart proceed and initiate the mechanical, pumping action (Kaniusas).The heart rate is the speed at which the heart contracts, per unit of time, typically in beats per minute (bpm).
In this project, we are not averaging the heart rate of the dancers over time; rather, we are extracting the peak of each heartbeat, called the R wave (the upward portion of the wave representing the depolarization of the ventricle) to generate a realtime click for a musician.Identifying each successive heartbeat is important, as it will preserve the dynamic patterns of variability within the heart activity of each dancer, creating unique musical rhythms.As such, with real-time readings of the ECG signal we may attempt to investigate close-to-immediate interactions between biological, behavioral, and computational processes, as well as potential for entrainment between dancers with one another, the music, and other aspects of the environment.

Temporal Correlations
As described in the "Primer on the Autonomic Nervous System", the heart itself has an intrinsic ability to regulate the electrical impulses that trigger rhythmic contractions of the ventricles.Additionally, the autonomic nervous system (ANS) influences the electrical activity of the heart through involuntary and unconscious processes.The ANS is comprised of the sympathetic nervous system (SNS) which activates in situations of stress with reactions such as fight or flight, and the parasympathetic nervous system (PNS) which works to maintain homeostasis in the body at rest, and following excitation .
Given the autonomic character of the cardiac system, it is impossible for the dancers in our project to exert control over the temporal activity of the heart directly.Instead, we must investigate and construct reproducible relationships between the behavior of the heart with other physiological process such as breath, and activities including movement, visualization and memory techniques, human interactions (e.g.eye contact, touch, and weight-sharing), and environmental interactions.Obvious instances of cause and effect between physical exertion with heart rate aside, we are researching in/with/through the bodies of each performer regarding a range of subtle and extreme physical-emotional-psychological states that interrelate with cardiac function.

Heart Rate, Respiration, and Movement
To date, we have conducted several experiments regarding temporal correlations of heart rate and heart rate variability with breath.As pointed out by Kaniusas, "The course of the heart rate reveals respiratory related oscillation, i.e., an increase during inspiration and a corresponding decrease during expiration" (18).We have witnessed this phenomenon repeatedly in our structured breathing experiments, during which a dancer wears both the ECG and a respiration belt sensor around his/her ribs.During sessions with two different performers, we identified reproducible patterns of temporal correlation that result from holding the breath, deep or shallow panting, slow inhales and exhales, and various other combinations of duration and amplitude in respiration.Importantly, we repeated the breathing structures with each dancer lying supine, sitting up, standing, walking, and eventually improvising movement, and still observed familiar patterns of relation between ECG and respiration data throughout, but with a base increase or decrease in heart rate related to the level of the dancer's physical activity.
Given this information, we developed a short study for a solo dancer with electronic sound (as pitchbased biofeedback), in which choreography of breath and movement was designed to guide intentional arcs in the heart activity of the dancer over time.Two different dancers performed this solo many times, with success in regards to achieving reproducible temporal correlations between movement, breath, and heart rate.Importantly, a wealth of variables related to physical, emotional, and psychological state exist within this study, and both dancers expressed that every performance of the solo felt quite different qualitatively in their own body and perception of time -despite similar outcomes in the ECG data.This observation highlights the complex relationship between live experience versus databased descriptions of biological and behavioral processes.The results of these experiments were documented with the time domain charts described in section 5.2.

TRAINING TOOLS
In this project, there are three basic tasks that are asked of each dancer: 1. Hold your heart rate around a certain speed.2. Change your heart rate over time.3. Match your heart rate to that of another person.These tasks require each dancer to develop their sense of interoception and an embodied knowledge of ways they can affect change in their physiological state.
Interoception refers to an individual's awareness of the internal, physiological condition of his or her body, including biological signals, and it believed to take place the insular cortex of the brain.Interoceptive awareness is often measured based on sensitivity to cardiac signals, with a suspicion that this outcome represents a generalized capacity for sensing across bodily systems (Herbert et al.).It is postulated that interoception plays a key role in the shaping of an individual's emotions and behavior, as well as empathy towards others, which may or may not be contingent on conscious awareness of interoceptive processes (Cameron; Brain).In our current research, the ability of each dancer to cultivate awareness regarding visceral changes such as the rhythms of cardiac function, would heighten understandings of the multidirectional flow between self, action, and environment.
In order to facilitate this type of training, we have begun development of a number of software tools that manifest the internal state of one or more performers, providing them with a kind of biofeedback in real-time.As defined by Rosenboom, biofeedback refers to "the presentation to an organism, through sensory input channels, of information about the state and/or course of change of a biological process in that organism, for the purpose of achieving some measure of regulation or performance control over that process, or simply for the purpose of internal exploration and enhanced self-awareness" (12).Further, Rosenboom states that: A biofeedback system may be mistakenly viewed as simply a method for stabilizing particular behaviors and, thus, as a static equilibrium system… In fact, these processes all involve the self-organization of dynamical regimes within the organism, aided by the additional information feedback loops of the biofeedback mechanism, in such a way that the evolution of these regimes will tend toward a dynamic that promotes the self-renewal of the organism.(Rosenboom 17) This is the conceptual paradigm with which we are approaching biofeedback.
In our first attempt at designing a biofeedback system, we simply sonified the heart rate of a dancer by producing a click for each heartbeat.We found in our initial experimentation that if the dancer danced along with the tempo of her heart rate, she could make reasonable judgements about the tempo.However, when the movement was unrelated to her heart rate, it was much more difficult to make judgements about the heart rate, and the dancers reported having to shift mental focus away from movement and into a more quantitative analysis of the tempo, which requires the listener to be attentive to a short window of time in the past.This situation was further complicated when the dancer was given a reference tempo to try to match; the task of comparing two tempos with variable phase relationships proved difficult even without movement.These experiments, while brief and informal, were enough to encourage us to look into other methods of biofeedback.
In our second attempt, which was much more successful, we sonified the heart rate of the dancer using a sine tone by scaling the eight-point simple moving average from a range of 40-220 Hz to 300-1200 Hz.We also introduced a reference tone that pulsed at a rate of 1 Hz as a way of making it audibly distinct from the first tone.We then devised four exercises: 1.The reference tone was fixed and the dancer attempted to "tune" her heart rate to the reference.2. The reference would jump to a new frequency periodically and the dancer would have a predetermined amount of time to "tune" her heart rate to the new frequency.3. The reference would change smoothly over a predetermined amount of time between two frequencies and the dancer would attempt to follow.4. The frequency of the reference tone was controlled by a second dancer and the two dancers would attempt to tune in relation to one another.Our initial experiments left us encouraged about the efficacy of this training tool, however, as the two tones enter the critical band and continue to approach each other, it becomes increasingly difficult to distinguish which one is above the other.We provided another cue to the dancer by visualizing the difference between her sonified heart rate and the reference tone using colored light.As her sonified heart rate went flat with respect to the reference tone, the light became increasingly red, and blue as she went sharp.As she approached the tone, the light became increasingly white.The dancers we worked with reported that the visual cues were much stronger than the auditory ones, however, the auditory biofeedback has a strong advantage over the visual in that it may be provided to the dancer privately in performance.For this project in particular, it is important that the dancers cultivate an embodied awareness and adaptive capacity in relation to characteristics of the musical score.

Prescriptive and Descriptive Music Notation
An aspect of this work is the exploration of the relationship between a musical score and performance.Music notation is prescriptive in the sense that it represents a set of instructions and cues that aid and direct the performer.It is not, however, descriptive of any given performance, although the relationship between score and performance may at times be close enough to suggest otherwise.The vertical alignment of notes between parts on a page, for example, suggests the possibility that multiple musicians may play simultaneously, however, given that they cannot occupy the same space, they cannot truly perform synchronously.
This detail about music performance is often overlooked and may seem insignificant, but is a fundamental property that is mitigated by conventional performance situations.In more extreme circumstances, in which performers are distributed across much larger spaces, for example, this property is quickly made manifest.The "now" of musical time is not an instant, but rather a window the width of which varies continuously from the imperceptibly small to the unacceptably large.Our current work is designed to explore this window of time-to enlarge it to the point that its contents become elevated to the level of musical material and even form.To this end, a score is an essential component of this project, not only for documentation and rehearsal purposes, but to set up a compositional tension that must be resolved.
Music notation represents time by encoding it symbolically rather than displaying it proportionally.Imagine that we pick a measure in a conventional score at random and we wish to know how much clock time should elapse between the start of the piece and that measure.The spatial distance from that point to the beginning tells us very little.Rather, in order to estimate how long it would take to get to that point, we must start at the beginning and look at each measure computing a running sum of their duration given all tempo and meter indications.There are many examples of proportional notation-notation that spaces notes proportionally according to the amount of time between them-in the 20 th and 21 st centuries.Notable examples include works by Iannis Xenakis (Psappha) and Gérard Grisey (Partiels), however, anecdote would suggest that these notational devices, while useful compositionally and analytically, are a hindrance to performers who often annotate the scores by replacing those elements, which represent metrical and grouping structures, that were omitted by the composer.In MacCallum's polytemporal compositions, in which performers play tempos given by click-tracks that vary smoothly over time and independent of one another, proportional notation in the score is essential for understanding the relationship between parts.Figure 2 is a short excerpt from aberration for percussion trio 1 in which the rhythms of the three parts are chosen so that they interlock to form a semblance of even 16 th notes in a fourth, continuously varying, tempo.While painstaking, the production of a proportional score with this degree of accuracy is essential to the rehearsal and documentation process.In our current work, the same notational strategies are employed to produce a prescriptive score, however, given that the tempo of each voice is tied to the heart rate of a different dancer, the vertical relationships expressed in the score become a fiction during performance.Despite this complex relationship between representation and realization, the score remains a useful document 1 http://johnmaccallum.com/index.php?page=./compositions/aberration that describes a certain ideal rendering of the piece and can be used as a reference by which to evaluate a given rehearsal or performance.The relationship between the score and performance, however, is far to complex to manage mentally, and it is important to be able to have a persistent representation of that relationship that one can return to at a later time to analyse.A thorough analysis during and after performance is important to communicate to the dancers and musicians in a productive way, as well as to construct an understanding of the contents of the musical window of the "now", mentioned above, in order to make adjustments to areas where compositional intuition failed to anticipate reality.While much information can be gleaned from time domain plots (see below) that illustrate the heart rates of dancers as functions of time and compared to model trajectories, such visualization is limited in its ability to represent musical material.
Therefore, central to this project is the construction of a mechanism for producing a descriptive score from a combination of the prescriptive score and the input from the ECGs worn by the dancers.The result must represent the relationship between the score and its realization; it should simply apply new spacing rules to the score in order to place each element according to when it happened.This is preferable to rewriting the rhythms of each part, which would produce a score that bared little resemblance to the original.

Initial tests done in OpenMusic by Jean Bresson during MacCallum and Naccarato's Artistic
Research Residency at IRCAM and as part of the ANR project Efficace were encouraging.Figure 1 shows a poly object in OpenMusic with two voices.The top voice represents a prescriptive part-this is what would be given to a musician.In the bottom voice, the elements of the top voice have been respaced according to the input from an ECG worn by a dancer (in this image, her heart rate is increasing rapidly due to having been intentionally startled).
While this is an encouraging test, a fully usable system poses myriad significant challenges.We feel strongly, however, that the construction of such a tool would prove immensely useful beyond the scope of the project as a tool for investigating the relationship between performance time and score time.Typically, this relationship is viewed on a time map, which plots score time as a function of performance time (or vice versa).We believe, however, important and illusive information about performance practice would be made clear with a tool that preserved the details of the notation being performed.

Time Domain Visualization
Figure 3 A typical interactive bokeh session running in the browser.
Time domain visualization of data is critical to this project, and while it in itself is not novel, the tools we have chosen and their affordances are worth mentioning.Most often, we produce plots of dancer's heart rates as functions of time in order to visually inspect the relationship of what we expected dancers to do with what they actually did.Additionally, we often consult plots of the ECG and the output of various stages of the beat classification algorithm.This allows us to evaluate the performance of the algorithm as well as to look at the output when things go wrong.Finally, we also visualize data representing certain characteristics of the jitter and latency in the system.This has been an invaluable tool in configuring the WiFi network, as well as helping to identify places where variability in the heart rate has its cause in technology, rather than biology (Figure 3 shows an example of the first two types of plots).
Visualization is currently done using bokeh 2 , a package for Python that provides highperformance, interactive graphs rendered in the browser.Bokeh's architecture, particularly the bokeh plot server, allow us to distribute computation among a number of computers and access the display quickly and easily.Bokeh code is written in Python3 and generates HTML and JavaScript code which runs in the browser using HTML5 and the Canvas.The bokeh plot server runs in the background and acts as a mediator between the plots and models and the JavaScript manifestation of that data in the browser (Figure 4).The bokeh plot server has the option to use redis as a backend for storing plots while operating, and the redis database can be used for storage between sessions.
This system has a number of strong points: 1. Free, open source, actively developed and supported.2. Easily portable between machines and operating systems.3. Dynamic.4. Efficient for large datasets.5. Has a method for persistent storage of plot models which can be reloaded rather than needing to be recreated.6. Distributable between multiple machines.

CONCLUSION
We have presented a practice-based research project for music in dance in which each musician follows a tempo determined by the heart rate of a dancer.This project presents several significant challenges that are mitigated by novel visualization, as well as sonification, techniques.The development of embodied knowledge around ways of affecting change in one's physiology, as well as interoception have led us to begin developing a set of sonification and visualization tools for providing real-time biofeedback to dancers.This project stresses the relationship between a musical score and its realization by highlighting the prescriptive nature of the score and problematizing its descriptive role.We are currently investigating ways of producing descriptive scores automatically from prescriptive sources and input that represents tempo as a function of time.We believe that this work will result in a tool or set of techniques that will have great utility beyond the scope of our current work.

Figure 1
Figure 1 Initial test of the construction of a descriptive score.The bottom line is a respacing of the elements of the top line based on input from a dancer wearing an ECG.

Figure 2
Figure 2 aberration for percussion trio.The three performers play in independent, smoothly-varying tempos.The rhythms are chosen to interlock to form a semblance of even 16 th notes in a fourth tempo.

Figure 4
Figure 4 Diagram of the bokeh plot server and its relationship to the browser and plot models.