Physicalizing Time Through Orientational Metaphors for Generating Rhythmic Gestures

Possibilities for cross-disciplinary interactive performance continue to grow as new tools are developed and adapted. Yet, the qualitative aspects of cross-disciplinary interaction have not advanced at the same rate. We suggest that new models for understanding gesture in different media will support the development of nuanced interaction for interactive performance. We have explored this premise by considering models for generating musical rhythmic gestures that enable implicit interaction between the gestures of a dancer and the generated music. We create and implement a model for generating dynamic rhythmic gestures that flow in, around, or out of goal points. Goal points can be layered and quantized to a meter, providing the rhythmic structure expected in music, while the figurations enable the generated rhythms to flow with the performer responding to the more qualitative aspects of performer.


INTRODUCTION
Performers in domains such as dance and theatre have adapted cutting edge tools for tracking gestures and controlling media onstage to develop new performance experiences.New advancements in tools for media rendering, for example 3D audio and visual systems, provide new possibilities for cross-disciplinary interactive performance.Yet, qualitative aspects of interdisciplinary interaction between performers and systems have not advanced at the same rate.We suggest that the development of new models for understanding gesture in different media will support the development of nuanced interaction for interactive performance.We have explored this premise by considering models for generating musical rhythmic gestures that enable implicit interaction between the gestures of a dancer and the generated music.
Systems interacting with musicians often leverage musical syntax, through self-similarity and other algorithms, to recognise and adapt the musician's gestures in relation to time.The composing of time in dance gestures is a bit different, in that it is not reliant on the division of time in the same way that music is, yet can manoeuvre space (H'Doubler 1940).While dance and music gestures both incorporate a rhythmic quality, the disciplines do not share a common relationship to segmentation and composition of time.A mapping strategy is required to translate between the gestures.Traditionally such mapping focused on triggering sounds or music clips with some relation to the dancer's gesture.As artists developed better algorithms for composing music in real time, mappings were made between parameters of the composition and parameters of the gesture such as duration and speed.As mappings become more complex, interactions attempt to mimic the interaction between human performers.However, interaction between humans involves the performer's situated cognition, balancing intention with the syntax of the domain, e.g.balancing the body's understanding of time with the metric grid of music.
Our work explores methods for constructing implicit interaction between dancers and rhythm generation.To do this we developed a model based on the embodied rhythm of breathing and leverage the rhythmic analysis for poetry.Poetic meter focuses on long and short syllables as a foundational approach, which acknowledges the embodied experience of breath in language, while also providing a basis for musical rhythm.In the context of music, the long and short often get applied to long and short events, enabling the discussion of rhythms with minimal consideration of meter.We couple this concept of long and short events with the concept of goal points and figurations adapted from Schenkerian analysis and Milton Babbitt's Time points (Babbitt 1962).In this way we create an approach to generating dynamic rhythmic gestures figurations that flow in, around, or out of goal points.Goal points can be layered and quantized to a meter, providing the rhythmic structure expected in music, while the figurations enable the generated rhythms to flow with the performer responding to the more qualitative aspects of performer interaction.

Time metaphors
We experience rhythm in many aspects of our lives outside of music and dance.Often literature will refer to rhythms within the body as a basis for human's understanding of rhythm, such as breath, heartrate, blood flow, to name a few systems (Hartley 1989).At the basis of any discussion of rhythm is the human experience of time, a shift or change of experience (Bradley 2008).While time is fundamental to human's interaction in the world, we tend to fall back on metaphor whenever we discuss or conceptualise time.The literal properties of time are limited to (Lakoff & Johnson 138):  Directional/irreversible: events can't be undone. Continuous: the sequence continues. Segmentable: periodic events have beginnings and ends. Measurable: reoccurring events can be counted.
But when expressing our experience of time we quickly fall back on standard metaphors that reflect three ontological categories: Space Time, Moving Time, and Moving Observer.We use language that implies a future that is coming towards us, such as Christmas is coming [Moving Time], or 'I'm just trying to make it to lunch…' [Moving Observer].Although these are not the only ways in which we metaphorize time, these examples illustrate how we do tend to conceptualise them in terms of space.
The idea that time can be conceptualised as a space being moved through is not new or surprising.However, these metaphors offer interesting conceptual possibilities that can affect models that we build from different ontological stances.Consider the similarities and differences between the 'Moving Observer' and 'Moving Time' stances outlined by Lakoff and Johnson.The authors explain the 'Moving Observer' means time is thought of as being locations or regions in space.This metaphysical approach not only implies that events happen in those 'locations' but that locative containers exist prior to anything filling them (Lakoff & Johnson 1999, p.156).A similar understanding can be said of a 'Moving Time'.These again are implied locative containers moving past the observer.Both are a series of containers passing by the observer.
To demonstrate this difference let's consider the experience of listening to a composition.I can conceive of the experience as a stream of events passing me, or as me making my way through the field of composed events.A key component to the stream conception seems to be that I cannot move backwards and therefore am at the mercy of its passing for my experience.In contrast, my moving through the field of events implies, conceptually if not actually, a more personal control to linger, to approach or recede at my own rate.So, by conceiving of events in time as objects in space that I can approach, the metaphor incorporates and encourages the experience of intention.As a solo performer/experiencer in the cognitive space this may or may not have importance.However, in collaborative situations reading intention is highly significant (Corness 2013).Collaborators reading each other's intentions to move towards a time object, or read the quality of other's movement away from a time object, can inform their own actions.This understanding of intention helps build trust within the collaboration (Corness 2013).

Prose vs. verse
Our experience of time and rhythm begins in our bodies, and developmentally expands to speech and movement (Hartley 1989).To explore time in speech we may look to the analysis of meter in poetry.In this context, changes in feet in closed form poetry often indicate points of stress or a shift in thought (Adams 1997).Feet are defined by patterns of short and long syllables.Without a codified unit of counting, these short and long syllables are more about the flow of the text then a direct count.And yet, the meter of the poem still relies on a set pattern that sets up expectations that the text can push against as part of the interaction and experience.Based on this simple understanding of rhythm in poetry, we may suggest that focusing on the relationship between events, the patterns of long and short, and the stress and flow developed through these relationships avoids a "dictation" of count while retaining structure and flow.

Viewpoints of time and gesture
Viewpoints offers an approach to understanding time and rhythm based on relationships of space and movement (Bogart & Landau 2004).The Viewpoints approach to movement training was developed by Mary Overlie as a way of structuring dance improvisation in time and space.Anne Bogart and Tina Landau describe Viewpoints as a philosophy translated into a technique for (1) training performers; (2) building ensemble; and (3) creating movement for stage; as well as points of awareness that a performer or creator makes use of while working (Bogart & Landau 2005).The training includes developing the performer's awareness of physical structures, space, time and other participants.This is done through identifying a set of conceptual perspectives, referred to as Viewpoints, that serve to focus the performer's perception and discussion around a given dramatic interaction or environment.The collection of Viewpoints presented by Bogart and Landau includes: tempo, duration, kinaesthetic response, repetition, shape, gesture, architecture, spatial relationship, and topography.
Three Viewpoints of Time that we focus on here include (Bogart & Landau 2004, p.8):  Tempo is the rate a movement occurs  Duration is how long a movement or sequence occurs or how long before a noticeable change  Kinaesthetic Response is how long it takes to respond and repetition the recurrence (in the scope of the body or the space) The reference to the perception of change and recurrence suggests a focus on flow rather than of count (common in music).As in poetry, the perceiving of change in a pattern/sequence becomes the process of parsing gestures.If we then consider the Viewpoint of Space and Gesture, gesture is the awareness of movement involving a part of the body.The concept of Gesture in Viewpoints includes having a beginning, middle, and end.Relationship then becomes the distance between objects on stage, the moving towards and away from something that can be perceived as having an intentional connection.
By combining the concepts from poetry and Viewpoints we start to put together a concept of gesture in which a there is:  A beginning, middle, and end  A stress that helps parse gestures as defined by a shift/ change in texture in the sequence of events.

Time & gesture in Laban Movement Analysis
To help structure this new concept in understanding movement gestures, we can also incorporate the concept of Phrasing in Laban Movement Analysis (Studd & Cox 2013).Rudolf Laban developed a method of categorization to analyse, notate and create movement.Movement is divided into elements of Body, Effort, Shape, and Space.Phrasing focuses on the location of emphasis within the dynamics of movement.By changing where and how emphasis is used it can change the intention of a phrase (Studd & Cox 2013).
In LMA, there are four types of phrasings:  Impactive: emphasis at the beginning  Impulsive: emphasis at the end  Swing: emphasis in the middle  Even: no emphasis Now that we have explored poetry, Viewpoints, and Laban Movement analysis we will consider different types of gestures from a music perspective.We present a variety of templates for gestures that can then be utilised during the generative process.

Applying Schenkerian concepts
With a framework of gesture now based on flow towards and from time objects defined by stress points or changes in actions, we considered how such a metaphor of rhythm could be applied in music.The prime theory we borrowed from was Schenkerian analysis.Schenkerian analysis suggests that any tonal musical piece can be broken down to three structural levels.While each level illustrates a sequence based on tonal consonance (arpeggios, scaler sequences, etc.) each level also illustrates the figuration and paths between the consonant tones that make up the lower level.Figurations such as Neighbour Tone, Passing Tones, etc. are metaphors of a movement through a tonal space where significant events act as posts that are moved towards, around, and from.Though Schenkerian analysis focuses on the tonal space in music, we suggest that a similar concept can be applied to the temporal space.The concept of time points, moments in time often used to discuss the onset of note, is a similar concept (Palmer;Lerdahl & Jackendoff 1983;Babbitt 1962).
We use the concept of points in time to define goal posts around which rhythmic gestures move.By considering points in time we can begin to physicalize the gesture's goal.By physicalizing the rhythm we hope to imply an intention to the gesture, and even implying a hierarchy to the goal points to provide opportunity for structural levels in a composition.

Orientational time metaphors in music
Much of the language around rhythm indicates the metaphoric expression of our experience.Phrases such as 'Pushing' the beat or 'Pulling' the beat reflect an ontological metaphoric referencing our experience with physical objects (Lakoff & Johnson 1999).Perhaps even more commonly the terms 'on' the beat and 'off' the beat leverage metaphor of  1990).From this perspective a beat is an object that a musician can be ON or PUSH as if it were a stool.A rhythm could be imagined as a line of stools.Leaping from stool to stool a physicalization of being ON the beat.However, we could now imagine pulling the stool closer for a slightly shorter leap, pushing it further away for a longer leap, landing in between stools and continuing with a run up to the stool.By applying a Moving Observer metaphor we can now see rhythm as a dance around points in time.

Musical gestures
The concept of evenly dividing time into pulses, meter and precisely recurring patterns seems to be fundamental to the concept of rhythm in western music.The pulse is what we tap our foot to, it is the basis the meter, the bar and rhythm is based on dividing and combining periods together (Elmsley et al. 2017).This view of rhythm suggests an object-based metaphor were periods are imagined as blocks fitting together to precisely fill spaces of time.The interaction suggested by such a metaphor is a rigorous consideration of the timespace to be filled through an organised plan of fitting the blocks together.While this metaphor may seem to fit with the tenant of music theory does it fit with musical practice?Humans are not perfect timekeepers and performers regularly vary temporal dynamics for expressive affect.This understanding implies a much more tenuous relationship between beat and flow in music and suggests that "any endogenous sense of pulse and meter is always being generated throughout the listening process" (Elmsley et al. 2017).

Initial approaches: Ariel I
We have iterated our development of this model for generating rhythmic gestures using orientational metaphors over two systems.Our first approach to modelling rhythm gestures focused on generating a dynamic flow.The concept was derived from an experiential understanding of gesture as a purposeful motion with a beginning, a direction, and an end.In order to give structure to this experience-based idea, we borrowed from Schenkerian analysis the concept of lower level patterns providing tonal goal points, in around which a high level embellishes in turn providing goal point for even higher level.
In keeping with the idea that time is first experienced through the body, we connected the system's concept of phrase-time with a built in breathing pattern.The system was programed to breathe in order to improvise phrases of flute music (see Figure 1).The duration of the phrase was based on the depth and duration of an inhale breath.The duration of the inhale breath was generated using a Brownian walk coupled with data from the tempo derived for previous gestures and the observed gestures of the dancers.The Brownian walk was used to provide a sense of general breathing so the breath could change from one tempo to another.The tempo of the breathing could progress faster or slower in a smooth fashion.All the streams of data (including data from the dancer's phrase durations) were combined and fed back to model the influence of the musical tempo on the player's breathing.In turn, the derived breath duration was used to generate a new tempo for the musical gesture/phrase creating a loop.The breathing pattern at rest contains the final element of an intermediate hold.The hold is modelled on the slight pause naturally found between the inhalation and exhalation of a breath.
For the system, this hold is a consistent 500ms.
The result is a pattern of breathing in with a given timbre and varying duration, a 500ms pause followed by an exhale with a given timbre and varying length all modelled to mimic a resting breathing pattern.

Rhythm parameters
While the use of the breath provided a basic duration for the musical phrase, individual gestures were generated based on a layered structure.Behind all the design decisions was the concept of moving towards a point.This design element is in aspects such as the breath moving towards the note, and the gesture moving towards an end.One key concept in the design theory is the idea that a gesture has an accent event that other events act as steps either towards or away from.As discussed earlier, an accent event that breaks a pattern, which can be understood as different in duration from the other events.This premise led us to generate rhythmic gestures based on short and long events, where either can act as the accent.By layering this structure at varying resolutions (duration of events) any number of rhythms can be produced.The system generates the rhythmic component of the musical gesture in four stages.First, a proposed phrase duration is generated using random processes influenced by elements in the performance environment (depending on the mapping being used), determining a preferred gesture length.Second, an actual phrases length is determined first by dividing the preferred length by the current tempo to determine the number of 'beats' for the gesture.The actual duration of the phrase is then calculated by multiplying the number of 'beats' by the tempo.Third, the actual duration of the phrase is then used to calculate the total number of possible short events and long events in the phrase by randomly determining the durations of what will be considered a long and a short event.This process of determining long and short events is influenced by the sensing of the performance environment.The sensing of the environment also produces data mapped to the calculation determining the number of short and long subgestures, phrases that make up the whole phrase.Finally, once the durations of the sub-phrases are determined the same process is used to determine the gestures that make up the sub-phrases.The number of gestures and number of sub-phrase are compared.The lower number, gesture or subphrase, for both the short and long are used to control the distribution function for determining the number of gestures in the sub-phrase.This process allows for gestures that are one event long but gives preference to gestures of multiple events.
The 'intentional' aspect of the musical gestures was in part related the relationship between its responsiveness in the performance environment and its inner state.The connection between the system's musical decisions and the performance environment was provided by a computer vision system (see Figure 2).The computer vision system tracked the change of various aspects of the performance environment and performers' movements.Events in the environment could be used to change the weighting of probabilities concerning decisions for constructing the musical gestures through the use of distribution functions.These changing probabilities were combined with preset probabilities.The structure allowed decisions to be influenced by the performance environment while still being perceived as originating in the system.The width and combining of the distribution functions were designed to give the system an autonomous feel since the final decision, most importantly the timing of the next gesture, could not be predicted.

Musical Decisions
This first system attempted to provide a sense of intention to generated musical phrases by implementing embodied cues of breath as well as using metaphors of goal points in the algorithms for generating the musical material.For this reason it was also important that the performer perceive the musical gestures of the system as having an intention within the improvisation.One participant commented: It feels like... ah... it feels musical, as in... what feels good... to come next.It... it's like an... it feels like it makes sense.Such comments suggest that the algorithms used were providing a sense of intention or at lease and adherence to musical expression.However, on further analysis, the study suggested that this sense of intention might have been largely, and somewhat surprisingly, from the presence of the breath.In reassessing the generative algorithms of this first system we reassessed the openness of the rhythm generator.While the process was able to produces gestures that moved in and out of points in a moment, the coherence of rhythm across moments was quite loose.

New approaches: Ariel II
Our first system, Ariel I, focused on creating singular gestures in and out of moments.While these individual moments provide a sense of direction we became interested in if the focus on singular gestures weakens the larger cohesion across multiple gestures that a more traditional musical meter may provide.It is the balance of these two concepts that became the interest in our second version.In our second system, Ariel II, we base our design on a Moving Observer metaphor, physicalizing the rhythm in order to find a common ground with the domain of dance.We currently do not use the breath in this version.We continued to focus on creating gestures that were perceived as having intention to temporal posts.We also worked to create a process that remained responsive at any point focusing on planes that could be changed.Finally, we wanted to add to our first version a stronger adherence to a quantized meter that could be slackened if chosen.
Generating a gesture is processed in several steps that include determining a gesture length and goal time point, the shape of the gesture and the sequencing of the gesture.The steps tend to alternate between proposed data and calculated data.
The first produces a proposed duration of the gesture.This process, as in earlier versions, combines both internal and external data streams such as duration of the dancer's gestures and the duration of previous gestures of the system.Next, the system uses the current position and proposed duration to determine a proposed goal point.The proposed goal point is compared to a database of goal points that include the current meter, rhythm templates and previous gestures.In our implementation, these data points combine to provide a tendency (probability) to choose an actual goal point.For example, meter tendencies such as preference for events on first beat of the bar can be combined with rhythmic templates such as a clave beat pattern to produce preference weights for points in time.This process produces an actual goal point for the gesture to move in, out, or around.When a point in time is chosen as a goal, a new gesture duration in milliseconds can be determined.The new duration provides the longest possible gesture INTO the goal point.To determine the duration of the gesture final gesture preferences such as preferred subdivision, number of events relationship between accent and passing events is determined.Currently we have determined these using simple processes; however, in future iterations we hope to consider additional processes.This process produces duration for passing events, and two different accent events.The relationship between these can have accents being longer, shorter or the same as than the passing events.The events can be used in any order limited to times that are divisible by the duration.In this way the process can generate impulsive, impactive, swing, and even phrased gestures while still giving preference to metric structure.The process also allows the system to vary its preference for adhering to a metric structure.In this way gestures of varying types may be created by focusing on intentional flow to a goal point in time, while referencing metrical structure to provide coherence across gestures.
The process outlined has been used to generate rhythm gestures at the level of sequenced sonic events however the general process is applicable of generated larger sections.These sections can then be subjected to the process for producing the sonic events producing a levelled approach to generating rhythmic structure.

Constructing the model
In looking to construct a system for generating rhythmic gestures we considered how humans experience and describe time.In describing time, we often fall back on metaphors of space (Lakoff & Johnson 1999).These metaphors may focus on such concepts as an observer moving through space, or objects set in the space moving past the observer.These metaphors help us understand the overlap between dance rhythm and musical rhythm.Consider rhythm as a series of objects in space being passed by the observer.The observer can be seen as moving towards, jumping off, or gesturing around the points in time, suggesting a common ground between dance and musical rhythm.The Observer's intention towards the time objects is thus reflected in the metaphor and can be developed as part of the model.

Implementing the model
Implementing this model computationally focused on finding significant points in time for the musical gesture to move towards, out of, or around.Our first implementation considered constructing gestures by subdividing the duration and implying a stress event by having the goal be a different duration then the surrounding events.Although this implementation created individual gestures that implied direction (intention) towards an event, there seemed to be only a weak connection between subsequent gestures.
In response, our second implementation added stages in which the process references a general meter.Although meter can place a dictatorial approach to time it can also provide a general context that provides connection between gestures.Our implementation, by always moving towards points, remains flexible, changeable at anytime by shifting the goal.

Future work
In future iterations we plan to further explore how events can be shifted.Slightly shortening or expanding events approaching or moving away from a goal point we believe we can add more flow and expressivity to the musical gesture.We also plan to work further on layering gestures to explore not only intentional direction in the event gestures, but also in gestures at more structural levels such as phrases and sections.
Finally, while our simple implementation of the model has proved promising, processes used for decision-making within the model provide opportunity for further work.One of the encouraging components of the model is that the implementation of it is open to more refined processes and varied sources of data used for constructing each step.

Figure 1 :
Figure 1: Diagram of process for generating breath duration.

Figure 2 :
Figure 2: Camera vision for the Ariel I system.
Physicalizing Time Through Orientational Metaphors for Generating Rhythmic Gestures Greg Corness & Kristin Carlson 283 spatial orientation.These metaphors are based on our physical experience through our bodies, developed from interacting with the world (Johnson