Oncheon: Learning to perform motion trails for virtual spaces

This paper focuses on a set of motion capture processes that were choreographed and performed by Eugenia S. Kim under the direction of Kim Vincs. The overall process includes a brief overview of the Centre for Transformative Media Technology’s general approach to motion capture recording and a benchmark testing session run by John McCormick which served as a seed for the project. A major finding of the process was the importance of artistic and technical coaching for motion capture performers in helping to develop a personal movement style.


INTRODUCTION
Humans have a continuous fascination with the body and its capacity for movement. In the realm of animation, the priorities range from perfect replication of subtle human gestures to the exaggerated caricature of acrobatic feats. In the realm of live dance performance, there is an increased emphasis on the simultaneous presence of emotional artistry, physical virtuosity and perfect geometrical lines. With the advent of digital technologies such as motion capture (aka mocap) and virtual reality (VR), these two realms come together in a way that facilitates the transcendence of bodily limits. This transcendence allows for a form of expression where motion becomes the focal point and the essence of human energies can be retained even without a physical vessel present.
Oncheon is a meditative VR work-in-progress that seeks to highlight the purity of motion and human movement by re-mapping the human essence to animated non-humanoid elements. Choreographically, it is a reinterpretation of the Korean traditional dance known as salpuri. This dance has evolved significantly over time from a shamanic ritual into an artistic performance. The main elements that are retained are the sense of releasing or cleansing negative energies. For this reinterpretation, contemporary and somatic dance movement practices were combined to create a more effective way of generating motion trails that would be hard to animate without a human performer but without necessarily retaining a humanoid shape. Conceptually, Oncheon is based on the ritual of bathing in hot mineral waters with the intended effect on the viewer to be that of relaxation and healing.
This project is a multi-year collaboration between Leonardo21, the Centre for Transformative Media Technologies (CTMT) at Swinburne University of Technology and MetaObjects. The first stage, choreography and motion capture recording, was completed in August 2019 and was carried out at CTMT. During this phase, several conceptual ideas such as pre-acceleration were explored through the choreographic and performance process. In the subsequent sections we explain the roots for the movement approach, the unique attributes of the CTMT that make them ideal for this project, our findings from the collaboration process and future directions.

BACKGROUND
The roots of this project lie in the previous research and creative work carried out by dance scholars such as Kim Vincs, Ruth Gibson, Stephanie Hutchison and Sarah Whatley. There are several fundamental themes running across their findings that are not necessarily present in literature on motion capture acting or use of motion capture for animation. These themes include: • The use of one or more somatic movement practices during recording (Hutchison & Vincs 2013;Kozel, Gibson & Martelli 2018;Whatley 2012). • The translation of 'internal' qualities in somatic movement practices such as energy, touch and breath into digital data and subsequently a virtual body (Kozel 2007;Whatley 2012). • The extension of the human body through motion capture and data visualisation (Hutchison & Vincs 2013;Kozel 2007;Vincs 2016). • The additional freedoms granted to the human body through motion capture and data visualisation (Hutchison & Vincs 2013;Kozel 2007;Vincs 2011Vincs , 2016. To understand the value and difficulty of realising these themes, it is easiest to start by looking at the challenges faced by a professional dance performer of any style. Performers are traditionally expected to conform to a certain aesthetic, whether it be through their physicality, ability or artistic expression (Reed 1998). In the 21 st century, there is a dilemma between being more accepting of non-traditional representations and individual limitations (Charnley 2011) while simultaneously pushing the boundaries of human performance and physique (Benthaus 2015). To that end there is a greater intersection of dance with science, technology and medicine in order to better utilise and understand the body from an external perspective.
In terms of internal perspective, however, somatic practitioners have already been engaged in selfawareness and examination of bodily mechanics for extending movement potential (Batson 2009). The connection of these practices to dance is apparent through the integration of existing practices into choreography and teaching and dance artists pioneering their own somatic practices (Eddy 2009). For most commercial motion capture animations, the result is usually a faithful approximation of humans as we know it or a means by which to make a non-humanoid living creature move more fluidly. In a project such as Oncheon, however, the emphasis is on capturing the energy of human motion for a potentially healing effect. This makes somatic movement practices ideal for not only the motion capture performance but the overall concept.

EMBODIED MOVEMENT DESIGN STUDIO FACILITIES AND WORKFLOW
The capture sessions for this project took place at the Embodied Movement Design Studio at Swinburne University of Technology in Melbourne, Australia. EMD is a specialist facility within Swinburne's CTMT dedicated to creative technology research into movement-based interactive systems.
The EMD Studio has a 16 camera Optitrack motion capture system using Prime 17 cameras. The studio has a capture volume of 8 metres x 14 metres and a sprung dance floor. This makes it ideal for the capture of a range of movement activities. The Optitrack system is the main motion capture system used although the HTC Vive and Perception Neuron (PN) are also used for tracking. The Optitrack is the most accurate system and can capture multiple performers.
Use of the Optitrack system is typical of most optical motion capture systems. The performer wears a suit made of Lycra and Velcro to which round silver-coloured reflective markers are attached. There are a large number of marker sets available targeted for animation or biomechanics. The cameras have built in infra-red lights that illuminate the markers and record the reflections enabling the system to triangulate the position of the markers. The system can then create a skeleton model based on the position of the markers. Additionally, the system is able to visualise the performer's markers in real-time in a number of ways, as individual markers, skeleton or humanoid figure.
Projects in the EMD Studio typically involve the visualisation of the motion capture data, either in real-time or in post-production. The motion capture software has rudimentary visualisation such as trails however further visualisation can be performed in software platforms such as Unity, Unreal Engine or Maya. This can occur in real-time or using recorded data. The performances are captured and exported in a number of formats depending on the final use case such as 3D animation, biomechanical analysis, data visualisation or character development.

APRIL BENCHMARK TESTING SESSION
A key event that led to this collaboration was Eugenia Kim's participation in a benchmark testing session as part of a study to compare the Perception Neuron (PN) and Optitrack systems. This was as a potential precursor to a full validation study for using the PN in a medical research project. This session was directed by John McCormick and included the involvement of two more researchers, one who was doubling as the session technician. During this session, Eugenia was invited to perform a series of tasks using dance movement while wearing both the PN and a marker set for use with Optitrack cameras.
The session used a 37 marker set for the Optitrack system, which was similar to the number of joints and degrees of freedom as the PN system. The PN employed 18 Inertial Measurement Units (IMU). Fingers were not captured for either system though both have some capability for finger capture. The marker set was also designed to take into account the limited area on the dancer's body for marker placements. Both systems require the markers or sensors to be placed on pre-determined places on the body as the systems have an internal model of the human body. The initial takes consisted of classical ballet and jazz dance phrases that were very upright and pose-based. After the researchers suggested more specific tasks to Eugenia, she started to improvise more and rely less on the real-time projection of an avatar animated by her movements. Towards the end, there was one particular take that was surprising for everyone. The movements in that take were based on the idea of a cat playing with a ball. The body stayed generally low to the ground, there were more subtle hand and foot gestures, and the quality was generally more fluid. It was this take that would become the seed for the Oncheon recording session.

Performer's perspective (Eugenia Kim)
There were several aspects of this experience that were new for me. The first was of course wearing two different suits at the same time. Rather than creating additional restriction on my movement, this made me think about and execute movement in two different ways simultaneously. The result was that I ended up trying more ways of moving than I might normally.
Another interesting challenge was to push the limits of the system. I had become so used to working within the specific constraints of a hardware system that this challenge became very liberating. Of course, if I had not already been aware of existing issues such as occlusion, markers falling off, etc., then I would not have been able to really explore all the possibilities. It was definitely a case of knowing the rules of a system first in order to bend and exploit them.
Finally, I really appreciated having so many people observing and providing input into my movement choices. Although I am accustomed to creating solo work on myself, I still rely heavily on video recording rehearsals and mental imagery of my body built up over time. With motion capture, I found that what my body looks like and how it becomes represented digitally differs in a way that I still cannot anticipate ahead of time. As a relative novice compared to experts like Kim (Vincs) and John, I found that receiving direction from someone familiar with both the technology and the habits of dancers was extremely helpful.

EMD studio perspective (John McCormick)
Having an articulate, trained dancer markedly changed the movement available to test the systems. Without the dancer, the movement would have been more utilitarian, testing basic, pedestrian movements and joint angles. With Eugenia's involvement, the movement became a far greater challenge for the two systems to reliably and accurately track.
The take involving the concept of the cat playing with a ball was also one of the most challenging for the systems as the shape of the body became less recognisably bipedal human. The extreme lengthening and flexion of limbs and the rapidly changing relationship to the floor was an extreme test for the capture systems. The systems handled the movement surprisingly well with the Optitrack showing more accuracy but both systems capturing movement that represented the movement well in terms of animation. This could be seen as an extreme test of the systems and simpler, everyday movements should be able to be handled with comparative ease.

AUGUST RECORDING SESSION
In June 2019, Eugenia approached the EMD Studio about whether it would be possible to do the motion capture for Oncheon at their facility in August 2019. Two half-day sessions were scheduled in the same week, one for rehearsal and one for final recording.
The sessions were directed by Kim Vincs with Casey Dalbo, Casey Richardson and Stephen Jeal running the Optitrack system. John briefly observed the second session.
Choreography was created in a mirrorless studio over a series of four rehearsals in one week. Mirrors were purposely avoided to help reduce reliance on external feedback. Three more rehearsals took place between the motion capture sessions to make changes for the final recording. The final structure was 50% set choreography and 50% improvisation. For both of the motion capture sessions, a custom 56 marker set was used. This set included specially sized markers for the fingers. The first session was largely meant to help identify problem areas and experiment with movement while the second was for recording the finalised choreography. There were a total of 10 takes with 5 takes per session. Each session started out with 2-3 takes of the set choreography alone followed by 2-3 takes of the full piece. For the final take, an experimental 80 marker set was used to see whether there was any impact on the motion trails.

Performer's perspective (Eugenia Kim)
The process for the August recording sessions was based on my experiences working with digital production studio MetaObjects and from the April testing session. It was very important for me to have a motion capture rehearsal session with time to adjust choreography, as I would not be able to create and choreograph beforehand in a motion capture lab. I also decided to shed my previous choreographic approach of focusing on what an external viewer might see in terms of poses and angles. Instead I wanted to see how elements like breath and motion would form the shapes. This exploration was inspired by exercises from somatic practices such as Contact Improvisation and tai chi.
What I had not accounted for was the intensity of "performing" during recording. I had some experience of this intensity in previous motion capture sessions and in shoots for dance films. In this instance, however, because the content that I wanted to show had to was obscured by layers of cloth, flesh and bone, each take required a higher level of concentration, effort and control than I might engage in for a stage performance.
During each take, I found that it was too distracting to "dance" with my avatar as Hutchison had described doing in her own process (Hutchison & Vincs 2013). Quite often I danced almost the entire time with my eyes closed. In retrospect, this was likely part of trying to dig inwards and pull out deeply buried energies. I was also in the midst of a series of fibromyalgia flare ups, so I had to dance around a constant flow of pain. In this sense, taking a somatic approach really helped because the movement could provide a healing effect.
After the last take, while watching the playback with only the motion trails turned on, I was able to actually see how the body 'pre-accelerates. ' Manning defines pre-acceleration as 'a way of thinking the incipiency of movement, the ways in which the body is always on the verge of expression (Manning 2009).' This was something that I was not aware of watching traditional video recordings. But for some reason, motion capture visualisations seemed to foreground the moment before, as well as the moment of, movement. Now that I have seen how pre-acceleration can add to the visual impact of the final animation, it has become a potential choreographic and performance element to consider for future projects.

EMD studio perspective (Kim Vincs)
The process we used in the motion capture session was to successively capture and watch play back of the motion capture data. We displayed the motion capture data in different ways, e.g. as a humanoid figure, a series of lines designating body segments, and as moving data points with and without linear trails (Figure 3). This flexibility in how we 'saw' the body enabled a sense of agency and interaction in relation to the capture process. Rather than simply capturing performances and then choosing the best, we were able to work iteratively to focus on the somatic and movement information we saw in the different visualisations.
This process opened up a choreographic dialogue for Eugenia, John and I, as we were able to discuss and share our impressions of the movement and our ideas for its future visualisation. This, for me, is one of the key affordances of motion capture, and one of its key advantages over video. While technology can be viewed as having an isolating effect, in this process the opposite was the case. Gathering, talking, moving around and comparing visualisations of the data became a shared and iterative process that informed Eugenia's successive movement experiments within different takes. As motion capture becomes lower cost and hence more accessible, this kind of 'somatic' creative flexibility seems to me to offer new approaches to how virtual dancing bodies are configured and created.

DISCOVERIES AND STRATEGIES
Most of the discoveries and strategies used during these motion capture sessions are not new for veteran motion capture performers. For novices, however, there is a lack of how-to guides on motion capture dance that are widely available as of early 2020. It is quite possible that there will be more resources and training available in the future.
The most significant change for Eugenia as a performer was shifting her mind-set from presenting the physical body to visualising the virtual body. In one week, her relationship to the body went through the following three stages: Accentuating the body. This initial stage was influenced by a combination of classical dance techniques (i.e. ballet, modern, jazz) and choreographing for venues where the body is always seen (i.e. proscenium stages, film and video). There is an emphasis on creating poses, transitions and lines that present the body in an aesthetically pleasing or visually intriguing way.
(ii) Feeling the body. In this stage, a more somatic approach is used to explore how elements such as breath and energy generate movement. More attention was paid to the internal state of Eugenia's body, initiating movement from that internal place and flowing organically. Tai chi was a major influence at this point.

(iii)
Obliterating the body. For Oncheon, the motion capture data will not be animating a humanoid nor other animal-like shape. In order to explore the fullest potential range of movements, it was necessary to imagine that there was no physical shape to the body while performing. Concepts that helped eliminate the human body shape during playback were constant acceleration and de-acceleration of areas of the body, initiating movement from specific markers and opposing directionality of limbs.
One vital factor in making this transition was having artistic and technical feedback from experienced individuals. Because the choreography was practised without the motion capture suit and markers, Eugenia had to adjust her execution of movements in real time. By having someone else analyse whether her movements were effectively translating into the digital realm, she was able to focus on her performance.
Since the ultimate goal of the project is to present the movement in a VR setting, all facings (e.g. front, back, top, bottom, etc.) and distances became potential viewing points. Staying in a static pose either meant that it needed to be interesting from all angles or risk having a "bad" side. Additionally, even though the movements were internally motivated, the initiation had to be mentally mapped to an actual marker on the body. Two previously assumed points were also challenged during the sessions. The first was that certain positions or movements would lead to automatic occlusion. In fact, the motion capture system was quite robust in maintaining a templated relationship between markers. It was not necessary to limit movement possibilities for the sake of the system. Rather, the focus needed to be on the movement itself, paying close attention to performative and somatic aspects that were, in effect, magnified in the capture data.
The second point was that adding more markers would lead to better capture of subtle articulation in places such as the spine. Since optical motion capture systems are designed to capture joint motion, adding additional markers did not necessarily add to the quality of the result. Both of these assumptions, which stem from the dancer's perspective, were found to not be automatically true.
To summarise, the key points that emerged from the sessions were: • It is very helpful as a performer to have separate artistic and technical direction. • Occlusion can sometimes be overcome under ideal circumstances. • More markers do not necessarily lead to more accurate motion capture data. • Performance technique for optimal presentation of the body is not the same as for generating interesting motion trails.
• Performing for motion trails requires changing movement initiation and intention to be more point-based. • Using an internally motivated movement approach that draws upon somatic practices can help a performer focus on utilising their fullest range of movements, playing with energies and continuous movement even when staying still.
The last point was particularly useful when generating movement that will not necessarily be used for anthropomorphic animations.

NEXT STEPS
Going forward the motion capture data from these sessions will be used to create an animated VR experience for meditative purposes. An augmented reality version is also being planned for viewing in an outdoor setting. Additional projects include doing a motion analysis of the data, representing the data using sound or other media and creating data sets for use by others. The strategies and knowledge gathered from these motion capture sessions will also be experimented and expanded using non-optical systems.