Augmenting Live Performance Dance through Mobile Technology

We present a pilot study investigating the use of mobile technology to augment live performance dance. An augmented performance was created and viewed through an tablet device then analysed from a technical and audience standpoint. Low complexity augmentations were found to be very effective, however the device placed restrictions on higher complexity augmentations, the performance length and the stage/audience spatial arrangements. A low-number audience test indicated that augmentation of live performance was a credible concept, though there are some clear challenges to be overcome specifically around speed of technology and weight of device.


INTRODUCTION
A recent interim report for the Arts Council (Nesta Digital R&D 2014) found that around a quarter of arts organisations use technology alongside/connected with their artwork, though the performative arts (theatre, dance) are only half as likely to use these as their non-performative counterparts.A 2011 study by Dance/USA (WolfBrown 2011) indicated that "younger audiences are categorically more interested in all forms of technology-based engagement" and that this demographic is the least interested in engaging with the traditional performance paradigm.
Augmented Reality (AR) is now well established in the visual arts (Cukrov 2011;Chayka 2011;Re+Public 2014) and artists are deliberately creating works to be viewed primarily through mobile devices such as phones and tablets.There is some evidence that use of these technologies are helping engage the viewer (Gilroy et al. 2008).Within the performance arts, however, AR has not yet been widely adopted possibly due to the difficulties associated with a multiple-viewpoint paradigm and its dynamic nature leading to high computational demands.Where AR is used within the performative arts it tends to be provided through the addition of pre-recorded video added to the scene when an establish marker is detected, such as within the Dans AR project (Affexity Research 2013).Those who attempt live augmentation typically provide a uniform viewpoint based on a single video stream capture, for example Deakin Motion.Lab and the Australian National Ballet (Vincs et al. 2011).
Live performance is characterised by an audience with slightly varying viewpoints.Additionally each viewpoint shifts independently as the audience chooses which point of stage to focus on.For AR to support such a paradigm it would require an individualised augmentation based on viewpoint.With the ubiquity of mobile devices, this becomes a matter of simply providing software that can interpret the scene from the appropriate angle and providing a correct augmentation.However viewing a performance through a mobile device introduces many questions around the user experience, for example: • Are audience members willing to engage with a live three dimensional performance viewed through a two dimensional screen?
• Can we maintain a kinaesthetic connection with the audience when viewed through a mobile device?
• Are all types of augmentation equally valid within live performance?
With the addition of a mobile device the traditional ideas behind creating and viewing performance need to be reconsidered, the traditional two-way interaction between audience and performance becomes a partially mediated experience: audiencecomputer-performance.Clearly both aspects need to be considered from a technical and user experience point of view.
In this paper we present the our initial findings into what is, to the best of our knowledge, the first attempt to create a personalised augmentation of a live dance performance.

Overview
Augmented Reality, or AR, is the set of techniques whereby synthetic sense data is added congruently to a live, observed reality.It is the movement to combine computed and real information to create a richer experience of both.This can include reactive musical compositions, haptic (touch-experience) sculpture, and visual additions and effects.
Augmented reality is often characterised by the use of a digital camera and a computer to enhance a visual experience.Within the context of a dance performance it makes sense to think of it as an extension of video dance, with certain caveats.
In video dance, one takes source material and performs editing actions on it, in augmented reality one does not modify the source video manually, but describes to the computer the time and manner in which the editing should take place.Further, the editing of the piece is beholden to maintain some semblance of continuity with the real-world dance or the augmentation would lose its connection to the observers real-world experience.
Taking control of a visual experience offers several advantages.We can position objects on the stage with the dancer, and not be restricted by construction costs, stage size, or constraints of reality.Our objects can simulate physical interactions, but are not necessarily bound by the laws of physics.Similarly, special effects become not only possible but limitless; pyrotechnics are perfectly safe and human teleportation becomes possible.Acknowledging the parallels with video dance, it is possible to add any video effect that can be computed in reasonable time.
Clearly a consideration of all possible augmentations is beyond the scope of a single piece of work, however here we focus on the effects that are most likely to have artistic use within all of the performative arts, including:

Visual effects
• full and part-screen washes • other lighting effects, such as fades • video filters, for example cine • temporal adaptions (slowing or stuttering the scene) • spatial adaptions, such as teleportation of performers/real-world objects • videos out of perspective (replacing the entire screen with a pre-recorded video) • videos in perspective (adding pre-recorded video to certain parts of the stage) • adding 3D objects to the stage in known locations • adding 3D objects to the stage in computationally determined positions A physics engine was used for manipulation of the added 3D objects, such as falling under gravity or shattering (Atkin 2009).

Audio effects
The sound-scape of an artistic work is considered integral and by using mobile devices to view a performance we allow for augmentation of a global sound scape with personalised elements.

Haptic effects
Haptic feedback, on the other hand, is not common within performance and was included here to discover whether it had artistic validity within the augmented performance.

Performance
Live augmented dance is a natural combination of live performance and video dance but cannot be treated as an extension of either.The audience is expected to view the performance through a mobile device, however their field of vision is likely to include elements of the stage/performer.Each audience member has a unique viewpoint and the dance must make sense to each of them.
It was considered important to use typical dance elements, including speed and spatial range, to maintain verisimilitude.Dance is, of course, much more dynamic than most other performance types and if the technologies could effectively work within this range then issues should not be found with most other performance types.
In order to reduce complexity an original work (Who We Were, (Golz 2014)) was choreographed using a solo performer.The performance used a typical range of contemporary movements based on the Graham movement vocabulary (Horosko 1991) and used traditional choreographic memes such as a range of levels and speeds.In order to facilitate effects such as teleportation, and to make lighting effects easier, the performance was held in a pure white studio with flat white overhead lighting.

Hardware
Tablet devices (Apple iPad 2) were chosen as the hardware platform through which the audience would augment their view of the performance.
The device provides sufficient graphics performance required to ensure that the augmentations are applied repeatedly on every video frame.If the frame rate drops too low, jittering and latency makes the augmented vision uncomfortable and unrealistic.
It is the consensus in the related field of virtual reality that a truly natural experience requires a video rate of around 120 frames per second (see, for example (Lang 2014)).This is impractical within the constraints of the current project, but the field is developing rapidly and 120Hz video screens are becoming available for portable devices in the near future.In the meantime, we are capturing as much expression as possible within a relatively highlatency system running at 8-12 frames per second.Latency is observable, but is low enough that several effects could be applied simultaneously.
Consideration must also be made for the audience member's interaction with the technology.The Apple iPad 2 weighs around 600g, is slim and easy to hold and easily manoeuvrable.We were expecting the user to hold it in both hands for a period of around ten minutes, regularly moving it to maintain the location of the performer roughly within the centre of the screen.

Software
The purpose of the software (code-named Full Spectrum Dance Software) is to combine computer vision, computer graphics, and the choreographic and artistic experience of the performance.
There are several software frameworks intended for use by digital artists that encompass these requirements such as Processing.org,Cinder or Open-Frameworks.Here OpenFrameworks was chosen because it is fast and well supported with both computer vision and Apple iOS development.
Several of the augmentations require accurate determination of the scene (adding 3D objects, teleportation) and OpenCV (Bradski 2000) was used for this aspect of the work.As each camera has its own view on the scene we were unable to easily use dedicated motion capture devices for scene analysis.
All the software used is licensed under an MIT free open-source license and can be used commercially or shared publicly with no restrictions or additional costs.

Software pipeline
The Full Spectrum app pipeline is: 1.The tablet device receives an image from its camera.
2. OpenCV analyses the image to work out the stage layout and the location of the dancer if required.
3. Objects and effects are drawn over the camera image using OpenGL.
4. The composite image is shown to the audience member holding the tablet.
This pipeline is run once for every frame of video shown, so it is computationally demanding and requires both a sufficiently powerful tablet device and efficient software.

Integration of 3D objects
Adding 3D objects to the real-world stage so they are in perspective and appropriately lit is complex, as there is no simple way for a computer to decipher the perspective of a scene from a 2D image of it.Markers made of decorating tape were placed on the realworld stage at measured positions and sizes.The OpenCV POSIT algorithm was used to calculate a transformation between these points in the 2D image to a known 3D position.

Audience
Due to the nature of the technology there are likely to be limitations of the effective number and positioning of audience members.Given the effective viewing angle/depth of the device and limits on the detection of the markering system, we were interested in what audience/stage set-up would provide the optimal viewing experience.
We identified the extrema positions of the dance and, viewing through the tablet device, identified the best viewing location.Here we considered three factors: • Ability to see the performer in their entirety • Enough of the markering system to be detected to apply an effect • Performer should take up between 10% and 80% of the height of the screen when viewed in landscape mode.
Clearly the last factor is subjective: angular subtends in live theatre are typically significantly smaller than this, however there is an expectation for users of tablets that the point of interest remains of workable height.

Focus group
This pilot study is primarily focused on the technical and artistic considerations of the technology, rather than audience engagement, so a very limited focus group with three audience members (with various experience of live dance) was conducted.Audience members attended a live performance within a dance studio/performance space within the Performance Hub at the University of Wolverhampton.They were provided with a conceptual overview of the performance concept -namely that we were investigating their experience of an augmented reality performance.They were instructed to view the performance through the iPad.No further information was provided on the performance meaning nor the nature of the augmentations they would experience.
Post-performance, audience members were led in a group discussion to discuss their experience in their own way.The authors did not steer the discussion towards particular issues, rather to discover what issues seemed pertinent to the audience.Clarification was occasionally sought where meaning was not clear.
The conversation was recorded and analysed for relevance to the research questions.

RESULTS
An augmented reality performance with individualised viewpoints as viewed through mobile devices was successfully created though several technical and audience related issues were discovered.Augmentations fell into one of four categories: • Full screen effects (including temporal manipulation) • Local effects whose location had been predetermined • Local effects whose location had to be determined through computer vision/computation.
• Non-visual effects For some local effects, such as applying a video filter to part of the stage, could be applied at a known stage location (see Figure 2) and simply had to be converted to screen position based on the marker's location, whereas effects relative to the performer such as teleportation required computer vision to analyse the scene.
All effects were triggered through a master computer and fed to the tablets via local WiFi, rather than triggering at a known time, or computationally determined.Each tablet was, however, responsible for calculation of the full effect.This system worked well and there was no noticeable delay between triggering the effect and the effect reaching the tablet each device responded effectively simultaneously indicating similar processing times.

Performance
The software/hardware successfully displayed an augmented reality performance on each of the audience's tablet.Full screen augmentations such as full screen washes (Figure 2), lighting and video filters were easy to produce, caused no noticeable lag and worked well artistically.However augmentations that required use of OpenCV/POSIT (placed 3D objects as in Figure 4 and videos in perspective) showed noticeable lag due to the relative intensity of the calculation.Teleportation, on the other hand, which required use of OpenCV without POSIT worked without noticeable delay.From this we conclude that the POSIT calculation is too computationally intensive to be carried out within a reasonable frame rate with current technology.

. Non-visual effects
The tablet was made to produce pre-determined sounds, as triggered from the master computer.
In our choreography whispered conversation was played locally over a global soundtrack and was felt to be, subjectively and based on audience feedback, a dramatic effect.Haptic effects worked in a similar way but seemed less effective and felt subjectively out of place with the other augmentations.

Marker Occlusion
Occlusion of the markering system by the dancer was a significant issue.Known stage locations are sited in space by reference to these systems and these need to be constantly refreshed due to potential movement of the viewpoint.When the dancer occluded the markering system the 3D object started to drift (Figure 5) A way to resolve this might be to chose a markering system in front of the stage rather than at the rear.The original stage size was 6m deep by 7.25m wide (see Figure 6) with two 1m square markers centred width-wise.One was sited on the floor starting 1m from the rear of the stage, the other on the rear wall with its base 1m from the ground.Based on the image size requirements stated earlier (and using no zoom) the audience tablet needed to be around 2m from the front of the stage.

Location
Given the tablets field of view of around 44 • (which is also incidentally about the limit of accuracy for the POSIT algorithm) this results in an available audience width of 2-3m, allowing for around 4-6 audience members.This is quite a restrictive audience count.

Focus group
The work has been performed several times though always with very low audience counts, but the following feedback has been consistent across audiences.Quoted words refer to audience members' description of their experience • It was not "odd" or unusual watching a performance through a mobile device.
• The augmentations were "cool" and "supported" the work.
• There was some kinaesthetic connection audience members felt subjectively different from watching a dance on an tablet/PC outside of the context of a live performance.
• The lag seriously affected those aspects of the performance.Any lag was deemed unacceptable.
• Many commented that they "wouldn't want to watch a whole performance through it".On clarification this was due to the weight of the device.No-one mentioned that the weight of the device was an issue within the actual performance (ten minutes in length).

CONCLUSIONS
Augmentation of live performance is still in its infancy but our work demonstrates that it is technically feasible using commonplace mobile devices, no dedicated hardware is required.A relatively inexpensive tablet device can currently be used by small audiences to augment short (less than ten minutes) works and, while intense computation remains an issue affecting placement of virtual objects on the stage, there is a wide range of augmentations that can currently be deployed to great artistic effect.
Feedback from limited trials reveals this augmentation supported and added to the artistic work.
Although more substantial trials are necessary to develop a deeper understanding of the nature of engagement, early indications are that there are audiences willing to engage with this new performance paradigm and that augmentation of live performance is a credible concept.
The next stage in the research will utilise more powerful tablet devices which will overcome many of the computational challenges addressed through this pilot study.We will explore various adaptions to our markering system to overcome placement issues and indeed the field-of-vision limitations of the device, which would open up the work to larger audiences.
The most significant change likely to affect this work is the arrival of smart glasses such as the SpaceGlasses 1 or similar.It is not unreasonable to expect audiences to wear a pair of AR enhanced glasses for a much longer performance, as they currently do for 3D cinema.Combining glasses with the research described in this paper should result in a dramatic and engaging new way of watching live performance.

Figure 1 :
Figure 1: The markering system used by the Full Spectrum Dance Software

Figure 4 :
Figure 4: Adding 3D objects at known locations