Live and Life in Virtual Theatre: Adapting traditional theatre processes to engage creatives in digital immersive technologies

The Round is an augmented reality platform that allows users at home to watch live streamed 3D characters being produced by real actors in a motion capture suite. The project holds the idea of “liveness” in high regard and considers it an important part of the theatrical experience that needed to be preserved in the creation of an at-home, digital experience. The live theatre experience in itself is often enormously technical with pre-recorded visual and audio experiences that are designed to be repeated. We ask the question, what is live theatre and how does this map to live streamed digital experiences, particularly involving motion capture? Whilst liveness, in this context, refers to the experience of the audience and their relationship with the performer, there is another aspect of live that affects the process of creating and designing a piece of digital theatre. The ability to affect and determine results from “live” changes to ideas on stage from the theatrical design team became an important part of the iteration of the production itself. This is part of the traditional theatre making process which doesn’t easily map into the limitations of digital immersive content creation. This project however attempted to rectify that by introducing innovative technologies that allowed existing theatrical control systems and the skills used to run them to operate Unreal Engine 4 game engine, live for the show. By retaining traditional theatrical equipment and skills, the entire production team who were all new to delivering theatre in a virtual medium, were able to adapt their existing practices and approach to communication in order to maintain fast design iteration and effective collaboration within an entirely new digital medium. This paper looks at the creation of the first prototype of The Round and the effect of using traditional theatrical technologies and processes in combination with real-time platforms that are trying to adapt to the live theatre model. Whilst trying to understand the role of the live design process in this context, the subject of “liveness” will be discussed from an audience point of view and the necessity of “live” with or without “life” in the medium the audience is watching.


INTRODUCTION
Theatre is a craft where the creation of illusions and simulations can take audiences into an imaginary world, where an actors portrayal of a character makes them a different person, and we allow ourselves to be transported into the world that has been created for us, even though our intelligence tells us that, in spite of reality being all around us, this is not real.
Theatre is an established industry with pre-defined notions of best practice which allows it to quickly iterate and produce creative works on a quick schedule and often with little money.(Long, Wing and McCann 2007;Radosavljević 2013) This is made possible by the established working practices of an industry which, through no formal mechanism, has established standardised processes of working.
The Digital Immersive industry, which covers mediums including Virtual, Augmented and Mixed reality content as well as real-time, interactive and digital experiences, is emerging and the practice of making content in this new field is still being defined by creatives and technologists.
There is not common agreement on which existing creative discipline this emerging industry is most like, but theatre professionals certainly feel that they are well suited to a medium which relies on audience agency, audience togetherness and a sense of "liveness" within the performance.
The two notions being presented in this paper are that the existing practice of theatre professionals is transferrable to the Digital Immersive sector and that the experience of an audience watching a digital or virtual theatre experience is more engaged if they believe it is "live".The questions this paper will introduce for the future, is whether it is necessary for the audience to know if what they are seeing is live, or if they only have to believe it.
The project that this paper is responding to is "The Round" which is an idea funded by Innovate UK to develop a live streamed motion capture performance to audiences at home via an AR capable device.The project is made up of a consortium led by Reality Check Productions and includes the authors company, Copper Candle, their academic institution, Rose Bruford College and a technology partner, Condense Reality.(Fellingham, 2020;Rose Bruford 2021;Simpson, 2021;Winsor & Brignall 2021)

EXISTING THEATRICAL PROCESS
The range of theatrical sub-sectors is exhaustive and as Schechner (Schechner 2017) has attempted to classify for us, nearly every production is different and has distinct processes within each of them.
Directors such as Katie Mitchell (2009) and designers such as Michael Pavelka (Pavelka & Chitty 2015) have shared their creative process for emerging artists to learn from them, and combined with the established pedagogy of theatre training establishments, the industry produces individual creatives who have learned to operate inside the established practices that are inherent in professional practice.
This has largely been guided by the use of published works and written philosophies of leaders in the field in our recent history as a template for many types of theatre including Artaud, Brecht, Stanislavsky, Piscator, Appia and Beckett to name a few.
Although these resources exist to guide professional practice, individuals can still be creative and their imaginations are as unique as they are, but they must conform through their process to the system they are participating inthat being the conventions of theatre craft.This is most evident within institutions such as the Royal Opera House where I worked for over a decade and saw first-hand how design teams were forced to conform to the limitations of the venue's operational requirements, be it the schedule, the budget or access to artists.Every production was creatively different, but directors and their design teams had to conform to a pattern of work that was deliberately efficient and uncompromising in order to maintain its business of making and selling productions.
Process is therefore entirely distinct within institutions and artistic sectors but when viewed as an industry outsider (comparing it for instance to other creative industries such as film, games and TV) is systemised with conventions that are designed to make the process efficient.(Baker 2013) has collected the points of view of a range of professionals in their roles which reveals some of the theatrical conventions that are commonplace.Some of these practices verge on the superstitious, such as the rule to not whistle on stage, but has a practical reason; the "flyman" who operates the movement of the scenery is a long way from the stage and can't always hear the communication, so a whistle is occasionally used to communicate when to start and stop moving the scenery.To avoid confusing the flyman, whistling was banned to prevent unnecessary risk from scenery being flown in.

Theatrical conventions
Another example of this is to never say the word "go" because it is used to verbally trigger lighting, sound, movement and effects cues.Having it said accidentally in conversation near to an operator, who is nervously poised to fire a live pyrotechnic (firework) could result in an injury.
In a more practical way, the language and terminology used by anyone working in the industry serves to help unify the creative team and performers whilst making and running theatre.
It is taken for granted with the profession that "Stage Left" for someone looking at the stage from the audiences point of view is actually to their right.To some this needs explaining as the directions are always given from the actors point of view, but within the industry, this is assumed.This also applies to the scheduling of design and workshop time which is fairly common across the industry, with rehearsals starting in advance of the stage time, moving into the venue with a "get-in", following by technical rehearsals (establishing the look and movement of scenic and visual elements of the show) and then into rehearsals with the cast, followed by dress rehearsals and eventually the opening night which in some cases includes one or many preview performances.
What will become clearer in section 3 of this paper, is how, during the creation of a production of Heavenly Bodies by Tallulah Brown (Brown 2020), the existing practice of a team of theatre professionals was often the solution to a technical or creative challenge within the digital and virtual creative content.

THE ROUND
The Round is a platform for the delivery of digital immersive content to an audience at-home using an AR experience on a smart phone.The audiences experience will involve logging into The Round app, buying tickets for an upcoming performance and watching it live as an AR experience on your table-top at home.This is produced at a motion capture studio where a live performer would be converted into a digital avatar and streamed to end user devices.This happens alongside digital 3D assets controlled via stage lighting technology into an Unreal Engine game server architecture in the same way a multiplayer computer game would.The research for this project focuses on a prototype production created during the heart of the pandemic.A section of the production Heavenly bodies by Tallulah Brown (Brown 2020) was chosen because the narrative supported the meta nature of the technology, having people within digital layers of themselves.It also supported the virtues of the virtual medium by promoting diversity, accessibility and inclusion in the live performances sector.
The technology means that artistic works are more accessible to emerging artists and particularly allows actors and designers from the black and brown global majority to be more representative in a sector that has a white bias in it's artists and audiences.(Collins-Hughes, Paulson & Tillet 2020) The production itself is about the life and discoveries of female astronomers who have been removed from history by the patriarchy.To support the authors message, this production focused on the Egyptian scientist, Hypatia, who was played by a black female artist.
The prototype proved the technology pipeline was viable for audience viewing, but also identified a new way of working that allows professional theatre practitioners to use their existing method of show making in this new digital medium by introducing tools that supported them.This became known on the project as the "what would we do normally?"challenge.This challenge was thrown up by the team whenever we struggled to find a practical solution to a creative problem and the answer usually came once we reverted to "traditional theatre" and thought about how we would have solved it if the lights, performer and scenery were real.

XR29lighting desk workflows in real-time
Working alongside the motion capture equipment, is an innovative implementation of theatre lighting desk technology to allow the operation and control of real-time content in Unreal Engine.The principle behind the idea, is that animating content (the three dimensional manipulation of digital assets which are programmed into a repeatable loop) using the built in timeline tools in 3D engines is slower and less efficient than the technology used to control and manipulate lights in the theatre and live events industry.
A moving light for theatre use can have up to 50 individual parameters to move and control (as opposed to a 3D element which typically has 9, the XYZ locations of position, rotation and scale).(Cadena 2013) A theatre lighting desk can allow one person to simultaneously operate hundreds of lights, live, in response to the music, action or dialogue they are hearing.It has been carefully developed using the theatrical conventions the industry is familiar with to enable fast and efficient programming of data by nesting looks and effects into "pallets" which are in turn nested into "cues" or "playbacks".
The nearest equivalent example would be an electronic keyboard which has pre-defined and recorded chord structures and musical sequences recorded to each individual key.By preparing these recordings in advance, a single operator could play complex musical sequences that would be impossible for a human to reproduce as individual tones.
When used to control a game engine scene such as The Round, the equivalent of 3 days of work for a single animator can be condensed into hours or even minutes using the XR29 lighting desk plugin.
It also has the benefit of providing a recognisable technology interface for the theatre design team.They are able to think in terms of cue structures, timelines, amendments and playback in the same way as they do in theatre practice.This was evident when the director of the piece decided to remove a section of the play during rehearsals but after the animation had already been programmed.A complicated sequence of 3D asset movements was designed into this moment that would have been difficult to extract from a 3D keyframed timeline without significant time and effort to fix broken animation paths.Using the lighting desks process, the director could suggest that we remove so many lines of text and the operator could interpret that to mean deleting cues 9 through to 12 and in this instance so that it blended into the scene once more.The lighting desk automatically re-tracks the animation sequences so that they blend together neatly despite the cut, which all happens instantly so the director can review the change and approve it or revert to the original.

Live in practice
The USP of The Round is that it is a "live" experience, with the audience watching a character on their phones that they know to be simultaneously performing somewhere else at that moment.
One of the questions that has emerged during the development of this product, is the necessity of the experience being live when the audience isn't colocated with the performer.How do they know it is live?What mechanism provides evidence to them that the show is live and why is that important?In section 5 we will examine the nature of live theatre currently and attempt to isolate the component of liveness that makes theatre a compelling artistic medium and social experience.
Liveness also benefits the creative process, and in section 4 we examine how The Round was able to use this iterative workflow to maintain design iteration in a way that is common to traditional theatre practice.

DIGITAL IMMERSIVE PRACTICE
The pre-visualisation system at the Royal Opera House taught attempted to adapt the 3D digital model to always work as the "real world" would operate.(Simpson 2018) This meant that not only did this speed up the process, but it also revealed errors in the real-world design of the scenery that fulfilled the benefits of using pre-visualisation.This may have involved the complicated manoeuvring of large objects around other structures and objects which was difficult to achieve in the dark during a scene change, or perhaps there wasn't enough room off-stage to store scenery at particular moments in the show.Simply switching the scenery off, scene by scene, hid these issues from the design team so they couldn't be rectified.
The discovery here was that by simulating digital practice as you would in the real world, it provided real world solutions.In the virtual, limitations are removed and designers are encouraged to explore all of the potentials of the virtual space, but during moments of creative or technical problem solving, reverting to the challenge "what do we do normally?"almost always provided the answer.

Digital entrances for characters
An example of the conversion of our traditional practice for The Round came when we were discussing how to make a character appear or disappear in the digital space to make entrances and exits.The motion capture character is always on, driven by the performer whilst they are inside the motion capture volume.If a performer walks out of that space, the computer fails to track them and leaves the digital character in their last known position which is usually mid-walk at the edge of the space.
In a game or a pre-recorded use of motion capture, the entrance and exit of a character is hidden by the camera shots or by activating or deactivating the entire character when they are out of shot.For live theatre, there needs to be a way for the character to leave the space on demand.Unfortunately, for various technical reasons, the characters can't be simply turned on and off in the space.
In theatre, directors are always trying to find new ways of bringing characters into the stage space in original ways but the majority of entrances are managed by having a character enter from behind scenery or by being revealed by light transitions.When we applied the challenge "what would we normally do" to the project, we realised something we always knew, that a character in darkness is invisible to the digital audience and they can be revealed by fading in a light and using scenery objects to hide their positions when they are not in use.Black boxes can hide characters as they walk into them and even hiding under the floor is an option if the real floor is lower than the digital floor.
Many technical challenges were overcome by introducing a game developers solution and as the technology is built on their platform, the game developer instinct is the first you turn to because they are in charge of delivering a technical solution.The learning here therefore, is that the theatre professionals with no experience of game technology should have confidence in their own experiences and the relevance of their processes in an alternative medium.It also demonstrates the opportunities that a convergence of gaming and theatre have to create solutions to issues that are unique to this new medium.

Live control
In section 3.1 I described the role XR29 has on The Round and here it is relevant to share the role it has to play in the conversion of practice from traditional to digital.
Although the lighting control equipment we use is digital, it's core market is for operating and controlling digital lights in an analogue performance with live actors.Also, in the last 15 years, the "media server" has become a core technology for theatre and live arts(Digital Media, Projection Design, and Technology for Theatre -Alex Oliszewski, Daniel Fine, Daniel Roth -Google Books, 2018).Although it can be controlled through it's own user interface, it is more common to operate and control it via a lighting desk.The principal of the media server is that it allows the advanced control and playback of video content which can be combined into layers or mapped onto surfaces.All of this requires control and timings to manage the repetitive playback of video content on a show, in time with the performance, the music and other theatrical changes.This is why a lighting desk is used to control the content as it allows the parameters of a media server to be isolated to individual channels, stored as "looks" and recorded into "cues" that can be played back, moved and re-timed as the show changes and adapts.
Because lighting desk technology has evolved over the last decade to accommodate this new way of working, it is ideally suited to the task of operating and controlling real-time game engines using the same parameter-based interface.Lighting desks allowed the complexities of media servers to be made more accessible to the vast range of talented lighting desk operators in the industry, bringing their talent for theatre lighting processes to bear without needing a vast re-training of skills.

Game engine technology, particularly Unreal
Engine, is a vastly different set of skills for most theatre professionals, and the type of person who is brilliant in theatre because they are creative and good at managing, is often not the type of person who would be accomplished at learning how to code and build game mechanics.
The lighting desk being used as an interface for the game engine becomes a translation tool that works on several levels. It allows skilled theatre professionals to operate the system without requiring upskilling. Teams of theatre professionals can continue to work using their own practice and experience, to maintain their natural collaboration as their bubble of skills will include someone who can operate the game engine via the lighting desk as a translation tool. Integrating live, analogue theatre with game engine technology becomes frictionless as the same interface can be used for both.This allows the design team to consider the game engine aspect of their craft into their project regardless of whether it is being delivered as part of the live production or as a simultaneous digital experience for audiences elsewhere.

LIVENESS
If an audience is watching digital content on their phone at home, what mechanism do they have to identify that it is live and not pre-recorded?Why does it matter to them if it is live?
During the pandemic, consumption of pre-recorded content through TV, films and even theatre experiences was high as it was one of the only forms of entertainment available to people.(Ofcom 2020) As on-demand TV services grow in popularity, the live scheduled TV model declines because audiences want the facility of choice in what they are going to watch and when.Even prerecorded content shown "live" as part of a terrestrial TV broadcast doesn't produce the viewing statistics that the on-demand recording will bring in after it has aired.
Theatres during the pandemic tried to maintain their income and artistic output by producing camera recordings of their productions, either as live broadcasts or as live screenings of prerecorded content.
The "Old Vic" Theatre in London produced several new shows which streamed live for their initial opening and were later repeated as scheduled, ticketed events.The distinction between the two was whether they were called "In Camera" or "In Camera Live".(Old Vic 2021) As an audience member you watched the same production; the story and the experience of watching it from home felt identical.
This was also true of the RSCs "Dream" project which saw the live stream of a real-time digital performance but also included interactive controls for the audience to feel they are contributing to the performance.By firing "fire-flys" into the space, you felt you were involved but even though they went to great lengths to prove it was always live, each separate performance looked very similar and could have been pre-recorded.The individually placed "fire-flys" were so numerous it was impossible to identify your own, which also would lead a cynic to say that this could have been prerecorded too and left you to believe that you maintained some control.(RSC 2021) In this respect, it is very similar to a fair ground experience where no matter where you aim, or how good you are at completing the activity, mechanisms are in play to ensure the outcome is pre-determined without you realising.
Creating live content, in any medium, is far more complex and expensive than pre-recording it, so it feels necessary to have a reason for introducing this extra complexity, cost and in some cases, reduction in quality.
The majority of live TV, particularly reality TV or game shows which air "live" are delayed by 4 minutes to provide a chance for review of anything indecent being aired or to allow synchronisation of content across various platforms across the world.Is this still live?(Grundhauser 2021) If you go to a cinema and watch a film, you know that the action in the story is not happening at that moment some place else, and this doesn't matter as you are enjoying a shared and social experience of watching this story with other people.No-one would claim that a film in a cinema is "live" but does it have more "life" because we are experiencing it together?
This brings us to theatre which has many assumptions made about it's liveness because it is happening in front of us in person.The performer is of course live, and possibly because much of the artistic leadership and journalism that creates the narrative of theatre have come from performers or performer-centric backgrounds, there is an assumed deference to the value of the live performer and their relationship to the audience that undermines the rest of the visual and sensory picture which is, in the majority of cases, not live even when it is apparently in front of us in real life.Much of the scenic picture is pre-recorded and repeatable using the same technology used to create recorded films.The illusion of the theatre is that each of our experiences is unique and even if it is only subtle, it will be different from anyone else's experience.Theatre is not so honest as we might think and it is intriguing that audiences would be disappointed by a show that "lies" to them by improving the experience with pre-recorded theatrical trickery, whilst at the same time enjoying a story and characterisations that they know to be a simulation and intentionally not real.This is most common in commercial theatre where the audiences experience is highly controlled, prerecorded and rehearsed to be identical through every performance.
 The lighting is recorded into cues which an operator triggers on the command of a stage manager following a score. The sound effects are pre-recorded and triggered in the same way. The singing is often backed by prerecordings of the cast in a studio, to allow them to dance simultaneously without worrying about their voices carrying as they sing. The musical director is following a click track, which controls the pace of the music to match all of the other technical elements. The performer repeats moves and their performance each night as closely as possible to coordinate with the other performers, the music and the scenic elements, partially for the reasons of safety to ensure they are in the right place when automation systems pass close by.
This contrived collaboration of disciplines operating a show at once is demonstrated perfectly by The ROH Machine, a YouTube video which shows the stage manager controlling the pace of the show whilst simultaneously seeing the reaction to those commands by the team delivering this (ROH 2013).
When it comes to producing digital content which is identical when viewed live versus pre-recorded, and given the much higher costs and risks associated with live performances, it stands to reason that producers either feel that audiences demand the live content or they are mistakenly spending money they don't need to.
Is it simply enough if audiences believe they are watching something that is live, as I experienced with the Old Vic productions where my belief that something was live when it was actually prerecorded meant my enjoyment of it was enhanced for knowing that I was part of something myself.
An audience member in a real, live theatre experience is playing their role in supporting the productions liveness by applauding, laughing, sighing etc.These become performative mechanisms in themselves and something which, despite preview and test audiences, is very hard to predict during the design phase of a show and is often dependant on the time of day or type of audience that attends each individual show.

Digital humans
Posthumanists such as (Hayles 1999) discusses that the characteristics of materiality in humans comes from random/patterned behaviour combined with a sense of presence/absence in what we are viewing.As random and patterned behaviours occur in the digital works, the sense of absence or presence increases and decreases.One interpretation of this is that for the human to believe what they are seeing is real, the interjection of patterns will increase the sense of presence, the presence of another human being co-existing in the artform.
In what (Baudrillard, Glaser and of Michigan Press, 1994) refers to as "precession of simulacra", we are able to identify the unnatural patterns of copies as he explains in his own example of the Mona Lisa, which describes that anyone who spends a lifetime studying the original would identify a reproduction instantly.
But what if an expert had spent a lifetime studying a reproduction was then asked to look at the original and determine if it had added qualities that the reproduction didn't?Taken from their own perspective, the original is a reproduction and the patterns the reproductions artist used are part of the code they expect to see in all other reproductions, even the original.
In the context of live performance translated to the digital, the original is never seen by the audience as it is instantly reproduced into a digital version of itself.An expert could observe a live, digital version of a product and compare it to an identical, recorded digital version and not see the patterns and randomness that would define the fingerprint of the original, because both digital versions are copies of something they have never, and were never intended to see.
In Hayles posthuman view, "conscious agency has never been in control" and therefore is an illusion where the audience member, as a pro-active and responsive viewer of the content is capable of understanding their own sense of presence in the experience they are viewing.They do this without being aware of the subtle re-purposing of prerecorded content to engineer their experience to be more reliable and repeatable without breaking their illusion.

CONCLUSION
Our research into liveness in theatre has shown that this is a very large subject that requires an entire industry to engage and support through their collective creative practices, leading to more research that will build on our initial studies.
Our findings for "liveness" and it's importance to audiences has opened new areas of potential research to help isolate the key components that audiences are drawn to which could become potential research for the future.
It is also clear that "liveness" is subjective to audiences and there is no model that fit's all potential users, but there is a suggestion that in the digital experience market, particularly where it pertains to performance and theatre, is not reliant on the "live" as long as the audience thinks it is live.
There are many examples of live shows that are pre-recorded yet still attract live audiences such as the water fountains at the Bellagio Hotel in Las Vegas, the new years eve fireworks on the Thames or the Hong Kong light show that can be watched from the rivers south bank, not to mention cinema and other media which can attract an audience even when there is no "live".(BBC 2021;Bellagio 2021;Hong Kong Tourist Board 2021) This suggests that what is valued in this situation is "togetherness" and "exclusive access".Sitting in an empty cinema to watch a film is not as much as an experience as watching a film surrounded by other people whose reactions and movements become an unwitting protagonist in the storytelling experience.Being amongst the first to watch a new film on it's release date adds to this because it is an exclusive experience that you celebrate yourself to be part of because of it's uniqueness.For example, you could watch the New Years eve fireworks from home, but thousands of people go there to feel part of it.This might be because you become a participant in the experience and you get to live it in a way that makes you feel included in the story, almost as a protagonist.The shared laughter, excitement or even silence of an intense moment in a film in a cinema is part of your experience and it enhances it, even if we know that the film itself is not live.This would suggest that the conclusion to the value of "liveness" is not in the performance, but in the audience and the togetherness of people in sharing an experience.
Our biggest discovery however, is that liveness in the creation of digital experiences can support our digital and virtual practice.The ideation and iteration of our creative content is improved by the live experience, whether that be by drawing on our experience as makers of live theatre and drawing that into our digital practice, or by allowing the liveness of creative people working together allows us to engage and collaborate in a more human way.
The Round is an exciting alternative to the traditional theatre model which seeks to maintain the value of theatre through the sharing of live performances.However, in it's ambition to maintain the highest of theatrical values, the focus has been to provide the live performance to audiences which our findings suggest is not as valuable as the human shared experience.Where The Round has managed to maintain true theatre value is in the exclusivity of a one off experience which, because it is live, can only be experienced at a particular time due to the performance being instant and simultaneous.In this way, we are replicating the live, scheduled, terrestrial TV model that existed before "on-demand" TV was introduced in the Netflix model.As TV has had to grapple with the intermediality of on-demand vs live media, the theatre industry must do the same and seek to understand whether the future of digitised and virtual theatre should be built around live performance or real-life audiences.

Figure 1 :
Figure 1: The Round Architecture