Augmenting the Power of Artwork Through a Virtual Reality Partnered Piece

This paper seeks to challenge the existing state of 'augmented reality' within the visual arts. Currently the majority of augmented reality applications are little more than gimmicks viewed through a mobile phone or tablet. The augmentation is generally limited by the hardware restrictions, and even when using high end equipment such as the Microsoft HoloLens the reduced field of digital overlay leads to a weakened immersive experience. Working alongside various artists and Anise Gallery A-VR have pioneered the 'partnered piece' concept through the creation of bespoke curated VR (Virtual Reality) experiences which accompany the original physical artwork. The VR pieces themselves can exist as standalone experiences but when viewed alongside related artwork they increase the power of the whole tremendously. Focussing on a number of example projects this paper charts our investigations into producing the most immersive additive experience to physical artwork. The implementation of such a virtual reality partnered piece has influenced and refined the language of our VR production. The strength of the immersive experience often having profound emotional effects on the viewers and demonstrates the power of a truly 'augmented' creative experience.


INTRODUCTION
AVR London have been producing CGI and video content, mainly for the architecture industry, since being established by Joseph Robson in 2006. In 2014 they began producing Virtual Reality (VR) content predominantly using Unreal Engine and in 2016 this offshoot venture was formalised with the incorporation of a sister company A-VR co-directed by Felix Dodd. Throughout the last five years we have been developing a methodology for the production of high quality immersive experiences.
The storytelling element and the development of the partnered piece came from multiple collaborations with Anise Gallery and the desire to add experiential value to artworks and exhibitions This augmentation of experience has culminated not in the current understanding of the term 'augmented reality' but in the development of the VR partnered piece. This paper looks at the increase in experiential value offered by True Virtual Reality augmented alongside artwork compared to the traditional sense of mixing reality and overlaid Augmented Reality (AR) through a mixed reality headset or similar.

AR VERSUS VR VERSUS 360-DEGREE VIDEO
Virtual Reality has been around much longer than most people are aware. The use of a headset to view images which immerse the viewers vision and offer stereographic depth were popularised in the 19th century with the invention of the stereoscope. The difference between Virtual and Augmented Reality can easily be described by focussing on the hardware used to display the experience.
The re-emergence and popularisation of head mounted displays (HMD) has led to an explosion of interest in immersive technology and a confusing array of derivatives broadly categorised by the public as 'Virtual Reality'; the differences however are great. True VR can be defined as being an immersive stereo display, which entirely replaces the viewers' vision with a simulation. The simulation's movement responds directly to the movement of the participant and offers six degrees of freedom (6DoF), typical headset examples able to show True VR content being the HTC Vive and Oculus Rift and rely on separate computing and graphics hardware to drive them.
Augmented Reality headsets such as the HoloLens or Magic Leap use on board computing power and holographic lenses to provide a pass-through display so that the real world can be seen upon which 3D computer generated content is superimposed and integrated to some extent. The main criticisms of the lenses being that of field of view of the augmented image, as if looking at the virtual world through a letterbox. Although this is increasing with the Magic Leap and HoloLens 2 now having a diagonal field of view of around 50 degrees compared to 110 degrees of the HTC Vive. AR headsets also offer stereo imagery and 6DoF, but the nature of the projected image being semi-transparent makes any augmentation appear ghostly.
A third type of headset is the Google Cardboard or Samsung Gear type, these are at the low end of the market and are simply viewing cradles to hold smart phones which show the content. Usually the mobile VR content on these devices is 360-degree images or video, often not stereo, and only provides rotational movement (3DoF) due to their origins being captured from a single point source similar to photographs or video.
For our simulations the intention is to create the greatest amount of immersion possible. For this to happen stereo imagery and 6DoF is essential. Hijacking the visual feed of the participant and entirely replacing the viewer's environment with computer generated (CG) input was important in order to complete the immersion and provide the maximum presence.
Presence is the magic of virtual reality, the feeling that you're actually in the virtual world. Presence will cause the user to suspend disbelief and critically have subconscious reactions to stimuli as if they were real. It can be described as the ultimate experience and the purpose of VR. The problems being that the CG needed as input through the visual cortex needs to be generated from a 3D model and rendered in real time (at least 90 frames per second) and with no latency. Currently we believe this is only possible through the implementation of True VR, the augmentation process described in the paper's title is through a process of witnessing the artwork physically followed by a separate but creatively associated experience which is delivered and experienced through VR in a HMD.

VR art as a stand-alone statement
Most VR artworks exhibited are stand-alone experiences like traditional art. The power of the medium has attracted a number of artists who now develop work in VR, or have migrated from traditional mediums and have diversified into VR. In most cases this results in VR content that rather than adding a dimension to the artists catalogue, behaves as a work of art in its own right. Notable divergence from this exhibition model is rarely seen as artwork in the physical space and VR need to accompany each other perfectly.

I Saw It Whole (2016)
The partnered piece concept we first showcased during Open House London (AVR London / Mueller 2016) was a collaborative work produced between woodcut artist Scarlet Mueller and digital artists at AVR London. It took the form of a 3D environmental recreation of a 2D woodcut. The original woodcut was hung in Anise Gallery alongside the VR installation on an HTC Vive. The gallery visitors would appreciate and study the woodcut before immersing themselves in the VR environment. The environment itself was not only a 3D recreation of imagined abstracted space, but also ebbed and flowed with a soundtrack (music the artists worked to when creating the woodcut), and there were also animated cyclic abstractions of the woodcut also moving. Some reactions from visitors are below: The environment reacting around me was quite surprising. The way they have responded to the artwork is really beautiful, really sensitive capturing the magic of the artwork.
It was quite euphoric… more about emotional tone than strict lines. As the piece was free to explore there were no time limits to the experience, which although giving users the freedom to control the length of their experience it did have a slight time disorientation not knowing when the end was. 3 Months after the finissage of I Saw It Whole, the Serpentine Gallery exhibited works by Zaha Hadid alongside VR pieces deconstructing her large scale painting to emphatic reviews and hour long queues for the experience (Hadid 2017).

Gallery in a Box (2018)
As part of Anise Gallery's stand at the London Art Fair in January 2018 A-VR piloted the curated experience format in order to showcase the five artists represented on the stand itself -an experiential artist catalogue. Visitors were encouraged to look at the artworks physically exhibited and then experience them virtually. The narrative progressed through each artist alongside music and scenario changes combined with very gentle pacing throughout. The idea here was to make people feel semiconscious in the VR, somewhere between sleep and being awake. The reason for invoking this state is that it amplifies our perceptual system, and provides people with an even deeper experience. As a result powerful reactions were recorded, with some viewers claiming to have a physical sensation, absorbing the artwork into the body like a kind of digital digestion, strange and powerful in equal measure.

Figure 4: A-VR/Anise Gallery, Gallery in a Box, London Art Fair 2018.
From the varied and plentiful feedback received at a public art fair we can draw conclusions regarding the most successful experience types and feedback into future projects to increase the levels of immersion felt by the viewer.  As part of the London Festival of Architecture (A-VR 2018) we endeavoured to envelop three of the participants' senses within the VR experience with the addition of smell. A 3D texture mapped environment was created to match the actual gallery space, a digital twin in which participants were subconsciously familiar with the simulation. The way a user feels in a room scale experience is very much dependent on what they know is around them in the real world, and this we call a perceptual anchor. Through an earlier exhibition (A-VR 2017) we had confirmed the belief that replicating physical space within the virtual environment and allowing the participant (in that case untethered) freedom of movement, brought a much greater sense of immersion. This direct correlation of virtual world mapped 1:1 to the real world space helped to contextualise the experience before participants put on the HMD. This perceptual anchor or 'déjà vu' of the environment brought increased immersion as participants moved in a manner which mirrored the real space installation and helped the sense of ease within the experience (Parker 2018).

Following on from the success of 'Gallery in a Box'
Jacek Ludwig Scarso approached A-VR and Anise Gallery to produce a partnered piece for his forthcoming installation at Tate Modern, giving us the opportunity to build on the curated format. In this case the VR was to become part of a wider installation on the theme of waiting: INLIMBO is a participatory installation, in which the public is invited to ponder with us what we are waiting for and whether it is worth the wait. Waiting is more than an action. It is, in many ways, a condition of our lived experience. As the speed of the world at large increases, it is not so much that we spend more time waiting, but that we are obsessively conscious of it. The relentless drive forward makes our experience of The NOW one of anticipation and of slightly irritated longing.
The public enters the INLIMBO Waiting Room, a space filled with stage smoke and guarded by "Bureaucrat Angels", who hand them a form to fill in and direct them to a long row of chairs. The form includes a range of triggers on their experience of waiting, from trivial to deeper questions. There is a doodling section in the form, to fill in while waiting. As they complete the questionnaire, they hand it over to the Angels, who hang them on a wall, creating an increasingly large collective artwork on what it means to wait. The artwork itself becomes part of the experience for the viewer.
Throughout the event, an immersive surround soundtrack layers the sound of a ticking clock in reverse, with a nostalgic waltz, which contributes to the playful tone of the experience and its symbolic reference to time.
In the meantime, other Angels are visible across the Museum spaces. They are not allowed to talk to the public: they walk in slow motion, knowingly smiling, and pass on secret messages directing them to the Waiting Room. This live presence is essential to the work, as it both generates substantial interest and contributes to the engagement of the public's imagination.
In a special collaboration with Felix Dodd and A-VR, INLIMBO features a unique VR journey, immersing the public in a virtual waiting room. As they navigate this, they are confronted by further questions, once again through the presence of the Bureaucrat Angels. The VR complements the live experience, by promoting a further level of playful engagement in the intimate nature of this medium. Across the physical space of the installation, TV screens play excerpts of the VR in 2D.
We built a virtual world made up of corridors and waiting rooms, complete with endless seating for the wait. The same music was used as in the physical installation, and participants were taken on a journey through this strange world. Two techniques were employed in order to enhance the artist's vision. Firstly time in the experience was modulated subjecting users to a kind of timedilation. This created a perceptual disconnection from the real-world and made people feel that the experience was much shorter than it actually was (generally participants thought the experience lasted a minute or two, when in fact it was twice that). The second technique was to alter the scale of the world, so that users felt like they emerged from the virtual experience inside the virtual experience -a powerful contextualisation effect. Over the course of a week the installation had around 1500 visitors and 1100 forms were completed. The emotional response to the piece was strong with one particular participant almost moved to tears stating the experience helped her deal with a traumatic separation.

THE PASSIVE JOURNEY
From our earliest VR exhibition and work (AVR London, 2015) to our most recent the storytelling and method of deliver have altered in a subtle but very powerful manner. Active navigation, whereby the viewer is given control over all of their 6DoF movements through a game controller proved to be one of the main contributors to motion sickness. This can be mitigated by locking rotational movements (yaw, roll and pitch) to only respond to movements detected by the HMD and not through the game controller.
It was also found that allowing total navigational freedom to explore a virtual environment through a game controller or 'teleporting' lead to a reduced sense of immersion. This may be due in part to the distraction involved with controlling the navigation, in the same way that a passenger in a car is more likely to appreciate their surroundings than a driver. It was for these reasons that we now pursue a navigationally passive journey.
With the passive journey format the participant can be directed and guided through a curated story with a clear emphasis on space and time. By removing all interaction, except freedom of movement, linear storytelling allows the developer to closely control the experience of the viewer. Multiple participants will have similar experiences and are able to discuss these and share impressions afterwards (Parker 2018). The power of the shared memories compounding the power of the experience.

CONCLUSION
Through the development of numerous VR experiences in an art and exhibition context we have developed a syntax of the immersive environment, one which when partnered with physical artworks in the real world offer a profound enhancement and greater understanding of that work.
By isolating the viewers experience for a limited time, the narrative storytelling is akin to cinema, and allows the developer to engage with the participant in a manner, which by engulfing their senses is free from distraction. Manipulative control over the experience through editing must be used cautiously as the power of truly immersive VR can profoundly impact the participants' emotional and physiological state much more than is possible through augmented reality.