Exploring Different Ways of Navigating Emotionally-responsive Artwork in Immersive Virtual Environments

In this paper, we propose a work-in-progress virtual reality system to explore how different ways of navigating emotionally-responsive artwork can change one's phenomenal experience of the artwork in an immersive virtual reality environment. Two sets of navigational scenarios are designed from artwork generated from our parameterized NPR (non-photorealistic rendering) system ePainterly. Preliminary studies are carried out to observe how different navigational environments affect one's viewing experience to artwork generated through a NPR system. Immersive virtual reality. Virtual environments. Gestural navigation. Non-photorealistic rendering. Head-mounted display. Generative art. Emotion. Biofeedback. Interaction.


INTRODUCTION
Traditional museums and gallery spaces were built almost as ritualistic and ceremonial monuments where works of art are selected and arranged within a sequence of spaces (Duncan & Wallach 1978, O'Doherty 1986).Paintings were hung on the walls of the gallery spaces for audiences around the time of the late baroque period in Parisian salons and that form of viewing artwork has evolved to this day into modern museums and galleries (O'Doherty 1986).How these museum and gallery spaces' physical settings provide a context to the works of art and in turn, how the artwork contributes to a larger meaning of the physical space has been an increasing interest for many art historians in this century (Duncan & Wallach 1978).
Digital technologies have enabled us to depart from the traditional museum and gallery spaces to create new contexts and novel ways to interact with the presented artwork.Interactive art installations integrating various sensor technologies involve the audiences as active participants of the presented artwork, often participating and co-creating the art (Watson 2007, Eyl & Green 2004).Online virtual art galleries provide a venue for artists to share and disseminate their artwork to anyone on the WWW.Gary Singh states that these online millieus "are beginning to circumvent the traditional analog art gallery experience" (Singh 2011).
Moreover, as technologies to create and experience fully immersive virtual environments such as the head-mounted displays (HMD) are becoming more affordable and accessible to the general public, more artists are experimenting and creating virtual art within a platform where the physical space is becoming replaced and/or altered with a virtual one (Chiaverina 2014).Virtual art experiences in these immersive virtual environments are pushing the boundaries of how one relates and interacts to the arts and in turn to its physical and virtual environments.
In this paper, we explore how different navigational environments can potentially change how we engage with emotionally-responsive artwork in immersive virtual environments (VEs).We are curious to find out whether given a non-traditional way of viewing a given set of artwork generated from a NPR (non-photorealistic rendering) system, do both the users' experience and how they relate to the artwork change?To investigate this further, we present an early prototype of a virtual artviewing system.We designed two immersive virtual environments, each with its own navigation for viewing the artwork.To keep the user's focus only on the presented artwork, the virtual environments have been designed with only the artwork with no other environmental elements (e.g.interior or exterior landscapes or backgrounds).Our goal is to further explore the possibilities immersive virtual environments have to offer as an extended platform for viewing and experiencing art in its various forms.
This paper is organized as follows.In Section 2, we present the background and related work to virtual reality, virtual art, and biofeedback technology.In Section 3, we present our generative system for creating our emotionally-responsive artwork that we are using in our virtual environments.In Sections 4 and 5, we present our system design and pilot studies.In Section 6 we conclude and present our future work.

BACKGROUND AND RELATED WORK
Virtual reality (VR) technology is not new by any means, it has been around for over 70 years with its applications ranging from medicine, gaming, flight simulators, scientific visualizations, virtual museums and galleries to therapeutic applications and many more.VR technology itself is wideranging from fully immersive environments (i.e.CAVEs, VR Theatres) to the current DIY cardboard HMDs.Although VR has extensively been used for scientific and engineering applications, artists have also been steadily contributing the development of VR from the 1970s (Em n.d., Davies & Harrison 1996, Dolinsky n.d., Goslin & Morie 1996).VR's core strengths lie in its ability to create a sense of immersion (Slater & Wilbur 1997) through technologies that replace many of the sensorial channels and in return create responsive environments where participants can interact and explore the VEs in real-time.Integrating psychophysiological measures into the VEs for creating affective loop systems as well as studying users' experience and emotions, especially in the field of interactive entertainment and gaming, is a growing area of research (El-Nasr, Morie & Drachen 2011).
In the first two sections, we briefly review how VR has evolved, its related technologies and examine virtual art created for and experienced in VEs.In the third section, we briefly examine psychophysiological measures currently being used for studying user experiences in VEs as well as affective loop VR systems.

VR Technology and Navigation
The notion of virtual reality has been around since the early 1900s.The term itself originated from Antonin Artaud's in 1938 book The Theater and Its Double (Artuad 1958, p. 49) but it is commonly thought to have been coined or popularized by Jaron Lanier (Lanier n.d.).
The first VR systems can be roughly traced back to the works of the first simulators at around the turn of the twentieth century ("A training 'class' at Chalons" 1909).The first head-mounted display system was built in the 1960s by Ivan Sutherland.He pioneered graphics simulation and produced the first system to replace video signals with totally synthetic images produced through computer graphics to his head mounted three-dimensional display using mechanical position sensors (Sutherland 1970).
VEs can be experienced on diverse platforms.CAVE (Cave Automatic Virtual Environment) is a fully immersive virtual reality environment with three to six walls (including the floor) of stereoscopic projection equipped with motion sensors.The 3D environment is seamlessly projected onto the walls and the floor while the participants walk around and interact with the projected environment.HMDs are goggles or a helmet fitted with one or two optic screens that are worn over the head.HMDs with two optic screens are capable of stereoscopic display and can either be fully inclusive or see-through.Some HMDs include a built-in head-tracker for position tracking (i.e. head rotation) and some also come with builtin stereo headphones.Researchers have built VEs where they have combined HMDs with customsensor floors for position tracking and have added other sensorial stimulants (i.e.smell, wind) to enhance the participant's experience (El-Nasr, Morie & Drachen 2011).Navigating within VEs is also an important area of research.Traditional navigation devices range from 3D mice, spaceballs and datagloves.Many researchers have integrated more natural ways of interaction using everyday objects such as a flashlight to explore a Virtual Dunhuang Art Cave (Lutz & Weintke, 1999).In 2013, the Leap Motion controller was introduced to the gaming and VR market.It is a small sensor device (3" long) designed for 3D motion and gesture recognition using the hand and finger motions as input (Leap Motion 2015).

VR from the Virtual Art Perspective
VR's origins and its evolution from artists and art historians' perspective are fundamentally different from the technological perspective.In art historian Oliver Grau's book Virtual Art: From Illusion to Immersion, Grau examines the evolution of VR from the perspective of culture and art history, focusing on the qualities of illusion and immersion (Grau 2004).He argues that even though VR is perceived by many as an entirely new phenomenon, VR was not born with the invention of the computer but has its roots in historical illusionary image spaces dating back to antiquity (Grau, 2004).He gives examples of illusionary spaces of the Great Frieze in the Villa dei Misteri at Pompeii (60 B.C.) to medieval cathedrals.Grau's separation of VR from its technology shifts our focus away from the technological aspects of VR and redirects it to the experiential aspects of the virtual space.Below we look at a couple of VR artists who have created VR installations on varying platforms of their art, its environment and interactions to be experienced in its entirety as an art form.
Margaret Dolinsky is a CAVE research artist who has been creating VEs with her artwork since the mid 1990s (Dolinsky, n.d.).In her ImmersaGram series, virtual surreal environments are filled with provocative objects that tell stories of their lives in the CAVE as the participants journey through the environment.Char Davies is a contemporary artist who integrates VR technologies into her art installations.In her VR installation Osmose, the participant puts on a HMD and a motion-tracking vest and journeys through metaphorical aspects of nature through his/her breathing and balance (Davies & Harrison 1996).Osmose is a classic example of creating an affective loop system where as one breathes, his/her own breath transports them through the VE.
In New York-based artist Ian Cheng's 2013 installation, two Oculus Rift (Oculus VR 2015) HMDs were on a sofa connected to the projection screens on the walls.Participants viewed his deeply abstracted 3D worlds through the HMDs as they also became part of the installation (Cheng 2013).VR researchers and artists have also recreating virtual art galleries and museums but many with purposes of preservation and providing access to a wider audience.Often, virtual galleries are recreated as exact replicas of the actual gallery space in 3D.(Cavazza & Mead 2001, Vosinakis & Xenakis 2011).

Psychophysiological Measures in Virtual and Interactive Environments
In this section, we briefly introduce research on using psychophysiological measures as a closed feedback loop to create affective environments in VEs as well as using it as an added measure for emotion studies in interactive environments.
Friedman, Suji & Slater (2007) created a CAVE art installation called SuperDreamCity with a Londonbased artist Kana Suji with 3D dreamscapes she modelled.They designed an affective loop system integrating the participant's galvanic skin response (GSR) for the main navigation in the CAVE.
Through their GSR, the participants were able to control the speed as well as select different dream sites.In the Virtual Meditative Walk (Gromala et al. 2011), the author created a virtual walking meditation system that was connected to a unidirectional treadmill and the participant's GSR.The participants were able to walk through the virtual forest while their GSR gradually changed their environmental elements in real-time.
Using psychophysiological measures in conjunction with qualitative measures such as interviews, selfreporting, observation analysis, and behavioural analysis continues to be an important area of research in human computer interaction (HCI) (El-Nasr et al.).It can provide rich data on one's emotional states and new non-invasive wearable technologies for tracking one's biofeedback for health research and well-being are becoming available for the general public (Empatica 2015, Microsoft Band 2015).
Psychophysiological measures (Stern 2000) are divided into signals from the peripheral nervous system (i.e.heart rate, galvanic skin response) and the central nervous system (i.e.EEG).EDA (electrodermal activity) also known as galvanic skin response (GSR) measures changes in perspiration and electrical conductivity of the skin.Respiration measures the breathing rate and amplitude (i.e.depth of breathing).EMG (electromyography) measures the electrical activation of the muscles.This can be measured on the face or other parts of the body (i.e.arm).Cardiovascular measures the organs in the body that regulate blood flow.A variety of measures can be taken including heart rate (HR), heart rate variability (HRV), interbeat interval (IBI), blood pressure (BP) and blood volume pressure (BVP).EKG (electrocardiograms) measures the electrical activity of the heart.Refer to Stern (2000) for an in-depth overview of psychophysiological recording.
Although studying human emotion using psychophysiological measures in relation to interactive environments is complex and more research needs to be carried out, many researchers are showing positive results.Lang (1995) showed that using picture stimuli highly correlated EDA with selfreported emotional arousal.Ravaja et al. (2006) also carried out studies using facial EMG and EDA and showed positive and negative affect responses (arousal) during game play.Mandryk, Atkins & Inkpen (2006) carried out an extensive study on evaluation continuous emotion during interactive play using four measure of psychophysiology: GSR, EKG, facial EMG (EMG smiling , EMG frowning ) and HR.They showed great potential for modelling emotional through physiological metrics during interactive play.
In this paper, we are interested in using physicological data as an added measure to the selfreport measures to detect physiological response to the presented artwork.Further, we are interested in controlling a subconscious aspect in the environment for added dimension to the main navigation.

GENERATIVE ARTWORK: GHOST IN THE MACHINE SERIES
The artwork from the Ghost in the Machine series is from Steve DiPaola's Computational Aesthesis 2014 art show.DiPaola took "selfie" photos from his social media and using his parameterized filterbased non-photorealistic rendering (NPR) software, he scripted the code and computationally "painted" them to bring out the deeper ghosts from the machine.

DESIGNING THE VR SYSTEM: A PROTOTYPE
The art-viewing VR navigation system is currently being developed in two phases.For the viewing and navigation hardware, we are using the Oculus Rift (Oculus VR 2015) HMD and the Xbox controller (Xbox 2015).Oculus Rift HMD is a VR HMD capable of full 360° view of the virtual environment.
The Oculus Rift HMD we are currently using is the DK2 version, which has a 5.7in display with 24-bit colour depth, which generates 1080p display in each eye.The Xbox controller is a gaming navigational device designed for the Xbox.It has built-in support for integration into real-time 3D graphics/gaming engine, Unity3D and has a series of customizable buttons for navigation.

Figure 4: Oculus Rift Head-Mounted Display with Xbox Controller
In the following sections, we describe the system design and the two navigational setups for Phase 1 and Phase 2 of the development.

System Design: Phase 1
For Phase 1 of the development, we selected eight images from the Ghost in the Machine series artwork for the VEs.These images were used to design two navigational setups: (1) panorama setup and (2) fly-through setup.The design of the two setups was focused on creating contrasting interactions in viewing the artwork utilizing the affordances of VEs.The first panorama design is a selection-based interaction where the participants can explore the artwork in the full 360° view and selects the artwork they would like to examine further.The second fly-through is a journey-based interaction where the artwork is in the participant's full field-of-view (FOV) and the participant is able to fly through to view the images either traversing forwards or backwards.Both these setups were first modelled in Autodesk Maya, exported to Unity3D and then ported over to run on the Oculus Rift HMD. . is an overview of our system design.Biofeedback integration into our system will be carried out during Phase 2 of our development.

Navigation Setup 1
For the first navigational setup, we created a panoramic view of the artwork.Through years of using and watching people use HMDs, we observed that as soon as people put on the HMD, many people start to look around and explore the VE.Our navigational setup was designed to exploit this viewing tendency by creating a full 360° view of the artwork.In addition to being able to view the artwork by looking around, the participants can rotate the artwork clock-wise or counter clock-wise and select ones they want to view closer.The selected artwork will zoom in closer in the FOV.

Navigation Setup 2
For the second navigational setup, we created a series of image panels the participant can pass through by moving forward or backward to view the images.For this VE, we wanted to create an environment that created more of an embodied experience that journeyed through the artwork.The participant can choose to move forward into the artwork to view the next one or choose to move back to view the previous ones.In this setup, the participant can only view one image at a time.For the biofeedback integration into the VR system, we will be using the Empatica (Empatica 2015) biofeedback wristband, which is a wireless wristband capable of reading continuous HR and GSR (Figure 11.).With this technique, we can obtain arousal states through the physiological signals to create an affective environment where elements in the VE change dynamically in response to the participant's biofeedback.
Research has shown that VR can be an affective medium for using affective interaction where the interaction between the two different environments of "anxious" vs. "relaxing" produced anxiety and relaxation (Riva et al. 2007).

USER EXPERIENCE STUDY DESIGN
Our pilot study was designed with two purposes in mind: (1) to obtain knowledge on emotional responses to the artworks, and (2) to assess the overall experience of the two different navigational setups.We used the two navigational setups from Phase 1 of the development.For the pilot study, we recruited six subjects (two male and four female) between 20 to 70 years of age, all of whom had minimal to no prior experience using HMDs.All subjects experienced both navigational setups; the order of presentation of the navigational setups was counterbalanced between subjects and the image order was also randomized for both setups.Subjects were given instructions during their exposure to each navigational setup, and a questionnaire after they had completed both navigational setups.We also recorded the amount of time each subject dwelled on particular images.
At the start of each study, the subjects were given short instructions on how to navigate in the VE with the Xbox controller.During the study, each subject spent approximately two minutes in total for each navigational setup and an average of four seconds viewing each image.During each setup, subjects were given the task to select the most calming image and the most upsetting image.After both setups were completed, each subject was given a short questionnaire to record their overall experience.Preliminary studies showed that subjects equally found both the panorama and the fly-through to be an engaging and comfortable way of viewing artwork.Subjects spent more uniform time viewing each image in the fly-through setup and for the panorama setup, subjects enjoyed being able to view the artwork by looking around rather than using the rotation button on the Xbox controller.Further, subjects responded stronger to the fly-through setup with the subjects selecting artwork with more muted colours in contrast to the stronger colour palettes in the panoramic setup.

CONCLUSIONS AND FUTURE WORK
In this paper, we have presented a work-inprogress prototype of an immersive virtual reality environ-ment for viewing artwork generated from our NPR system ePainterly.images from the Ghost in the Machine series were selected and put into two navigational setups in the VE.Currently we have finished the implementation of Phase 1 of our development and have the VR system running on the Oculus Rift HMD using a Xbox controller.User studies were carried out to assess emotional responses as well as overall experiences of the two navigational setups.Studies showed that subjects enjoyed both navigational setups but found the flythrough navigation to be stronger in the artwork presentation.Subjects enjoyed both setups and commented on how unique and striking each piece of artwork was.They further commented on how the same artwork felt different in each setup and the subjects selected artwork with a more muted colour palette in the fly-through navigation.Some subjects also commented that they loved the personal relationship they could build with the images within their personal VE space.
Based on this preliminary pilot study, we have started our next phase of development.We are currently replacing the Xbox controller with the Leap Motion controller to use the participant's gestures to navigate the VE.We are also integrating the participant's physiological signals using the Empatica biofeedback wristband.The Empatica wristband is a wireless wristband capable of reading continuous HR and GSR while still letting the participants freely use both hands to do other tasks.We will be creating an affective loop system where the participant's emotional states (i.e.arousal, excitement) obtained through their physiological signals dynamically affect subconscious aspects of how they view the artwork in the VE in real-time.We are currently investigating different aspects of the VE that can be affected in real-time during the navigational experience.We will carry out a larger user study to test how the participant's physiological states affect changes in their navigational experience.Further, we hope to extend our system to be used in virtual artistic therapeutic applications using our emotional modelling work.

Figure 5 :
Figure 5: System Design Figure 5. is an overview of our system design.Biofeedback integration into our system will be carried out during Phase 2 of our development.

Figure 6 :
Figure 6: Navigation Setup 1: Panorama The Xbox controller was used for the interface controls (see Figure 7.).The Left Stick button rotates the panels clock-wise or counter clock-wise.Holding down button A zooms the image closer and releasing it zooms out of the image.

Figure 8 :
Figure 8: Navigation Setup 2: Fly-through The Xbox controller was setup in this navigational setup as follows: button A moves forward through the image one at a time and button B moves backwards through the images (see Figure 7.).

Figure 9 :
Figure 9: Oculus Rift Head-mounted Display with Leap Motion Controller Integration The Oculus HMD has a field-of-view (FOV) of 100° and the Leap Motion controller has a FOV of 135° (Figure 10.).

Figure 10 :
Figure 10: Oculus Rift HMD FOV and Leap Motion Controller FOV Currently, the integration of the Leap Motion controller for navigation setup 2 has been completed.We created 'hot spots' in the FOV of the Leap Motion controller to recognize the swipe motion of the hands.When the participant swipes with the right hand from right to left into the FOV, the participant moves forward to the next image and when the participant swipes with the left hand from the left to right into the FOV, the participant moves backwards to the previous image.