Framework for a Bio-Responsive VR for Interactive Real-time Environments and Interactives

In this paper, we propose a framework for a bio-responsive Virtual Reality (VR) system that evolves and reflects the immersant’s physiological data such as gestures and movements to biofeedback measures over time into a tightly coupled real-virtual immersive space. This paper will present current prototypes in the various application and experiential spaces we see our work being applied to from therapeutic applications to meditative spaces and artistic installations.


INTRODUCTION
Over the recent years, virtual reality (VR) has entered the mass consumer market with a wide array of head-mounted displays (HMDs) ranging from low-end (i.e.Google Cardboard) to room-scale VR (i.e.HTC Vive) that has made VR much more approachable and affordable outside of the research and academic institutions.This has brought on a paradigm shift in how VR content is being designed and developed to how it is being viewed and experienced.The focus for VR application space, which has predominantly been applied to engineering and scientific visualizations to simulations has shifted to entertainment (i.e.gaming, VR films & storytelling) and individual wellbeing (i.e.meditative, relaxation).Current searches on the Google Play Store yields over 100 VR applications related to meditation and relaxation.This is not including the flux of 360° VR meditation videos on YouTube.Coupled with current commercial sensing technologies (i.e.FitBit, Apple Watch, Spire) that can track, monitor and analyse personal physiological data in real-time, people are becoming hyper-aware of their health in relation to their physiological measures and states.At our research lab, iVizLab, one of the core areas of research focuses on the human body as the embodied extension between the tightly coupled real-virtual immersive space.Our goal is to create evolving experiential spaces interfaced through human gestures, movements, biofeedback measures and voice.The immersive environments the immersants interact with in real-time are created through computational systems that incorporate biological, cognitive as well as behavioural knowledge models.This paper will present an early framework for generating a bio-responsive VR system that evolves and reflects the immersant's physiological input data from movements, gestures and biofeedback measures in the immersive virtual environment.In our lab, we have been critically examining the role of the virtual and immersive space as a therapeutic space from the perspective of traditional expressive arts therapies.The intersections between the immersants, the realvirtual environments and the generated experiential space has potential for a variety of applications including stress relief, meditation, museum exhibitions and artistic interactives.In this paper, we will also present current prototypes in these application spaces we see our research being applied to from therapeutic applications to meditative spaces and artistic installations.This paper is organized as follows.In Section 2, we present the background and related work on VR and biofeedback technologies and expressive-arts therapies.In Section 3, we present our System Framework in detail.In Section 4, we present our early prototypes and its application spaces.Lastly, in Section 5, we conclude and present future work.

BACKGROUND AND RELATED WORK
VR technology has been around for well over 70 years, however, these past 5 years has been quite remarkable in terms of technical advancement making VR much more accessible both in terms of portability and affordability.With the introduction of room-scale HTC Vive VR headset and its outside-in tracking with 2 lighthouses (Buckley 19 May 2015), traditional sit-down VR experiences are now experienced with the user naturally moving and walking around in the real-virtual space.How one creates and views art in VR has drastically changed with artists being able to draw, sculpt, view and explore art creations in real-time directly in the VR space (Raab 2016).More in-VR 3D object creation tools are currently being developed (i.e.MakeVR for HTC Vive (Vive Team 27 March 2017), Unity's EditorVR (Ebrahimi 15 December 2016) and Unreal Engine VR Editor) making it possible for faster content-creation in VR and potentially eliminating many months of a tedious 3D modelling/rendering pipeline.In addition to its outside-in tracking, integrating bio-affective tracking to drive the interface in affective loop systems and studying users' experiences and emotions, especially in the field of interactive entertainment and gaming, is a growing area of research (El-Nasr, Morie & Drachen 2011).
Our research draws broadly from the disciplines of creative and expressive arts therapies (i.e.art, music, dance/movement) and research in biofeedback as it is applied to therapeutic spaces including stress relief, meditation and well-being.In this section, we introduce the background concepts our research draws from and examine current research being carried out in related fields of VR and immersive spaces applied to therapeutic applications, gestural input and psychophysiological measures and exploration of art installations.

Expressive Arts Therapies
Expressive arts therapy is defined as the therapeutic use of the art, music, dance/movement, drama and poetry/writing.In the U.K., the 4 recognized expressive arts therapies are: Art Therapy, Music Therapy, Drama Therapy and Dance Movement Therapy (Meekums 2002).Engaging in one or more art forms in therapy makes the therapy multimodal (Halprin 2003).Ideas of combining different forms of art is not new.In The Arts and Psychotherapy, McNiff (1981) describes how expressive arts therapy is linked to world healing practice traditions and cultural precedents where indigenous healers or shamans might sing, dance, make images, or tell stories in their healing rituals.Bernie Warren emphasizes in the book Using the Creative Arts in Therapy: A Practical Introduction (2003, p. 5) that there should be a shift in our thinking from 'arts as therapy' to 'arts for health' with the emphasis on indulging in the actual creative processes.
Artists and therapists who embark on the journey to become a therapist within the expressive and creative arts disciplines often devote their lives to training and practice.Each of these disciplines are vast and there are many philosophical and theoretical approaches and schools, frameworks, and models the artists and therapists employ in their practice.It is beyond the scope of this paper to present these approaches and schools, frameworks and models.Here we will present the fundamental definitions art therapy, music therapy and dance/movement therapy.

Art Therapy
According to the American Art Therapy Association (2017), art therapy is defined as: Art therapy is an integrative mental health profession that combines knowledge and understanding of human development and psychological theories and techniques with visual arts and the creative process to provide a unique approach for helping clients improve psychological health, cognitive abilities, and sensory-motor functions.
Art therapy is built upon the premise that the most accessible form for communicating human experience is in the form of visual symbols and images (Ford-Martin 2001).According to Malchiodi (2003), there are two main views of art therapy from the therapists' points-of-view.One is 'art as psychotherapy' where art is used in conjunction with psychotherapy to facilitate the therapeutic process through image making and verbal exchange.The other view is 'art itself is the therapy' where the actual creative process of art making (drawing, painting, sculpting, other art forms) is life enhancing and therapeutic.Both views seem to coexist and contribute to how art therapy is practiced to this day.

Music Therapy
The World Federation of Music Therapy (WFMT) defines music therapy as (Wigram 2002, p.30): Music therapy is the use of music and/or musical elements (sound, rhythm, melody and harmony) by a qualified music therapist, with a client or group, in a process designed to facilitate and promote communication, relationships, learning, mobilization, expression, organization, and other relevant therapeutic objectives, in order to meet physical, emotional, mental, social and cognitive needs.
Two distinct uses of music therapy have been identified from the international music therapy community (Darnley-Smith & Patey 2003, p. 6) as the following: 1) using music for it inherent restorative/healing qualities; and 2) using music for interaction and self-expression within a therapeutic relationship.A growing number of practitioners and researchers in music therapy and ethnomusicology are researching the concept of rhythmic entrainment or entrainment and its applications to music and the therapeutic space (Kret 11 April 2014, Thaut, McIntosh & Hoemberg 2014, Darnley-Smith & Patey 2003).
Entrainment, first identified by a Dutch physicist Christiaan Huygens in 1665, is a universal phenomenon in which two or more independent rhythmic processes synchronize and "lock" into a common phase and/or periocity (Darnley-Smith & Patey 2003).Thaut et al. (1999Thaut et al. ( , 2014) ) first established rhythmic entrainment as a function for rehabilitative training and learning in the early 1990s.Kret (11 April 2014) describes rhythmic entrainment in music therapy as "a specialized practice used to assist in helping people become more 'in tune' to their own rhythm and the rhythms of the world around them."

Dance Movement Therapy
The definition for Dance Movement Therapy (abbreviated as DMT) from the Association for Dance Movement Therapy, UK (ADMT UK 2002, p. 1) is: the psychotherapeutic use of movement and dance through which a person can engage creatively in a process to further their emotional, cognitive, physical and social integration.
It is important that DMT is differentiated from therapeutic dance because although the two disciplines are similar in many ways, there are key differences which Meekums (2002) outlines in her book.According to Meekums (2002), the key difference is that therapeutic dance can be practiced by highly skilled and talented dance teachers in many different environments, however, the teachers are not trained therapists.The concept of 'movement metaphor' (Halprin 2003, Meekums 2002) is quite central to the therapeutic practice of DMT.Movement metaphor is "a symbol encapsulated in either a movement or posture" (Meekums 2002).In DMT, movement metaphors are used as a way of communicating nonverbally between the therapist and the client.It can provide valuable insights into a person's patterns of behaviour, beliefs, relationships and emotional states (Meekums 2002, Ellis 2001).

Gesture Recognition
Gesture recognition researches human gestures for human-computer interaction.Gestures are body motions and physical movements from the fingers and hands, arms, head and face (including facial expressions) and the body to convey information or interact with the environment (Mitra & Acharya 2007).This research has been quite active (Pavlovic, Sharma & Huang 1997) over the years as it eliminates interaction devices and enables natural interaction through our own bodies.Many gesture sensing devices have been developed over the years and can vary on the recognition quality and feedback based on the device's accuracy, resolution, range of motion and latency (Mitra & Acharya 2007).Some of well-known sensing devices are: Microsoft Kinect V.2 motion sensor, LeapMotion and VicoVR.

Biofeedback and Applied Psychophysiology
Biofeedback has been recognized as a technique for "therapy" and "intervention" from the late 1960s for physiological, emotional, social and self-regulation to improve one's health and physical performance (Rosenbaum 1989, Mayo Clinic 14 January 2016.).It is, as the term suggests, the process or group of procedures where electronic recording equipment is used to obtain immediate and continuous information about physiological responses such as skin temperature or blood pressure (Rosenbaum 1989, Stern 2001, Mayo Clinic 14 January 2016).The main biofeedback measures are: brainwave frequencies through EEG (electromyography); cardiovascular measures such as HRV (heart rate variability), IBI (interbeat internal), BP (blood pressure) and BVP (blood volume pressure); EKG/ECG (electrocardiograms) that measures the electrical activity of the heart; respiration which measures the breathing rate and amplitude; EMG (electromyography) which measures the electrical activation of the muscles; and EDA (electrodermal activity)/EDR (electrodermal response)/GSR (galvanic skin response) which measures the changes in perspiration and electrical conductivity of the skin.
Psychophysiology is the science of studying the causal and interactive processes of the physiology, behaviour and one's subjective experience (Rosenbaum 1989).It studies of the reciprocal relationship and the interactions between psychology (brain) and physiology (body).As a form of applied psychophysiology, clinical biofeedback can help people modify their behaviours through learning to regulate their physiological feedback (Stern 2001).There are many biofeedback devices both commercially and for research purposes on the market today (i.e.Thought Technology Ltd., Emotiv EEG Headset, Empatica, Heart Math).

Therapeutic Applications for Virtual Environments
Virtual reality and its applications in therapy and healthcare have been rapidly expanding and becoming more commonplace in the medical fields.In VR therapies, simulated virtual environments are created to help individuals learn how to overcome predominantly psychological disorders such as phobias (i.e.Spider phobia: Hoffman et al. (2003)) and PTSD (post-traumatic stress disorder).VR technologies can create safe non-pharmacological, simulated environments for individuals to carry out therapy.These forms of therapies can be seen as extensions of psychotherapy as many of these therapies include therapies in the field of psychology and mental illnesses.(i.e.Schizophrenia: Freeman (2008)).Several other forms of therapy are for "training" the user to reduce stress or for rehabilitation (Kim 2005).Research integrating varied forms and concepts of expressive arts therapy in the VR realm is being carried out from music therapy (Optale 2001), entrainment (Papagiannakis et al. 2015) and movement therapy (Park & Park 2016).

Gestural Input and Psychophysiological Measures in Virtual and Immersive Environments
Research integrating psychophysiological measures (El-Nasr, Morie & Drachen 2011) and gesture input for interaction in VR and immersive real-time environments have been steadily growing over the years.An active community of developers are contributing to gesture-based interaction for VR applications on the Leap Motion gallery site.Leap Motion provides natural gestures (i.e.pinch draw, pinch grab, tap, throw, swipe) that are easy to learn for both VR and interactive desktop settings.At our iVizLab, Wang (2012) created a digital drawing tool using body movements and gestures to draw in space for art therapy.Kinect V.1 motion sensor was used for gesture and movement recognition which was fed into the system and projected on the wall.Studies showed positive implications for using digital tools to support art therapists.

Virtual and Interactive Space for Art Installations
Recently, more and more artists are creating VR art installations in immersive spaces and for HMDs with visuals and content departing from traditional gamelike 3D forms to much more organic, amorphous 3D forms.Many artists are taking inspiration from nature itself as we can see in Nature Abstraction by Matteo Zamagni (Zamagni 2017) and Miguel Chevalier's Complex Shapes (URDesign 4 May 2016).In Zamagni's VR film installation, fractal nature patterns are endlessly created with music creating a meditation-like experience.This VR film installation, which was originally created for VR HMDs, was readapted for Times Square's electronic billboards recently and displayed each night in March 2017 from 11:57PM to midnight.

SYSTEM FRAMEWORK
In this section, we present our early system framework for our bio-responsive VR system (see Fig. 1).There are four main components: 1) Immersant's Body Data, 2) Real-time Sensor Technology, 3) 3D Environment and 4) Bio-Responsive Immersive Space.The components are described in more detail below.

Immersant's Body Data and Real-Time Sensors
In our system, the human body is the main interface into our system.Different forms of body data are fed into the system in real-time using the Microsoft Kinect V.2 motion sensor and the Empatica E4 biosensor watch (see Fig. 2).The types of body data we are acquiring from the immersants are: gestures, body movements and psychophysiological measures -specifically EDA and IBI.

Microsoft Kinect V.2 Motion Sensor
Currently, the Microsoft Kinect V.2 motion sensor is connected to our Windows 10 machine and the gesture/movement data is processed through the Unity3D game engine.A Unity3D plugin from RF Solutions detects the Kinect V.2 sensor information which can analyse player's common gestures such as lean, move, wave, jump, squat among others.
There are approximately 24 gestures that can be recognized through the Kinect MS-SDK plugin.The main movements and gestures we are using are: user detection, moving sideways/forward and backward and wheel motions using hands and arms.

Empatica E4 Biosensor Watch
Empatica E4 biosensor watch is a medical grade wireless biofeedback device capable of reading

Amorphous 3D Environments
One of the main research areas of our lab is in generative AI art systems and real-time behavioural systems for 3D avatars.Over the last 3 years, we have been extending our research in generative art into 3D immersive space (Song & DiPaola 2015) and have been heavily experimenting with various 3D forms in conjunction with real-time biofeedback loops to control the 3D visuals.Our user studies showed that the art generated through our AI system ePainterly elicits emotions (Salevati & DiPaola 2015).The art is amorphous, with no clear shapes or boundaries.Through numerous iterations, we started to translate the AI artwork into the 3D space through amorphous 3D forms that started to respond organically to the psychophysiological measures (see Fig. 3).This work serves as the foundation for the design of our 3D environments and further, we have taken inspiration from nature-inspired cell structures, the human form and non-rigid particle entities as the basis for the generation of the 3D space.

Nature-Inspired Entities
The cell structures for our 3D environments are based on Voronoi diagrams which can be observed in nature around us.Two such examples are the patterns on a giraffe's skin and a dragonfly's wings.

Figure 4: Voronoi Diagrams on a Giraffe and Dragonfly Wings
To generate the Voronoi sphere cell structures, we are using Grasshopper 3D which uses scripts and generative algorithms to create 3D forms and objects.It sits on top of the Rhino3D software.Through Grasshopper3D, we are able to quickly change parameters of the models mathematically.From Grasshopper3D, we exported the models into Unity3D.

Human Forms and Movement
To create accurate human forms and movements, we are using MakeHuman and exporting the models to Adobe Mixamo.Mixamo is an online software for rigging and animating 3D humanoids and has a builtin library for natural human motions and movements (i.e.walking, sitting, dancing).The library of animations we created for our system are: sitting, standing, breathing, standing up, sitting down and lying down.

Particle Systems
To generate real-time particle systems from the 3D models, we are using PopcornFX.PopcornFX is a VFX software developed specifically for real-time game engines (i.e.Unreal and Unity3D).Once our particle systems were created, they were exported to Unity3D.

Bio-responsive Immersive Space
Our bio-responsive VR and immersive space is created and displayed on the HTC Vive HMD or a large projection screen or monitor.Both displays require physical space for the immersant to move around.The 3D visuals are controlled through direct mapping of the sensor data to the 3D forms in Unity3D.Below, we discuss the space and mapping specifics for our system.

Microsoft Kinect V.2 Sensor Setup
The minimum distance requirement between the immersant and the Microsoft Kinect V.2 is 1.8m.The sensor needs to be between 0.6m and 1.8m off the ground and it needs to be facing the immersant directly.For the sensor to work properly, it needs to see the entire body (see Fig. 5, left).For our setup, we mounted the Microsoft Kinect V.2 to our projection screen in a physical floor space of approximately 3.2mx3.5m.Our space allows for full body detection of multiple immersants (up to 6) and audience participation.

HTC Vive Room-Scale Setup
The HTC Vive HMD and controllers require more space than the Microsoft Kinect V.2.The two lighthouses track the immersant from the outside, allowing the immersant to walk around freely in the mapped space.The recommendation for the physical space for the HTC Vive is 5mx5m (see Fig. 5, right).The lighthouse technology (Buckley 19 May 2015) provide accurate, high resolution tracking that is able to detect the immersants and their movements and map it to the 3D space that the immersant experiences on the HMD.The virtual 3D space inside the HMD that is displayed can be of any dimension from an office to infinite planetary space.
For our setup, we are using our 3.2mx3.5mphysical floor space.The virtual 3D space consists of Voronoi spheres that are quite vast in size -ranging from 3m to 5m in height and width -that the immersant sits or stands inside of.

Data Scaling and Mapping
Being able to interact with the 3D environment through body movements, gestures and biofeedback can create powerful embodied experiences.However, if at any point the movements and gestures or biofeedback data is not recognized, it will just as quickly break the embodied experience and leave the immersant feeling disconnected and disjointed.To create meaningful experience through a tightly coupled feedback loop, the virtual space and real space need to be scaled and mapped from the sensor data to the 3D forms and environment.The axis and range of mapping the immersant's body data is currently in its infancy as we are iteratively testing the mapping of movements, gestures and biofeedback data.Currently, we have identified the following for our prototype systems.

Movement Mapping
We have limited the movement and gesture mapping to gross movements and gestures to ensure correct and immediate feedback for this iteration.For the initial mapping, we have tested two different movement/gesture scenarios with the same particle human animation.The mapping schema is based on our experiments (see Table 1.).In the first scenario, when the immersant walks into the immersive area and she/he is detected, the particle human animation of standing up starts immediately but with a slower motion so that it is mapped to the entire motion of the immersant entering and standing in front of the screen.When reversed, we scaled back the animation to normal and mapped the animation trigger of the particle human sitting when the immersant was lost from the Microsoft Kinect V.2 sensor.In the second scenario, we tested the wheel gesture.For this gesture, the particle human's animation was trigger at the end of the gesture with the slower motion.For the reverse, when the immersant started the wheel gesture, the particle human started to sit down.Irrespective of the gestures and movements, when the immersant stayed idle in front of the sensor, we had the particle human standing and breathing in place with a much slower breath animation.

Biofeedback Mapping
For our research, we are exploring streaming EDA and HR data.EDA data processing for real-time applications can be tricky as there is a delay in the EDR.HR data is streamed as beats per minute (BPM) and is more immediate.For the biofeedback mapping, we are acquiring the IBI to map the cycles between each beat to the 3D entities.The movements on the Voronoi spheres are generated through real-time Perlin noise algorithms.We chose to apply Perlin noise to our Voronoi spheres to emulate a more naturally-occurring, undulating and wave-like phenomenon.The code allows for movements to occur in all 3-axis or be constrained to any combination of the 3-axis.For the purposes of prototyping and testing, we have constrained the movement to the Y-Up-axis.The mapping from the IBI to the Voronoi sphere movements is not a direct 1:1 mapping of minimum and maximum values.Because there is a lead time for putting on the Empatica E4 biosensor watch and the time for the system to calculate the baseline, the VR space the immersant initially enters is dark while the baseline is calculated.The Voronoi spheres start to gradually appear into the visual field as the immersant's IBI is mapped to spheres.Currently, we are experimenting with mapping the IBI cycles to the cycles generated by the Perlin noise algorithms on the Voronoi spheres.

Syncing
When the syncing occurs between the immersant and the 3D visuals in the bio-responsive space through the synchronization of biofeedback data and/or gestural and movement data, the immersant can achieve a sense of attainment where she/he feels a oneness with the real-virtual environment and the body interface becomes an extension of the 3D visuals and vice versa.Through this, the immersant can start to create his/her own embodied experiential space.

EARLY PROTOTYPES
Multiple virtual and immersive systems are currently being developed under our framework for therapeutic, meditative and artistic purposes.We present two work-in-progress prototypes here.The first work-in-progress prototype (see Fig. 6) is an installation where multiple people (up to 6 at any given time) can enter the interactive space where the human forms on the screen will start to react by standing up or sitting down, exploring the boundaries of real-virtual space.The second work-in-progress prototype explores experiential bio-responsive cell structures in immersive VR where immersants are encapsulated inside Voronoi-cell structural domes that are constantly pulsating to the immersant's own heart rate (see Fig. 7).This prototype will be further developed for therapeutic and meditative applications.It is currently being extended to the mobile WebVR platform to be viewed on the Google Pixel Phone and Daydream Viewer.

TEMPORARY CONCLUSIONS AND FUTURE WORK
In this paper, we presented our early system framework for a bio-responsive VR environment and interactives.There are 4 main components to the framework: 1) Immersant's Body Data, 2) Real-time Sensor Technology, 3) 3D Environment and 4) Bio-Responsive Immersive Space.In generating the 3D visuals and forms, we have taken inspiration from our own lab's work in generative AI art systems as the foundation to move onto nature systems and structures as well as non-rigid amorphous 3D forms.By integrating current sensing technologies such as the Microsoft Kinect V.2 motion sensor and the Empatica E4 biosensor watch, our framework aims to design affective-loop VR systems and immersive environments that can be applied to many experiential application spaces.Over the next couple of months, we will be refining and further integrating HRV research on emotion and the heartbrain connection from the Heart Math Institute (McCraty 2015) to our framework.With the maturity of in-VR creation tools tied to current sensor technologies, multimodal therapeutic applications will soon be a reality where therapists can use bioresponsive VR spaces to track their client's physiological, spatial and emotional states in realtime while their clients create art and music and move seamlessly through the real-virtual worlds.

Figure 5 :
Figure 5: Kinect V.2 & HTC Vive Physical Setup (Images from Xbox Online Manual and HTC Vive User Guide)

Figure 6 :
Figure 6: Particle Human Forms Responding to the Immersant

Figure 7 :
Figure 7: Viewing the Bio-Responsive VR on the Daydream Viewer

Table 1 :
Movement and Gesture Mapping