An Evaluation of Different Strategies to Represent Performance in a Tropical Garden for Immersive Environments

Prior to its extensive redevelopment, the authors captured a tropical garden with significant cultural heritage and reconstructed it for immersive environments. Two approaches were employed; 360° video panorama recording and a photogrammetric capture as foundation for a complete 3D-reconstruction for VR. Common to both approaches, was the use of dancers as agents of user engagement and exploration. This paper evaluates the different methods of collecting the necessary data for each approach and their implementation of dancers in a cultural heritage context.


INTRODUCTION
In 2016, plans were announced to redevelop the historical Yunnan Garden, which is situated on the campus of Nanyang Technological University in Singapore. Constructed in the 1950s and inspired by classical Chinese gardens, Yunnan Garden holds cultural heritage value as one of the few remaining gardens of this style in Singapore and has been described as a "cultural gem" (Zaccheus 2016). In this paper, we evaluate different approaches in capturing a garden in VR and work with diverse methods to insert performers into the immersive environment as a way of enhancing user engagement.
Our team of artists/researchers initiated a project to digitally capture and preserve this cultural heritage site prior to its extensive redevelopment. Consisting of pavilions, garden structures, pathways, well over 700 trees, hundreds of plants and shrubs of a wide variety of species, Yunnan garden covers an area of approximately 3.4ha.
To keep the memory of the original version, we explored a range of capturing techniques. Capturing a garden is significantly different from capturing traditional immobile cultural heritage: plants with their leaves are not static, their colour, shape and size change over time, even daily. Their scale ranges from tall trees many metres high to tiny blades of grass only a few millimetres in width. The textures and detail of the garden's flora are central to the experience of being inside the garden, an experience that cannot be easily recreated.
In the following, we will introduce two of the different approaches that we explored and discuss their advantages and drawbacks. For the first stage of the project, a 360°-video recording with dancers performing in the garden was created, through which we could evaluate different modes of presentation from dome projection to headset (Reinhuber, Seide & Williams 2018). In the second stage, we decided to build a room-scale VR representation of the garden. While we considered laser scanning (LiDAR) as part of the capturing process, we settled primarily for photogrammetry for its fidelity to detailed textures of the trees and plants.
To cater for the different scales, drone images served to obtain a scale map of the garden and to determine the exact location of major components, while the details, such as the trunks of specific trees or stone sculptures, were captured with detailed DSLR images. This footage constituted the foundation for a VR room-scale experience that might serve as a container for different approaches such as gamification, artistic or educational and informative applications. Multiple versions of the VR experience have been created but for better comparison, only one version will be discussed here. We will focus on two artistic interpretations of the garden, both utilising performers to provide a human focal point and each having connection to the four cardinal directions, inspired by Chinese mythology and a defining feature of the garden.

CAPTURING THE GARDEN FOR DIFFERENT APPLICATIONS
While technology for immersive media has developed rapidly in the past four years, we could only estimate the outcome of our project. Our idea was to preserve the original garden as a virtual replica with the intention to transfer the impression of being in the garden and meander through its forking path structure with pavilions, monuments and lush greenery.
Therefore, as soon as we learned that the site would be undergoing major changes, we quickly decided to capture the garden with a range of imaging technologies that were accessible to us at that time in 2017. Imagery from 360°-monoscopic cameras, a camera drone, DSLR cameras for panoramas, timelapse walk-through, macro and wide-angle shots served later as reference for the reconstruction of the room-scale VR experience. Although the initial aim of our research was to find strategies to preserve the garden according to its cultural heritage status, the original imagery inspired our artistic endeavours. Therefore we decided to represent the concept behind the garden with performers. A 360°-video would give a good representation of the sense of being in the garden with its forking path structure -no predefined orientation as in a linear movie, but with the possibility to explore the surroundings in all directions and to follow any of the four dancers.
The garden, with its adjacent buildings, was laid out using feng shui principles (Urban Redevelopment Authority for Preservation of Monuments Board 1998), in particular in accordance with Chinese lore in which the four cardinal directions are represented through colours and animals: North -a white tiger, East -an azure dragon, South -a vermilion bird and West -a black tortoise. With this concept in mind, we collaborated with choreographer Susan Sentler and four performers.
The 360°-video allows the audience to observe the surrounding with three degrees of freedom (3DOF), offering rotational movements such as pitch, yaw, and roll from a single standpoint. However, our aim was to create an even more immersive experience with six degrees of freedom (6DOF), providing translational movement as well along the x, y, and z axes, understood as moving forward or backward, moving left or right, and moving up or down. The reconstructed VR-model of the garden in this way invites the viewer to walk around and explore the surrounding from any perspective. Inserting performers into the VR garden provides another challenge as the viewer does not have one specific viewpoint. Ideally, performers are captured with volumetric video through a camera array, pointing to the moving subject in the centre. In this way, the viewer is able to see the movement and body from any direction while walking around and exploring the VR-world. In contrast with large production houses, we had to work with a small team, limited budget and time constraints. Therefore we decided to capture the performers in the university's motion capture studio.
In the following, we will describe the different approaches for capturing the 360°-video and VR room-scale experience.

Immersive 360 approach
Based on our early experience with QuickTime VR for stills and video through meticulous capturing and unnerving stitching, we were fascinated by the ease of use of the 360°-point-and-shoot cameras such as the Ricoh Theta S. Two opposing extreme wide-angle lenses capture the whole surrounding spherically with little parallax due to the slim camera body; this proved to be a good starting point for obtaining an immediate impression of the garden.
However, low resolution and extreme compression artefacts did not suffice to meet our high expectations. To achieve a resolution of 8K for the spherical video, we worked with three GoPro cameras, which were retrofitted with extreme-wideangle lenses. Each of the cameras recorded in 4K resolution, while the compression could not be avoided at this stage.
The actual performance and shoot was a new challenge for all participants: everything was visible in the spherical movie stage and there was no framing for which the performance could be rehearsed. At the same time, we had no control screen for the three camera set up to indicate if the performance was satisfying or another take required. Also, reviewing the recorded material provided an approximation as we could only evaluate one third of the footage, but never the fully stitched spherical video. While hiding behind trees or stones, we had to rely on the gut feeling of the performers and the previously stitched results from our rehearsals.
Despite the immediate and fast possibility to record an impression of the garden, the veritable process of stitching the footage was rather time-consuming and we were only able to present the first version of the 360°-video-short Secret Detours half a year later, by the end of 2017. One of the challenges was the three stitch lines through which the dancers kept moving and software supporting 360°video stitching in a high resolution of 8K was not easily available.

Virtual reality approach: interactive roomscale VR experience
Our second approach to representing an artistic dance performance in Yunnan Garden examines the use of a 3D-environment combined with performance motion capture, aiming at producing a result that can be explored as an interactive roomscale Virtual Reality experience. The technical challenges are many and diverse, from creating a realistic environment to representing the motioncaptured performance adequately as an artistic expression in the virtual environment.

Yunnan Garden as virtual reality environment
The process stages of recreating Yunnan Garden as a realistic virtual environment can be described as follows: • Researching all available plans and maps • Acquisition of photographic references • 2D reconstruction of an accurate orthomosaic map from aerial survey • 3D reconstruction of static structures from photogrammetry • Approximate 3D recreation of vegetation such as grass, trees and plants In the following, we will describe key considerations and findings of some of the above processes.

Orthomosaic map
An orthophotograph, or orthomosaic map, is a geometrically corrected aerial photograph that represents the Earth's surface accurately and can be used to measure true distances. We used orthophotography for our project as a reference for measurements, and in conjunction with a highly detailed topographic and planting map to manually position objects such as plants, trees, walkways and pavilions.

3D reconstruction
In addition to the aerial survey, a terrestrial data acquisition was conducted to create 3Dreconstructions from photogrammetry. For our project, we used the software Agisoft Metashape to create 3D-models of seven stone rock statues, static structures such as the monument and several prominent tree trunks (Figures 2 and 3).

Approximate 3D recreation of vegetation
While the process of photogrammetry is ideal for hard-surface static objects, it is unsuitable for small crisscrossing tree leaves, which produce an almost infinite-appearing number of occluded areas. Therefore, we decided to use digital plant assets, which are optimised for real-time projects. Even though these plants were carefully selected and matched according to plans and photos; and replicating variety, position and size as close as possible. Ultimately the garden shown is not a true 3D reconstruction but an approximate recreation.
A detailed report of the processes and issues involving the approximate recreation of the garden's vegetation in 3D can be found in our paper on the topic (Seide, Williams & Reinhuber 2020).

Performance capture
Our university's research facilities offer a Vicon motion capture system, set-up as a circle of 14 cameras at a height of 2.5 metres. This camerabased set-up can produce high-quality results, but often introduces errors at ground level when parts of the feet are occluded, a limitation of the onecircle set-up, requiring a time-consuming clean-up post-process of the captured data. At the time of writing, our facilities have acquired a lower-cost system from Rokoko, which in contrast to the camera-based Vicon system, is an inertial sensorbased system. Skogstad and Nymoen analyse both concepts and conclude "If high positional precision is required, OptiTrack (a camera-based system) is preferable over Xsens (a sensor-based system), but [...] Xsens provides less noisy data without occlusion problems" (Skogstad, Nymoen & Høvinet 2014).
Comparison of the two concepts for the research project is ongoing. Together with Choreographer Susan Sentler and motion-capture expert Biju Dhanapalan, two dancers and four performances were captured with the Vicon system and postprocessed in Autodesk Motionbuilder.

Flux Garden: an interpretation of the garden
While working to recreate an accurate and realistic representation as a VR experience, we revisited the actual garden and were confronted with the construction site: a disassembled remembrance of what was once a green locale of recreation and enjoyment of nature. Structures such as the pavilions, monument and gate were still intact but most of the flowers and many of the trees and shrubs had disappeared. It became apparent to us that we needed a visual representation for this new reality: the transformed and fragmented version of the site. The state of the garden at this time was that parts were still there while others were gone. This confirmed our decision to experiment with a combination of abstract and realistic elements, leading us to merge our accurate recreation with a low-resolution 3D reconstruction of the entire garden, generated from a limited data set with photogrammetry acquired through aerial survey. That insufficient data generated a roundish, almost abstract landscape, resulting in our VR experience "Flux Garden" (Figure 7), a partially realistic representation, dissolving into an impressionistic dream-like memory of the garden.
In the Flux Garden VR experience, the audience can freely walk and teleport to explore the environment. At first, the audience find themselves in a realistic replication of Yunnan Garden, which then over time transforms into an impressionistic fragmentation. The roundish forms of the abstract landscape can be entered by walking or teleporting right through the structure. Inside the forms, the audience find themselves in a dark cave-like environment filled with a particle field that can be disturbed by hand movement. In the darkness of the cave, the audience can observe the expressionistic performance of a dancer represented by a slowly disintegrating and then accumulating particle field (See Figure 8). The dancers are not representing directions as in Secret Detours, nor are they evenly arranged according to the cardinal directions. In Flux Garden, the harmony of the path structure and cardinal directions appears disturbed; 13 variations of performances and colours are scattered over the fragmented landscape, inviting the audience to meander and explore its secrets.

Figure 7/8: Dancer as particle simulation in Flux
Garden/Flux Garden environment.

Music and sound
With both approaches we are comparing dancers, the nature of their implementation dictated how sound and music would be deployed. We decided that the dancers would be accompanied by original music, composed for the performance. The 360°video is a linear work and so has a fixed duration. The user has little agency and is restricted to choosing the direction in which to look while the ambisonic sound they hear changes slightly with the viewing direction. All users will receive very close to the same sound world and will experience it for the same amount of time. In this work the music serves a strong narrative function.
Being non-linear, the VR-version has no fixed duration and much greater interaction. Users have greater freedom to navigate through the garden on a path of their choosing and to linger in one location while skipping others entirely. Sounds or music mapped to specific locations may or may not be encountered by the user, meaning each user could potentially experience quite a different sound world compared to others. Sound, and music in particular, was designed to encourage the user to explore the entire garden.
Our approach was to employ sound in much the same way as in a regular game. Multiple ambiences define larger zones within the garden, while specific locations are embedded with their own unique sounds and music. Four dances were captured and versions of them placed in 13 locations around the garden. Each dance has its own music at each location, so although the dances are repeated (with a different particle colour) the music is unique to the location. The veiled amorphous nature of the particle dancers renders their movements somewhat unclear and more receptive of variations in synchronisation with music. This allows the application of different music to the same movement to give the impression of a completely new dance and greatly saving the time needed to capture and render a new one. While musically related, the mood and style of the music varies considerably and is designed to encourage exploration of the whole garden. In contrast to the 360°-video of the first approach, the music does not have an overt narrative function but suggests a journey, a narrative progression through variation. It's a journey with multiple possible directionsreflecting the forking path structure of the garden.

DISCUSSION AND CONCLUSION
We have evaluated and compared two quite different approaches to capturing a tropical garden. While different in technique, they shared the same motivation: to capture an artistic interpretation of a site of cultural heritage value and to encourage its exploration by the user. The employment of dancers as agents for audience engagement and exploration became central to our approach to these works.
The challenges of capturing dance performance for incorporation in the two works differed considerably. The linear 360°-video of the first approach created challenges in several regards. The placement of the dancers was challenging as they could not be too close to the camera array to avoid major stitching issues while moving between the three camera lenses. The extreme wide-angle lenses of the GoPro cameras caused the size of the dancers in the frame to vary considerably with small changes in proximity to the camera. Coupled with a lack of possibilities for previewing the spherical result or easily viewing a playback, these factors posed difficulties in achieving optimal framing and blocking of the dancers. As a consequence, we were not able to utilise the spatial aspects of the garden to the extent we would have liked and encountered issues when the dancers crossed between cameras in the array. Although the process of stitching the footage from the cameras to create the 360°-field was timeconsuming, the process of capturing the dance performance was relatively short. Once captured, the dancers are fixed in space and their performance is unchangeable and the user can only view them from a single perspective. The work is linear; it has a beginning and an end with matching soundtrack and is limited to 3DOF, which only allows the viewer to look around, up, down and tilt the head but not to walk around in the virtual representation. This is the most visually realistic presentation of the garden with the dancers serving as the principal agents in fostering narrative engagement.
Employing 3D-motion capture in Flux Garden to record the performance of the dancers and placing them in a virtual recreation of the garden had different challenges to those we encountered in the 360°-video. Creating 3D simulations of the dancers from the motion capture data is time-intensive but allows for near limitless possibilities and flexibility in their use in the VR-Garden. For this work, the dancers are represented as a changing particle field and this aesthetic decision had the benefit of saving considerable time but creating fully-formed representations of the human form. The nonlinear work allows for 6DOF, so the user can view the movements from any angle or perspective. Discovering the dancers hidden in the green elements allows the viewer an interpretation of the culturally significant garden.