The Virtual (Reality) Museum of Immersive Experiences

This paper outlines an investigation and describes strategies to capture, simulate and reproduce experiences originally designed for large-scale immersive architectures within Virtual Reality. Applications and experiences created for a specific immersive platform depend on the complex and costly technical infrastructure they were originally designed for. Descriptions and video documentation only go so far in illustrating an immersive experience. The embodied aspect, the emotional engagement and the dimensional extend, central to immersion, is mostly lost in translation. This project offers a prototypical implementation of a large-scale virtual exhibition, incorporating various immersive architectures and applications situated within a fictional 3D scene. The motivation behind this project is to provide a framework to showcase and for the conservation of immersive experiences and systems outside specialised facilities and labs. Furthermore, it presents a test-bed and space for experimentation to design and evaluate immersive experiences and architecture before they are developed at full scale.


Immersive
environments, architectures and systems (thereafter 'immersive architectures') have been utilised in simulation, visualisation, entertainment, the arts and museological context for a long time before Virtual Reality (VR) made its resurgence only a few years back.As Kenderdine (2010) observes 'These immersive architectures and their associated visual, sonic and algorithmic techniques offer compelling means for mapping and remediating the tangible, intangible and abstract aspects of culture and heritage landscapes'.
Applications and experiences created for a specific immersive platform depend on the complex and costly technical infrastructure they were originally designed for.Descriptions and video documentation only go so far in illustrating an immersive experience.The embodied and kinaesthetic aspects, the emotional engagement and the dimensional extend, central to immersion, is mostly lost in translation.This project offers a prototypical but fully functional implementation of a large-scale virtual exhibition, incorporating various immersive architectures and applications situated within a fictional 3D scene.The target platform is a VR system with spatial tracking and controller (Vive).
The selection of immersive architectures and applications originated from projects the author was personally involved in over the last 15+ years at UNSW iCinema Centre and ZKM Karlsruhe, either as lead artist, in the design or conceptual development.The motivation behind this project is to provide a framework to showcase and for the conservation and preservation of immersive experiences and architectures outside specialised facilities and labs.Other use cases for the framework developed for this prototype are as a test-bed for future immersive projects.It provides a space for experimentation with content and display architecture before they are developed at full scale.The simulation gives a good sense of how a design works regarding scale, field-of-view, peripheral vision, audience viewing position and perspective.

BACKGROUND
In his seminal essay The Work of Art in The Age of Mechanical Reproduction, Walter Benjamin (1936) reflects on the loss of the aura through the mechanical reproduction of art.The aura represents the originality and authenticity of an artwork.A painting has an aura while a photograph of the painting does not.According to Benjamin, the sphere of authenticity is beyond the capacity of technical reproducibility and withers in the age of mechanical reproduction.However, Benjamin also states the potential of technological reproduction: [T]echnological reproduction can place the copy of the original in situations which the original itself cannot attain.Above all, it enables the original to meet the recipient halfway, whether in the form of a photograph or in that of a gramophone record.The cathedral leaves its site to be received in the studio of an art lover; the choral work performed in an auditorium or in the open air is enjoyed in a private room.
Applied to the context in this project, this observation holds true.It allows immersive media applications to be experienced outside the institutional context and has the potential to reach a broader audience.
In postmodern culture, the question of authenticity of time-based art works is in general more complex then with traditional art forms.Digital video and computer-generated imagery do not constitute an original in the sense of a painting, or as Lippard (1997) put it the 'Dematerialization of the Art Object'.The originality of such an artwork derives from the content and the context.A VR reproduction might use the exact same media files or computer code as in the actual artwork, it represents an 'authentic' one-to-one copy.In Baudrillard's (1981) words 'The real is produced from miniaturised cells, matrices, and memory banks, models of controland it can be reproduced an indefinite number of times from these.'A Virtual (Reality) Museum of Immersive Experiences should focus on providing a context for the digital time-based works.
Research by Falk & Dierking (1992) shows that from the visitor's perspective the museum experience consists of three contexts: the personal context which incorporates personal experience, knowledge and motivation; the social context which refers to the social environment in which the visit happens; and the physical context which relates to the architecture of the building as well as the objects contained within.A VR reproduction does not influence the personal context of a virtual visitor so it needs to focus on the physical context.What is missing is the social context, which plays a fundamental role during a visit to a museum (Economou 2008).Speculative scenarios to address the lack of social context during a visit in this VR exhibition could be: A shared VR experience where multiple visitors enter the virtual museum at the same time.Similar to a social VR application, the visitors are represented as avatars and have the ability to talk to one another.Zizza (2017) explored the notion of social VR in a learning environment and participants expressed a higher level of enjoyment and presence in a social VR setting.(ii) A system to record voice commentary, for instance personal observations about the works on display.These voice recordings are spatially situated within the VR scene and can be accessed by a subsequent visitor.This would make it possible to engage in an off-line conversation with other people.Schweibenz (2013) analyses the complex relationships between the real and the virtual museum, between object and reproduction and between mediated and unmediated museum experience.He observes that in recent museological discourse, virtual visits are considered as a secondary and surrogate experience to the physical visit, and continues: This discussion should be closed by now as it seems obvious that visiting experiences inperson and online are different, both having their own strengths and weaknesses, nevertheless being experiences in their own rights.
His focus is on online portals for museum collections, and he does not consider the experiential quality VR can afford.But this observation still applies if a physical visit is compared to a VR experience.Instead of trying to answer the question to what extend a VR reproduction can deliver a compelling experience if compared to the actual situated work, I consider the VR simulation an experience in its own right.It is not a substitute for an actual visit, but it meets the recipient halfway.

VIRTUAL (REALITY) MUSEUM
The term Virtual Museum often refers to online portals of museum collections or pretty much anything a museum does in terms of public engagement outside its physical manifestation.Panciroli et al. (2007) proposes the definition 'A virtual museum is a digital entity that shares some features with the traditional museum, being accessible to an audience, referring to a cultural asset and with educational intent.'However, there is a significant difference between browsing an online database and a spatial VR representation.The former is an educational information space whereas the latter has an experiential quality for the viewer.In 1996 the author developed an interactive CD-ROM for the Vitra Design Museum (Figure 2), spanning 150 years of furniture design.In this project, a viewer can navigate a photographic representation of the museum in form of interactive 360º panoramas (QuickTime VR) and access archival material about the objects on display.This model is located between the aforementioned information space and the VR representation, it allows for spatial navigation.This model is now widely adopted, amongst others by the Google Cultural Institute, an online platform from which the public can access artworks, collections and places from around the world.It incorporates Google's Street View technology, enabling users to visit these institutions through a digital 360º photographic representation.This follows a trend to democratise and open up culture to a global audience.An artistic work taking the notion of a virtual museum more literal is Jeffrey Shaw's 1991 Virtual Museum (Figure 2).A motorized revolving platform holds a large monitor and a chair from which the viewer is able to interactively explore a virtual 3D exhibition.In it, the virtual space is a replica of the actual gallery space whereas the artworks are immaterial.This is a reversal of the model described in this paper, where the artworks are reproductions and the gallery space is fictional.
With the medium VR, the distinction between a 360º photographic/videographic (cinematic VR) representation and a real-time interactive 3D world is not jet fully adopted.There are fundamental differences in terms of presence, interactivity and the immersive nature of an experience for a viewer.One could compare this to the difference of watching an action movie to playing a computer game.Real-time VR affords a viewer to navigate freely inside the 3D world whereas in cinematic VR the viewer is restricted to an inherently passive role (Table 1).The VR experience described in this paper is one of only a few attempts to create a 'realistic' simulation of a museum and the only one focused on immersive applications in the arts.The Steam VR distribution network is a good starting point to get an overview of existing VR museum applications outside of academia, the count is currently under ten.

Sound
Pre-recorded and mixed.
Real-time generated or recorded.Optional HRT and position.

Immersive architectures
If compared to traditional screen based media, immersive architectures provide a delivery platform for ultra-high resolution digital content at a realworld scale and for multiple simultaneous viewers.This makes them the ideal stage for impactful experiences in public museums, festivals and exhibitions.Kenderdine (2010) states: [T]he design of the immersive systems demands that people ambulate and circumambulate, continuously re-orienting themselves in relation to real-world scale imagery of augmented virtual landscapes.The acoustic spaces in these systems are dynamic in relation to the positioning of visitors and their movements.
Participants interact and perform with both the imagery of the virtual world and with the other people who co-inhabit the space.The issues of inhabiting the immersive space and its relationship to real environments and the performative qualities of people within these hybrid and multimodal spaces are of central concern. 206 The kinaesthetic and the acoustic aspect of a viewer in relation to the immersive architectures translate directly into VR.It affords navigation and gaze direction control and the sound is position sensitive.The immersive architectures included in this VR application are (Figure 3).

EpiCylinder
Designed at the UNSW Expanded Perception and Visualisation Interaction Centre in 2016.It consists of four rows of fourteen slim-bezel DLP rearprojection display cubes (EyeVis EC-SLIM) each in full HD and 120Hz active stereoscopic 3D and a 32.1 ambisonic sound system.It is configured as a 340-degree cylinder.Resolution: 26880x4320 pixel, dimension: radius 3.2m, height 2.98m.

iDome
The iDome, developed at UNSW iCinema in 2004, is a proprietary platform that offers a cost effective and compact immersive visualisation environment for spherical representations.It is configured as a 3 or 5m fibreglass hemisphere, a single HD projector, a spherical mirror as reflection surface, 5.1 sound system and a user interface.Resolution: HD or 4k (Kuchelmeister et al. 2009).

ReACTOR
A hexagonal configuration with six rear-projected screens for stereoscopic 3-D viewing.Viewers stand outside the hexagon, and by moving around it are able to see a 3D virtual representation from six distinct points of view.Resolution 6 x 1280x720 pixel, dimension: diameter 5m, height 2.4m.

Turntable / Placeworld
A custom-built, 360º revolving projection platform, comprised of a suspended 2m diameter cylindrical rear-projection screen with a rotary platform at its centre.An active 3D projector with a wide-angle lens and a Mac Mini is fitted on the revolving platform.The operator manually rotates the platform, and therefore the projected image.This rotation of the virtual viewing window around the panoramic screen continuously reveals new and discrete sections of the image (Kuchelmeister 2013).

Experimental curved screen
A simulation of a vertical curved projection screen for stereoscopic 3D content (Figure 11).This architecture has not yet been realised.The variable 3D model allows for experimentation with scale, proportion, placement, type of content and stereo 3D separation.With this VR simulation, it is possible to evaluate the point-of-view, peripheral vision, depth perception, sightlines and context from a viewer's perspective.

Applications
The criteria for the selection of content for this functional prototype, are related to the availability and copyright of the material, and exclude work that relies on complex interactivity or real-time computer-generated imagery.This would have been beyond the scope of this implementation.Stereoscopic, spherical and panoramic video material is relatively easy to integrate and the majority of applications do not provide user driven interactivity beyond the selection and subsequent playback.In all of the applications on display, the author was personally involved as lead artist or in the production.This allowed direct access to the original media files from the authors' digital archive.A total of 23 immersive projects distributed over the six primary immersive architectures are on display.
There are also two VR projects accessible, they are represented as VR headsets inside the simulation (Figure 13) and play in full screen after selection.An additional 11 conventional screen based works, 5 of them in stereoscopic 3D are part of this virtual exhibition.The total duration of the video material is 117 minutes.
The Virtual (Reality) Museum of Immersive Experiences Volker Kuchelmeister 207

IMPLEMENTATION
The Virtual (Reality) Museum is implemented in the game-engine Unity 3D.Unity has excellent support for various VR platforms and as of the current version (2017.1) it supports video textures up to 4k.Both are crucial features for this application.The project was realised exclusively by the author, over a timeframe of about 3-4 weeks in 2017/18 and with a minimal budget, however with pre-existing resources such as a VR-ready computer and a HTC Vive headset.With only limited experience in Unity 3D, the open structure helped to implement the functionality and plug-ins and code-samples make Unity very accessible.The main challenges were the stereoscopic 3D video playback in VR, controller interaction and the complex lighting of the large 3D scene.

Context and 3D models
For aesthetic reasons and to provide a spatial context for the immersive architectures, the VR exhibition is situated within a fictional 3D scene of a large derelict warehouse/factory.A place that is reminiscent of the ZKM Karlsruhe 'Hallenbau' (Figure 4).The ready-to-use 3D model was sourced in the Unity asset store.
The 3D representations of the immersive architectures are based on 3D models the author created during the design and development phase at the UNSW iCinema Centre.For performance reasons, 3D models in VR should have a small polygon count, and simplification of existing models was necessary, particular the relatively detailed physical user-interface models.The screen surfaces (plane, curved, cylindrical, hemispherical, spherical and hexagonal) were re-modelled as primitives with UV texture coordinates.What remains, is just simple frame geometries to reproduce the original configuration and to anchor the screen surfaces in the 3D scene.

User interface mapping and locomotion
An interesting challenge is mapping a physical user interface and the interaction modalities form the immersive application to VR. Commodity VR systems do not provide haptic feedback.However, mapping of a VR controller input to the simulated physical user interface is possible.For instance, the iDome utilises a large trackball as a means to control the gaze within 360º spherical imagery (Figure 6).In VR, the controller trigger button in combination with proximity and movement can simulate the rotation of the virtual trackball.Visual feedback is provided by the display system as well as the virtual trackball rotation.
Other user interface devices and systems for immersive architectures and applications such as marker and marker-less object and viewer position tracking, wands, touchpads, consoles with push buttons, pan-tilt devices and so on can be mapped in a similar fashion.In VR, the modalities include the controller and head-mounted-display spatial orientation and position and the controller button inputs, possibly in combination with a virtual representation of the physical interface device.Established conventions in VR for locomotion in a scene outside the room-scale tracking area by either teleportation or transportation do apply in this context (Figure 7).
A virtual laser pointer (Figure 7) attached to the controller in combination with the trigger button allows the user to perform selections, for instance selecting and launching a specific work inside the virtual exhibition.Visual feedback is provided in the form of a dynamic colour change of the 'laser beam' to indicate a valid selection target and its tip expands into a small sphere at the depth of the target.By observation, this proved to be an efficient and intuitive interface to interact within the VR application.During a demonstration session, for an audience already familiar with VR, no introduction to the UI functionality was required.

Stereoscopic 3D Media in VR
Immersive display systems frequently employ stereoscopic 3D representation, achieved by active or passive projection and screen technologies.Stereo 3D does not only provide binocular disparity and therefore depth perception, but is also an important tool to spatialise content.Objects and scenes within a stereoscopic representation are able to operate in front of the screen.They share the space with the audience and a more tangible and embodied experience.VR technology, with its inherent binocular design, is an ideal platform to simulate and demonstrate screen-based stereoscopic 3D content.The notion of negative, zero and positive parallax and its spatial relationship to the virtual screen surface stay intact in VR representation.Other stereoscopic properties such as perceived depth and scale do also translate directly.
One challenge with this approach is the technical implementation and adaption of stereoscopic 3D video content for VR viewing.3D imagery is usually formatted as either discreet or composited views for the left and the right eye.A requirement for implementation in VR is to split the image for the left/right head mounted display channels while keeping precise time synchronicity across the channels.Composite 3D video, where left and right views are mapped either on top or next to each other (Figure 9) do guarantee time synchronicity in playback and are best suited for this application.To distribute the composite left/right video texture to the head mounted display left/right channels, it is tiled and offset on the fly while mapped to the UV space of the virtual display geometry.This 3D geometry is rendered twice, with a different UV offset, once for the left channel and once for he right.In Unity 3D, layer culling masks help to achieve the stereoscopic channelling.

CONCLUSION
In this paper, the author explores strategies for the implementation of a large-scale virtual exhibition of immersive architectures and applications within VR.
The highly specialised systems are mostly limited to research labs, visualisation and simulation facilities and museums.A VR implementation has the potential to reach a far broader audience.In addition, hardware and software for immersive environments are mostly custom designed and have a finite life span.This project is a model for the conservation and preservation of immersive experiences.Furthermore, it presents a test-bed and a space for experimentation for the design and evaluation of immersive experiences and architectures before they are developed at full scale.At the core, this project represents an investigation in how VR can capture and reproduce not only the physical characteristics and the content, but also the experiential quality for a visitor.VR can afford presence and to a degree simulate the embodied and kinaesthetic aspects, central to immersive art.It able to preserve what an experience felt like.

Figure 1 :
Figure 1: Composite image of the virtual exhibition.

4. 2
Cylindrical projection environments 4.2.1 AVIE Advanced Visualisation and Interaction Environment Developed at the UNSW iCinema Centre for Interactive Cinema Research in 2004.It is comprised of a 360-degree cylindrical screen supported by a steel frame, six active 3D projectors, motion tracking system, a 16.1 channel sound system.Resolution: ~8x1k pixel, dimension: radius 5m, height 3.9m (McGinity et al. 2007).

Figure 7 :
Figure 7: Locomotion outside the tracking area in VR with controller and teleportation (l).Selection of a work inVR with controller and virtual laser pointer (r).

Figure 8 :
Figure 8: A catalogue page attached to the VR controller to provide contextual information about the works on display.