Beyond the Audience Seat: The recording and production of immersive opera and interactive concerto programmes for VR experiences

New modes of expression have always defined a central attribute within the ambitions of classical music. The extended realities (XR) hold the potential to expand this expressive repertoire by transporting audiences into the middle of the action within an immersive music performance. We argue that implementing these innovative formats in contemporary opera performance, composition, production, and recording opens up new aesthetical paradigms that can reciprocally inform the artistic disciplines and reach a wider demographic of audiences. To our knowledge, this paper presents the first ever interactive virtual reality (VR) production of an entire opera, “She” (2017) and “Croak” (2018). The recordings devise 3-dimensional 360-degree high-resolution video and 3rd-order Ambisonics audio from multiple perspectives within a bespoke software interface. Following a discussion of both operas from the composer’s and engineer’s perspective, the paper will present an enhanced rendition of “CroakVR”, a 6-degrees-of-freedom (6DoF) volumetric simulation of its corresponding opera, that raises the bar of interactivity by enabling audiences to freely explore various mimetic and diegetic perspectives of its staging. Furthermore, we discuss on-going work on “Climb!VR”, a 6DoF volumetric VR production devising multiple Ambisonics recordings of an interactive electro-acoustic piano concerto “Climb!” based on ludonarrative structures. It is argued that VR recordings can amend the life span of musical compositions by offering listeners a concert experience that can extend beyond the walls of the physical concert hall.

Content

Author and article information

Contributors

Hans-Peter Gasselseder

Maria Kallionpää

Conference

Publication date: July 2019

Publication date (Print): July 2019

Pages: 409-416

Affiliations

[0001]Aalborg University

Dept. of Communication & Psychology

Musikkens Plads 1

9000 Aalborg, Denmark

[0002]Hong Kong Baptist University

Rm AST801, Sing Tao Building

Hong Kong Baptist University

Kowloon Tong, Kowloon, Hong Kong

Article

DOI: 10.14236/ewic/EVA2019.75

SO-VID: ef74ab17-b76c-4a61-9ea5-0b13fbeb6b5c

License:

This work is licensed under a Creative Commons Attribution 4.0 Unported License. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/

Conference name: Proceedings of EVA London 2019

Conference acronym: EVA 2019

Conference number:

Conference location: London, UK

Conference date: 8 - 11 July 2019

Conference sponsor: Electronic Workshops in Computing (eWiC)

Conference theme: Electronic Visualisation and the Arts

History

Product

1477-9358 BCS Learning & Development

Self URI (article page): https://www.scienceopen.com/hosted-document?doi=10.14236/ewic/EVA2019.75

Self URI (journal page): https://ewic.bcs.org/

REFERENCES

Audionamix 2019 ADX Trax Pro 3 Audio Spectral Editing Tool https://audionamix.com/technology/adx-trax-pro/ retrieved 19 March 2019 )
H.-P. Gasselseder 2015 The Role of Agency in Ludoacoustic Immersion: Experiencing Recorded Sound and Music in Situational Context Audio Mostly Thessaloniki, Greece 7-9 October 2015 ACM New York
J. Gladstone 2018 Stereo2Depth http://pseudoscience.pictures/stereo2depth/pseudo_sciencestereo2depth1.8.5.zip retrieved 19 March 2019 )
J. Gladstone 2018 Use depth maps to create 6DoF in Unity Immersive Shooter https://www.immersiveshooter.com/2018/01/23/use-depth-maps-create-6dof-unity retrieved 19 March 2019 )
Insta360 2017 Insta360 Pro 360° 3D Camera for VR https://www.insta360.com/product/insta360-pro retrieved 19 March 2019 )
M. KallionpääC. GreenhalghA. HazzardD. M. WeiglK. R. PageS. Benford 2017 Composing and Realising a Game-like Performance for Disklavier and Electronics NIME Copenhagen, Denmark 15-18 May 2017
M. KallionpääA. ChamberlainH.P. Gasselseder 2018 Under Construction – Contemporary Opera in the Crossroads Between New Aesthetics, Techniques, and Technologies Audio Mostly Wrexham, UK 12-14 September 2018 ACM New York
Kodak 2015 Kodak SP360-4K 360° Camera https://kodakpixpro.com/Europe/cameras/vrcamera/sp3604k retrieved 19 March 2019 )
G. Santini 2018 Linear (Live Generated Interface and Notation Environment in Augmented Reality) International Conference on Technologies for Music Notation and Representation Tenor 18 Montreal 24-26 May 2018 33 42
S. SchützeS. Irwin-Schütze 2018 New Realities in Audio Boca Raton, FL CRC Press
E. StrasnickA. ChambersL. JiangX. Tong 2016 Pianolens: An Augmented Reality Interface for Piano Instruction https://cs.stanford.edu/people/estrasni/otherprojects/pianolens.html retrieved 19 March 2019 )
M. van der Aa 2018 “Eight’ description https://www.vanderaa.net/eight retrieved 19 March 2019 )
Zylia 2019 Zylia ZM-1 Ambisonics Microphone Array http://www.zylia.co/zylia-pro.html retrieved 19 March 2019 )

Comments

Comment on this article

[1] Audionamix 2019 ADX Trax Pro 3 Audio Spectral Editing Tool https://audionamix.com/technology/adx-trax-pro/ retrieved 19 March 2019 )

[2] H.-P. Gasselseder 2015 The Role of Agency in Ludoacoustic Immersion: Experiencing Recorded Sound and Music in Situational Context Audio Mostly Thessaloniki, Greece 7-9 October 2015 ACM New York

[3] J. Gladstone 2018 Stereo2Depth http://pseudoscience.pictures/stereo2depth/pseudo_sciencestereo2depth1.8.5.zip retrieved 19 March 2019 )

[4] J. Gladstone 2018 Use depth maps to create 6DoF in Unity Immersive Shooter https://www.immersiveshooter.com/2018/01/23/use-depth-maps-create-6dof-unity retrieved 19 March 2019 )

[5] Insta360 2017 Insta360 Pro 360° 3D Camera for VR https://www.insta360.com/product/insta360-pro retrieved 19 March 2019 )

[6] M. KallionpääC. GreenhalghA. HazzardD. M. WeiglK. R. PageS. Benford 2017 Composing and Realising a Game-like Performance for Disklavier and Electronics NIME Copenhagen, Denmark 15-18 May 2017

[7] M. KallionpääA. ChamberlainH.P. Gasselseder 2018 Under Construction – Contemporary Opera in the Crossroads Between New Aesthetics, Techniques, and Technologies Audio Mostly Wrexham, UK 12-14 September 2018 ACM New York

[8] Kodak 2015 Kodak SP360-4K 360° Camera https://kodakpixpro.com/Europe/cameras/vrcamera/sp3604k retrieved 19 March 2019 )

[9] G. Santini 2018 Linear (Live Generated Interface and Notation Environment in Augmented Reality) International Conference on Technologies for Music Notation and Representation Tenor 18 Montreal 24-26 May 2018 33 42

[10] S. SchützeS. Irwin-Schütze 2018 New Realities in Audio Boca Raton, FL CRC Press

[11] E. StrasnickA. ChambersL. JiangX. Tong 2016 Pianolens: An Augmented Reality Interface for Piano Instruction https://cs.stanford.edu/people/estrasni/otherprojects/pianolens.html retrieved 19 March 2019 )

[12] M. van der Aa 2018 “Eight’ description https://www.vanderaa.net/eight retrieved 19 March 2019 )

[13] Zylia 2019 Zylia ZM-1 Ambisonics Microphone Array http://www.zylia.co/zylia-pro.html retrieved 19 March 2019 )

Studying business & IT? Drive your professional career forwards with BCS books - for a 20% discount click here: shop.bcs.org