Augmented Symphony: An augmented reality application for immersive music listening

Augmented Symphony is an augmented reality (AR) prototype that allows for an immersive music listening experience by introducing liveness through interactivity and visual engagement. An outcome of a Canadian research-creation project between Ryerson University and the National Arts Centre (NAC) Orchestra, funded by the Social Sciences and Humanities Research Council, Augmented Symphony provides an example of how orchestras can leverage remote delivery through AR technology to enable new ways for listeners to engage with a composition. The research team included a musicologist with expertise in inter-arts aesthetics, a visual designer specialized in performance, and a new media creative scholar, in fluent conversation with the Senior Director of Learning and Engagement at the NAC, the Managing Director of the NAC Orchestra and other personnel. Undergraduate and graduate students participated in a project whose goal was to explore ways of mitigating the impact of the reduction of live performances caused by the COVID-19 pandemic, both on orchestras and on their audience. Accompanying the app demonstration at EVA 2022, this short paper discusses the research, collaboration, and creative workshops that informed the design of the prototype and delineates opportunities for further development.


INTRODUCTION
Augmented Symphony is an augmented reality (AR) prototype that allows for an immersive music listening experience by introducing liveness through interactivity and visual engagement. An outcome of a Canadian research-creation project between Ryerson University and the National Arts Centre (NAC) Orchestra, funded by the Social Sciences and Humanities Research Council, Augmented Symphony provides an example of how orchestras can leverage remote delivery through AR technology to enable new ways for listeners to engage with a composition. The research team included a musicologist with expertise in inter-arts aesthetics, a visual designer specialized in performance, and a new media creative scholar, in fluent conversation with the Senior Director of Learning and Engagement at the NAC, the Managing Director of the NAC Orchestra and other personnel. Undergraduate and graduate students participated in a project whose goal was to explore ways of mitigating the impact of the reduction of live performances caused by the COVID-19 pandemic, both on orchestras and on their audience. Accompanying the app demonstration at EVA 2022, this short paper discusses the research, collaboration, and creative workshops that informed the design of the prototype and delineates opportunities for further development.

DEVELOPMENT PROCESS
The AR prototype was the creative outcome of a process that included quantitative and qualitative research, collaboration with stakeholders, and exploratory workshops. The initial stage of the project documented how orchestras connected with their audiences during the first year of the pandemic (Cristia et al. 2021). Two hundred and forty two initiatives implemented by a purposive sample of thirty-three orchestras in four continents were documented. These initiatives included concerts by the full ensemble or some of its members, in-person, live-streamed, and ondemand, as well as a variety of digital media. We looked at audio-visual integration and innovation. The data showed that, although most of the orchestral music was accompanied by a traditional "image track" reproducing the audience experience at a concert hall, a small number of initiatives explored innovative strategies to engage the audience. These included the integration of storytelling and animation, the addition of digital art, the recontextualization of musical works, and the use of GPS-based apps as an interface with listeners in an open-air setting (Buh 2021).
The second stage of the project consisted of discussions among researchers and stakeholders via two creative workshops. The first workshop, facilitated by the researchers via Zoom on April 27, 2021, included students from Ryerson University's Performance and New Media undergraduate programs, as well as graduate students from The Creative School. The students were presented with an overview of the NAC Orchestra, by their Managing Director, and the initial research questions and the collected data, by the principal investigator. Students were then asked to reflect about their own remote engagement with performances during the pandemic. Using Google Jamboard as an online whiteboard platform, they brainstormed on what would make them want to engage with orchestral performance remotely, considering format, interactivity, and diversity, among other aspects. Upon analysing the results of the workshop, four aspects emerged as the main opportunities of the experience: (1) audio spatialization, (2) re-imagining space, (3) content layers & overlays, and (4) community & shared space. The second workshop, on May 19, 2021, included the same students and musicians taking part in the NAC Professional Development Program. Participants were divided into groups and focused on brainstorming around one of the four areas mentioned above. As before, the results of the discussions were shared back, and overall themes emerged. Audience agency and interaction, environmental storytelling, opportunities afforded by a virtual platform not possible in real-life, and supplementary material surrounding the performance were some of the important aspects to consider, according to this group.
As a final stage of the project, the research team synthesized the collected data, the analysis of the discussions, and the creative workshops in the design and development of a prototype AR application.

PROTOTYPE FUNCTIONAL DESCRIPTION
Augmented Symphony leverages orchestral music and integrates the listener/user in the imaginary space of the orchestra. Users can distribute 'instruments' throughout their own environment, each playing the audio track of an individual instrument in the piece. Through this, users are able to create their own spatial mix of the piece which they can move through and explore. Instruments may be selected by centering them in the screen, which reveals the instrument name as well as exposing two options for users: a button that triggers a pop-up with further information on the instrument, as well as the option to delete the instrument. They may also be repositioned by tapping and dragging them on the screen. To further extend user experience and sonic comprehension, each instrument is represented by a pillar made of eight sections, taking the form of an eight-band audio spectrum analysis.
The application is built using Unity and takes advantage of its internal audio spatialization engine as well as the Oculus Audio Spatialization plugin to accentuate the effect. Considering the application was built for organisations staffed by assumed nondevelopers, details of the experience are altered through an online spreadsheet that is downloaded at runtime.
More detailed documentation of the prototype is available online at: https://deadpixel.ca/projects/augmented-symphony/

FUTURE WORK
The application that we developed hints at alternative modalities that orchestras can explore to engage with audiences in a remote context. We have identified a few possible vectors for future work. First, we see the technology as a potential bridge between orchestras' ageing core audience and younger generations. Second, and perhaps most importantly, the affordances introduced by augmented reality enable the possibility of a visualisation layer to be added to a performance. We have hinted at this by modulating the shape of each sound source according to their frequency spectrum, but we believe that much more could be explored. For instance, an educational layer could provide listeners with a harmonic analysis of the performance, or a representation of how different instruments in the performance relate to each other musically.
Although spatializing individual audio tracks is not a new idea (related work include the collaboration between Mod Studio and the Australian Chamber Orchestra, for instance), apps like Augmented Symphony provide an immersive experience in a domestic setting, more in line with the social changes brought about by the pandemic and emphasize the perception of liveness by the users' interaction with the musical components.