Visualising Rave Music in Virtual Reality: Symbolic and interactive approaches

The virtual reality (VR) experience Cyberdream VR was presented at various events in 2019, including the Event Two exhibition of computer art held at the Royal College of Art. This short demo for Oculus Gear VR provided a c.5 minute sonic journey, in which the user moves through a series of symbolic environments based on the futuristic techno-utopian or dystopian imagery of 1990s rave flyers. These environments accompanied an original soundtrack of rave music and vaporwave, allowing users to enjoy the music whilst feeling as though they are inside synaesthetic virtual spaces related to the symbolic imagery of rave culture. This paper will discuss the subsequent development of this project, which is now being adapted for the Oculus Quest VR headset. Rajmil Fischman’s concept of ‘music in the holodeck’, suggests a possible new paradigm for performing with sound in VR, which is informative for the latest phase of Cyberdream's development. Drawing on ideas such as this, at a macro level of compositional structure, Cyberdream seeks ways to provide a continuous experience analogous to a DJ or VJ set, through blending of music tracks between scenes. Meanwhile, at a micro level of compositional structure, the user can trigger audio-visual ‘sound toys’ with the Oculus Quest Touch controllers. These are conceived as a means through which the user can intuitively ‘paint with sound’ in 3D space, improvising with the music, whilst also generating synaesthetic imagery. The design of the audiovisual 'sound toys' allows both the macro and micro elements to rhythmically interlock, so that sound can be generated in synchronisation with the music, while the corresponding visual imagery merges with the spatial environments. This paper will discuss on-going work on this project, advancing the discourse regarding the visualisation of hardcore rave music in virtual reality.


INTRODUCTION
At the 2019 edition of EVA London (Electronic Visualisation and the Arts), I presented a paper about my various creative projects (Weinel 2019b), including Cyberdream VR, a virtual reality (VR) project created with the game development environment Unity for Oculus Gear VR (Weinel 2019a). This project extended various strands of my earlier work in electronic music composition and VJ performance, much of which seeks to represent synaesthetic forms of visual hallucination that are experienced in respond to music, such as may occur during altered states of consciousness. As I discuss in the associated paper, these projects have grown out of many years of artistic activity and practice-led research (a research methodology where creative practice is used as the primary mode of investigation, see Smith & Dean 2009;Weinel & Cunningham 2020) across areas such as electroacoustic music, visual music composition, abstract painting and creative coding. With my piece Cyberdream, many of these earlier ideas and approaches have been translated into VR. In the 2019 iteration for Gear VR, the piece consisted of a sequence of scenes that the user flies through, with a rave music and vaporwave soundtrack. The scenes were visually based on the design of rave music flyers (Savage 1996), vaporwave graphics (Tanner 2016), demo effects (as seen in demoscene projects, see Polgár 2005). In 2019, Cyberdream was presented at a variety of events including the VR programme of Sci-Fi London film festival, the ACM Audio Mostly conference (where it received an award), and the Event Two exhibition of computer art, which was curated in association with EVA London, the Computer Arts Society (CAS) and FluxEvents.
In my recent work, I have been developing a new version of Cyberdream for the Oculus Quest VR headset. The main reason for making an Oculus Quest version is that this platform offers a better quality of VR experience than the Gear VR. Like the Gear VR, the Oculus Quest also has the advantage of being an untethered VR device that provides improved convenience and freedom of movement over tethered devices (i.e. those that must be connected to a high-spec PC). By making a new version for the Oculus Quest, I can also expand and develop the project with the benefit of the Oculus Touch controllers, which provide more possibilities for interactivity than the Gear VR. In Cyberdream 2019, the experience was largely noninteractive; that is, the user could look around in 360-degrees, but there were few other interactive features. In contrast, in the new 2020 version I have begun to add interactive audio-visual 'sound toys'. The new version also includes a variety of other optimisations and improvements, such as seamless blending of music between scenes, to improve the feeling of continuity. In what follows, I will discuss the current work-in-progress version of Cyberdream, focusing in particular on how these new features provide symbolic and interactive approaches to the visualisation of rave music.

SYMBOLIC APPROACHES
A key feature that has been retained from Cyberdream 2019 is the use of what I refer to as a 'symbolic' approach to the visualisation of music. Often one may associate 'music visualisations' with the visual plugins that are found in various music players such as Winamp's 'Milkdrop' or the 'iTunes Visualiser'. These are somewhat like enhanced versions of the oscilloscope, providing visualisations of the audio signal, translating amplitude or spectral properties of the waveform into moving patterns of light. These music visualisations can be excellent, and they benefit from flexibility, since they can be used with any audio source. However, a trade-off is that these types of visualisation cannot provide specific correspondences that are tailored to the music being played. Dannenberg (2005) notes this limitation, arguing that composed forms of visual music can benefit from improved specificity regarding correspondences between sound and image, which he suggests may lead to richer and more interesting artistic experiences. To design such experiences, the visualisation of music may need to be approached primarily as 'audio-visual composition', rather than as an algorithmic process, though of course, these approaches may also overlap.
Following this idea, the approach that I use in Cyberdream does not prioritise the visualisation of the audio signal of music. Instead, the main approach I have used is 'symbolic', where representational 3D environments are designed using computer graphics, in response to the visual culture that is associated with the style of music being used. This is possible because both the music and visual design are bespoke: Cyberdream features original hardcore rave music and ambient techno fragments that I compose specifically for this project. To create 3D environments that suit these styles of music, I draw upon the existing visual languages of rave music, particularly 1990s rave flyers which often use computer graphics to depict utopian or dystopian technological landscapes (for examples see: Savage 1996).
Using this approach, Cyberdream provides 'symbolic' visualisations of music by creating corresponding 3D environments that are related to the music (Figure 1). The intended effect is for users to feel as though they are inside the music as a physical audio-visual space. This could be compared to the idea of actually going into a rave flier, or the artwork on an album cover. VR is well suited to this, because it allows the development of audio-visual spatial representations that can give rise to feeling of 'presence': the feeling of 'being there' in a virtual environment (Slater & Wilbur 1997). For the purposes of music this is exciting, because it provides the possibility to actually journey into audio-visual spaces of music -to be inside the music. I describe this approach as 'symbolic' because it relies on depicting visual symbols that are derived from the visual culture connected with the music. While DeNora (2010) highlights that music may not have absolute or objective interpretations, she argues that associations with music can be established and reinforced through repeated patterns of use. When considering how visual images or symbols can become associated with music, we may consider that they emerge through patterns of cultural use over time. Through this, we may come to associate 1960s surf music with images of surfing, or heavy metal music with skulls and various fantasy depictions (to give just two examples).
As I discuss elsewhere (Weinel 2018a), these representational properties may serve to frame the affective properties of music with different forms of conceptual meaning that are appealing to audiences, relating to their sense of identity and personhood. Yet, because these associations are gradually negotiated over time, with each new sound/image combination, they may be adapted and evolve in new directions. Therefore, whilst on the one-hand Cyberdream is drawing on the established visual languages connected with rave music and vaporwave, on the other-hand the piece itself also weaves these dreams, forging new audio-visual relationships.

WAVE PATTERNS
One of the technical approaches used in the design of many of the symbolic environments in Cyberdream is the use of wave patterns to animate various visual forms. These are mostly created using one or more wave functions, which can be used to animate the sizes of cubes, tiles or colour properties. Equations similar to those shown in Figure 2 are used to control some aspect of all of the scenes shown in Figure 1. In electronic music production music these wave patterns would be considered as 'low frequency oscillators' (LFOs), which are often used in synthesizers to modulate filters and other properties of sound over time. We may consider that the use of these wave patterns, though independent from those used in the music, contributes to a sense of 'musicality' within the environments. Perhaps the most successful implementations of this approach are those that use multiple combinations of sine waves to produce 'plasma' effects (e.g. see Vandevenne 2004). Whereas this technique is normally used to change the colour values of individual pixels, in this case the effect is applied to animate the sizes and colours of cubes (as shown in Figure 1, forth image down). Along similar lines, in another scene combinations of sine waves are used to make Lissajous figures (as shown in Figure 1, fifth image down).

MACRO STRUCTURE
The symbolic approach described in this paper was already present in the 2019 version of Cyberdream, and is retained in the new version. This provides a series of 3D environments based on the visual language of rave flyers and related artefacts (Figure 1). The user is either suspended in these or flies through them, and each has a corresponding music track that provides the soundtrack. At a macro compositional level, the structure of Cyberdream is analogous to a DJ/VJ mix, where each audio-visual scene is equivalent to a record that a DJ might play, and the combination of these together forms a mix. This approach has developed from earlier performances in which I mixed audiovisual tracks to create DJ/VJ sets (see Weinel 2018b; 2019b). Cyberdream uses a similar macro approach to structure audio-visual materials in VR.
The purpose of a DJ or VJ mix is to provide continuity between musical elements over a longer duration. In a nightclub or at a rave, the DJ aims to take listeners and dancers on a musical journey, usually over an hour or more. To do this successfully requires an awareness of musical structure at a macro level. The DJ utilises existing tracks as tools or components, and assembles these to create a whole that is often greater than the sum of the parts. A significant reference point for Cyberdream, are the 1990s 'tape packs' from megaraves such as Fantazia. For example, Carl Cox's early 1990s DJ mixes such as Fantazia: The Big Bang (Cox 1993), or the various Eclipse cassettes (e.g. Cox 1991; recorded in Coventry, exhibit a frenetic, rapid turntable style, which encapsulates a sense of rawness and energy from rave music at that point. Tracks are cut abruptly together on three turntables, providing cascades of chaotic adrenaline and hysterical emotion. With Cyberdream I eventually hope to capture a similar sense of energy in the form of a synaesthetic VR music experience. The original music I have composed for Cyberdream consists of fragments of hardcore rave and ambient techno, created using breakbeat samples, synthesisers and sample libraries appropriate to this production style. These have been created using a mixture of hardware and software synthesisers, such as the Korg MS-20 mini and the Roland Cloud suite of software (which recreates classic Roland synthesisers hardware such as the D-50, SH101 and others). Each rave music production provides the soundtrack for a 3D scene, which the user moves through rapidly, with a transition occurring approximately every 30-60 seconds. While the Cyberdream 2019 did not feature smooth transitions between music tracks, in the 2020 version the music now flows together seamlessly. This is technically accomplished with the game audio middleware software Wwise, which allows the tracks to be synchronised and crossfaded in the typical manner of a DJ mix. Lastly, an experimental feature of Cyberdream 2020 also allows the order of the scenes to be randomised, thus generating a different 'mix' each time.

AUDIO-VISUAL 'SOUND TOYS'
Whilst the organisation of scenes into the form of a DJ/VJ mix provides the macro compositional structure, the micro compositional structure of Cyberdream is provided by interactive audio-visual 'sound toys'. As Tom Szirtes notes in a podcast interview about immersive music visualisations, there is an expectation for users of VR music applications that there should be some form of interactivity; that is, users expect that there should be something for to do (Immersive Audio Podcast 2019). Exactly what this should be has not yet been entirely solved by developers. Music applications such as MelodyVR allow the user to watch a concert that is filmed in 360-degrees video from different perspectives. Following the established form of a 'rhythm-game', Beat Saber is another music application for VR, in which the user must use light-sabers to slash at cubes in time with the music. In terms of Calleja's (2011) theory of 'player involvement', this prioritises a ludic approach, immersing the listener in an experience of music through challenges and objectives. However, it is likely that other forms of interaction are also possible, which are less orientated towards ludic or 'game-like' challenges, and instead provide possibilities for musical expression in a manner analogous to performance with a musical instrument.
To consider what these alternatives might look -or sound -like, we may turn towards Rajmil Fischman's (2011) concept of 'music in the holodeck'. Fischman proposes that a multisensory immersive environment (like the 'holodeck' seen in the science-fiction TV show Star Trek), could one day provide an arena in which accessible, collaborative electronic music-making takes place. Central to Fischman's argument is the idea that such a facility might overcome barriers in music making, where often it is only trained expert musicians who can participate in musical performance. Instead, Fischman suggests a return to music-making as a social activity that is accessible to all. This idea of allowing music making to be available to non-experts resonates with ideas of improvisation discussed by Derek Bailey (1992), who finds validity in musical improvisation by untrained amateurs and children. Furthermore, while Fischman describes 'music in the holodeck' as an electronic equivalent of "a weekend afternoon in a middle class parlour during the second half of the nineteenth century: a soiree, with music performed live by family members gathered around the piano", the idea of accessible, social music-making may also resonate with earlier forms of communal drumming and music making in ancient or shamanistic human cultures. Arguably game technologies provide an interesting opportunity to explore these ideas, because the technologies are already available in society and widely used by young people.
In Fischman's own creative practice in real-time electroacoustic compositional, he uses a dataglove to intuitively perform real-time computer music involving complex processes such as granular synthesis (see Fischman 2011;2012). This and other work in computer music research (e.g. Serafin et al. 2016) investigates various approaches to electronic music-making instruments using gestural interfaces, custom controllers and 'sound toys'. However, whilst these usually provide user interfaces with visual feedback, recognising that VR also has visual capabilities, we might also wonder what 'music in the holodeck' should look like. Work in the areas of visual music and VJing may suggest ideas for this, however these possibilities are also perhaps best envisaged in science fiction TV shows: for example, both Futurama (see also Weinel et al. 2014), and the gabber-themed anime Culturesport feature technologically-enhanced reed instruments, which can be used to perform music that is accompanied by symbolic visualisations of virtual environments.
Though fictional, these depictions tantalisingly suggest possibilities that could soon be realised with immersive technologies.
Drawing ideas such as these together, we can consider the possibility of audio-visual 'sound toys', which generate sounds and corresponding visual images together. In the latest version of Cyberdream, an early prototype of one of these audio-visual 'sound toys' has been created. Using the motion-tracked Oculus Touch controllers as inputs, the user can activate laser beam emitters (referred to as 'ZigZagToys') that produce jagged orange beams, and corresponding pulsing sounds recorded with a Roland SH-101 ( Figure 3). The design of these sounds is partly inspired by speedcore techno and flashcore sub-genres of electronic dance music, which utilise rapid percussive rhythms (see Weinel 2007). The sounds are spatialised relative to the controller position for both the left and right hands, and each triggers slightly different variations of the pulsing sounds in order to provide more interesting stereo effects when using both the left and right-hand toys together simultaneously.
Using a combination of 'callbacks' (messages that are sent from the audio-engine to code in the main game engine) and 'stingers' (musical elements that can be triggered in synchronisation with the music), both the pulse sounds and the visual animation of the laser beam occurs in a rhythmic pattern that is synchronised with the beat of the music. This is technically accomplished using a custom 16-step sequencer that stores patterns as integers, where in binary form a '1' signals a percussive trigger as 'on', while a '0' signals the trigger as 'off' for the given step. For instance, as shown in Figure 4, 'Pattern A' triggers regular 4/4 percussive hits on the beat, while 'Pattern B' has triggers hits off the beat. This provides an efficient system for sequencing rhythmic patterns, which sends triggers to the audio-engine pre-emptively to take latencies into account. Using this system, the user can use the left or right index triggers of the controllers to trigger rapid streams of percussive pulses with the 'ZigZagToys' following 'Pattern C' (as shown in Figure 4). Also, while holding the hand-trigger, the user can also temporarily switch to a secondary pattern, specified as 'Pattern D' for the left-hand 'ZigZagToy', and 'Pattern E' for the right-hand 'ZigZagToy'. This allows the user to playfully manipulate the rhythm in real-time by switching between rhythmic patterns, producing various interlocking effects between the left and the right hand.
While the beam is activated with the index trigger controls, using the thumb joysticks allows manipulation of the sound using real-time parameter controls (RTPCs) in the audio-engine that affect pitch and filter properties. This allows further manipulation of the rhythmic sounds that the 'ZigZagToy produces. In correspondence with the changes in these parameters, the maximum width of the beam patterns expands and contracts when moving the joystick axis vertically, while the colours oscillate with shades or yellow and purple when moving the axis horizontally. The appearance of the beam is also intended to match the visual composition of the scene. Lastly, once more have been designed, the A/B and X/Y buttons of the controllers will eventually allow the user to cycle between different 'sound toys', thereby producing different combinations of audio and visual effects.

SUMMARY
This paper has discussed work-in-progress on Cyberdream, a project in which I am exploring ways to visualise electronic music in VR. As discussed, the 2020 version of Cyberdream advances the project in several ways for the Oculus Quest VR headset. The project continues to use a 'symbolic' approach to the visualisation of music, where 3D environments are designed in correspondence with the visual designs of rave culture. Each environment has a corresponding musical soundtrack, and these are organised to provide a continuous experience, giving the project a macro structure similar to that of a DJ or VJ mix. As a complement to this, the first prototype audiovisual 'sound toy' allows the user to interactively emit orange laser beams with corresponding sounds from their hands, using a step-sequencer to allow the user to control the generation of micro structural elements. These micro elements allow the user to 'paint with sound', and they interlock with the macro elements by means of rhythmic synchronisation (quantising).
Though the usability of this project has not yet been formally tested, it is hoped that the game controllers will provide an intuitive and accessible means of control, whilst still providing flexibility in sound generation. This flexibility will be further improved with future iterations of the project in which I intend to add more audio-visual 'sound toys'.
One of the exciting aspects of working in this area is that we are beginning to see various new projects emerge that use game engines and immersive technologies to generate entirely new forms of experience that are related to music. For example, other work in this category includes Ricardo Climent's musical 'duels', created with the Unreal game engine; Chris Holtum's dark audioelectronica performances, realised with original Unity environments; or Carl Emil Carlsen's audiovisual performance pieces that also use Unity to generate 'Pepper's ghost' holograms. Each of these works suggests new compositional approaches, and perhaps collectively, they point towards a completely new paradigm for immersive audio-visual experiences of music.

ACKNOWLEDGEMENTS
The original Cyberdream VR project discussed in this paper was made possible by the loan of Oculus Gear VR equipment from the Music and Sound Knowledge group at Aalborg University, Denmark.