Experiencing Visual Music From an Artist’s and Listener’s Point of View

Visual Music is an art form that can enhance the emotional impact of music performance through the simultaneous projection of mood-congruent visual imagery. I illustrate the transcendence of the art by reference to two of my pieces. In Abîme des oiseaux , it was Olivier Messiaen’s ability to accurately transmute a wide range of celestial colours into audible sound combinations that enabled me to hear ‘the colours that moved with his music’ and, thereby, see what was in his mind’s eye. Visual mental imagery during music listening plays a pivotal role for the majority of listeners. In creating my own visual responses to music, I seek to modulate listeners’ emotional response by implanting my own form of visual expression in their minds. This is both the challenge and delight of making Visual Music. In my latest work Passeggiata , based on Luciano Berio’s Sequenza IXa for solo clarinet, I’ve used patterns and images gleaned from Deep Dream artificial intelligence to give an impression of an other-worldly ‘passeggiata’ in and around a mountain village in Liguria – the province of Italy where Berio was born and lived.


INTRODUCTION
Visual Music has a short history. It can be argued that the idea was initiated, in the 1920s, by Oskar Fischinger who was active as an artist in Frankfurt and Berlin. He invented techniques whereby a series of frames, showing abstract images reflecting the rhythms of music, were put together as short films ( Figure 1). His innovation caught the attention of Hollywood where, in 1926, he emigrated but, unfortunately, achieved only very limited success. Walt Disney used some of his ideas in Fantasia but gave him no credit. However, since those early days, Visual Music has gained momentum for audiences who can experience it increasingly in concert halls where visual imagery is projected simultaneously alongside music performances.
Visual Music is an art form that can enhance the emotional impact of music performances through the simultaneous projection of mood-congruent visual imagery. Usually, I find that when I'm 'illuminating' or visualising a musical score, the process involves penetrating the mind of a composer to uncover the hidden imagery that lies behind the notation. This is what happened during one of my first presentations, at EVA 2016 (London), when I set out to 'Reveal the Colours of the Apocalypse through Visual Music' (Trickett 2016a;2016b).

ABÎME DES OISEAUX
In that early paper + demo, I aimed to throw some new light on the synaesthetic world of Olivier Messiaen, a composer who had revealed much about his sound world but remained comparatively reticent about his world of colour. My Initial research into the subject quickly revealed that the shimmering stained glass of Chartres Cathedral had provided, throughout Messiaen's life, a joyful experience, a place where he could fully indulge the sensory impact of his synaesthetic world. It was this new-found knowledge that enabled me to locate the hidden imagery that lay behind the composer's notation for Abîme des oiseaux -a piece, for solo clarinet, that became eventually a central movement within Messiaen's Quartet for the End of Time.

VISUAL IMAGERY IN THE MIND'S EYE
My experience of producing Abîme des oiseaux leads me now to uncover some of the mystery that surrounds visual imagery as it relates to music. Recent research has shown that as many as 77% of music listeners experience visual mental imagery during music listening (Taruffi & Küssner 2019). This form of imagery rather than kinetic, auditory, olfactory and gustatory imagery (i.e. pertaining to flavours and the sense of taste) predominately is triggered by music listening. It can be argued that an ability to represent the world in our minds has immense value in its application across other aspects of life; it enables us to navigate the environments in which we live, travel back in time and remember past experiences, or picture ourselves in the future.

Steven Kosslyn, in his book Ghosts in the Mind's
Machine, describes mental images as private creations (Kosslyn 1984). Although mental imagery and perception (i.e. that which we see with our eyes) operate in similar ways, they are far from being identical: Visual percepts are stable, reflecting the reality around us, whereas visual mental images are unstable, at the mercy of the full range of our powers of fantasy (Kosslyn 1984).
With mental imagery, we can think about and transform what our eyes have told us. This is a key feature of mental events -the ease by which they can create scenes that never really existed or, as Kosslyn comments, transform the commonplace into the extraordinary. Another striking fact about mental images is that we don't have them all the time; they are transient. It can be assumed, then, that images must be stored in our long-term memories in some way that allows us call on them when we want them. In this regard, mental imagery is quite different from vision, which operates whenever our eyes are open and brings us a continuous stream of images whether we choose to concentrate on them or not. This voluntary quality of mental images and our capacity to get rid of them when we don't want to look at them explains the fleeting periods of 'mind wandering' that most of us experience when attending a concert. We can decide to enjoy a temporary period of inattention at those moments when spontaneously evoked visual mental imagery, unrelated to the sensory impact of listening to music, comes into our mind, or we can decide to switch it off. I maintain that, for audiences attending a performance of Visual Music, the projection of prescribed imagery, in acting as an added focus of attention, will mitigate against listeners calling on their own less stable and fleeting sources of imagery. But, in saying this, I might be underestimating the power of mind over matter!

THE RAMIFICATIONS OF PICTORIAL GHOSTS
Unstable images formed in the mind's eye can be described as quasi-pictorial ghosts; they can't easily be compared to the form pictures take in the real world -e.g. photographs, paintings, slides.
There must be something more diffuse than paper, canvas or projected beams of light that enables visual mental imagery to take shape. Kosslyn has put forward the idea of a 'visual buffer' in the brain, which reveals, at its centre, an image that is fully resolved (i.e. in focus) but with decreasing resolution towards the periphery. As a neuroscientist as well as a psychologist, Kosslyn was not only concerned in shape and diffusion but, also, in what he perceived as three types of process that operate on images in the visual buffer (Figure 4). The 'generation' process acts on information about the appearance of objects and their spatial structure, stored in the long-term memory, to create an image in the buffer. We become conscious of this pattern of activity taking place and, through a process of 'inspection', we can then recognise the shape, spatial configurations and other characteristics of imagined objects. Finally, through 'transformation' processes that rotate, scale in size and translate the pattern of cells in the buffer, we are able to examine visual mental images from all points of view. I can illustrate this extraordinary mental facility by referring to those much-maligned intelligence tests, which ask us to pick out one specific representation of an object, which is incongruent with others. Such tests can be immensely puzzling because we have to take an imaginary rotational journey round a number of representations of an object so as to 'see' it from all directions ( Figure 5). Only then, do we know if one 'seen' 3-dimensional form of the object is incongruent with others. These tests are considered to be fair because they use a common pictorial language spoken by everybody but, in fact, this assessment fails to take into account people's differing innate abilities to sense pictorial mental imagery. It seems that those of us who have a spatial pictorial mind are at an advantage over those who do not.

Figure 5: Mental rotational processes (image transformation) illustrated by objects which are congruent (left) and incongruent (right).
These deeper ramifications of visual mental imagery have been the cause of an 'imagery debate' that has occupied the minds of philosophers for hundreds of years. David Hume, for example, underlined the "great resemblance" between percepts and mental images "in every other particular, except their degree of force and vivacity" (Hume 1739(Hume /1978. Others, including Jean-Paul Satre, argued that mental images have a radically different phenomenological status from percepts (Satre 1940). It's a debate that will continue to rage for as long as the activities of the human brain remain mysterious and inaccessible to finite scientific thought ( Figure 6).

Figure 6:
The 'imagery debate' has occupied the minds of philosophers for hundreds of years.

Figure 7: Terry Trickett aims to produce a Visual Music experience, which, in colouring the minds of listeners and viewers, succeeds in generating an audience's heightened state of awareness and enjoyment in music.
For me, it raises the question: "can I or should I try to manipulate or control listeners' visual mental responses when I create a piece of Visual Musicresponses which might differ radically from my own very personal visual take on music?" I always answer 'yes' to this question with the proviso: "as long as the Visual Music I produce adds to listeners' experience of music by informing on its context and arousing emotions that might otherwise lie dormant" (Figure 7). It's for this reason that, in seeking common patterns between musical notes and visual imagery, I always undertake background research into a composer's sources of inspiration. It's a process that can produce unexpected discoveries with unforeseen outcomes as occurred in the case of Messiaen's Abîme des oiseaux as described above.

PASSEGGIATA
To further describe the challenges and delights of making Visual Music, I'll turn now to my most recent piece -a visual interpretation of Luciano Berio's Sequenza IXa for solo clarinet (Trickett 2019). The composer wrote the piece in 1980, midway through his lifelong exploration into the idiomatic potential of instrumental sound. It makes extreme technical demands on the performer and, at the same time, invents a musical language that gives the clarinet a completely new mode of expression.
Berio's music can be theatrical; in some Sequenze he insists that the instrumental soloist acts out a prescribed dramatic 'scenario'. For instance, in Sequenza V, for solo trombone, the composer makes very specific demands. He instructs that the trombonist should wear a white tie (i.e. implying a tail coat rather than a dinner jacket) and strike the pose of a variety showman about to sing an old favourite! The result, of course, is hilarious with trombonists often wearing make-up and a clownish tailcoat (like Berio's neighbour Grock) so as to make explicit the circus atmosphere of the piece. The trombonist's physical behaviour and demeanour become an essential part of the piece's musical expression as he or she is asked to act clownishly, sing into the instrument, vocalize vowel sounds and make continuous side-to-side head movements (Figure 8).  Of course, all of this negates any purpose in producing a visual interpretation of the piece but, for the clarinet Sequenza, Berio didn't issue any specific performance instructions so I've felt free to make Sequenza IXa the basis for a piece of Visual Music, Passeggiata. As in all Berio's Sequenze, virtuosity becomes an essential element of performing Sequenza IXa because the composer believed that it's the theatrical nature of virtuosity that capture's an audience's attention. Tongue-incheek Berio stated: I hold great respect for virtuosity even if this word may provoke derisive smiles and conjure up the picture of an elegant and rather diaphanous man with agile fingers (Halfyard 2016).

USING ARTIFICIAL NEURAL NETWORKS TO PRODUCE IMAGERY FOR PASSEGGIATA
In the piece, Berio explores, at length, one specific harmonic field but avoids repetition by springing constant surprises in terms of speed, pitch and rhythmic variety. It's an exercise in musical discovery that has inspired me to dig deep in finding appropriate imagery -imagery that again, as in the case of Abîme des oiseaux, seeks to enter the mind's eye of the composer. My approach has been to engage with machine learning to produce images, which mimic and interact with human visual perception. The resulting images, slightly out of focus and dream-like, give the impression of an otherworldly 'passeggiata' in and around a mountain village in Liguria -the province in Italy where Berio was born and lived ( Figure 9). Of course, I can never know whether or not my images convey an accurate impression of the visual mental imagery that Berio might have experienced when composing Sequenza IXa; it's only in my imagination that I'm putting myself inside the mind of the composer. Often, the creation of Visual Music demands taking creative leaps into the unknown but, in Passeggiata, I've placed some reliance on the use of artificial neural networks, inspired by the brain and nervous system, to produce pictoralist imagery that reflects Berio's own journey of musical discovery. Additionally, a sense of visual theatricality is produced by matching the composer's sometimes hectic notation with similarly fast-moving virtuosic, geometric patterns.

THE CONTINUING ADVENTURE OF VISUAL MUSIC
Overall, does my imagery for Passeggiata succeed in adding to the piece's emotional impact and can it assist audiences in their understanding of a formidable work by one of the last century's most experimental composers? I hope so because Berio's lifelong ambition was to relate his music to various fields of knowledge -poetry, theatre, linguistics, anthropology and architecture. I believe he would have welcomed a visual interpretation of Sequenza IXa as long as it served a purpose in amplifying his idiosyncratic method of communicating complex musical ideas. This is the type of interpretive challenge I've needed to face in both Passeggiata and Abîme des oiseaux and in many other pieces that I haven't referred to in this paper. Overall, I'm aiming to produce a Visual Music experience, which, in colouring the minds of listeners and viewers, succeeds in generating an audience's heightened state of awareness and enjoyment in music.