Creating Affective Visual Music

This paper delineates the on-going development of a methodology for creating affective visual music. Visual music is a creative practice that does not split the eye and ear; it problematizes this long-standing duality and seeks to make a singularity. Visual music encompasses many types of output: abstract paintings, time-based performance art such as colour organs, abstract film, projected light shows, art installations of film and expanded cinema (digital media). The impulse to find correspondences between music and visuals and use these to create a new genre has a long history. This practice-based research is historicised and highlights both the key seminal influences underpinning the work and the innovations embodied within it. This paper will discuss experimental, visual music pieces from my own practice (Watkins 2016, 2017, 2018) that employ Ron Kuivila’s strategy ‘over’ technology and seminal works from the visual music canon. Visual music is often approached from the viewpoint of a musical composer, this practice-based research is visually led; visual structures, rather than musical structures, are explored. Visual music can be perceived as overly repetitive, cold and alienating if it has a purely mechanical alignment of music to image, or if it seems disengaged from both human emotions and natural imagery. A key objective is to create work that is non-figurative, non-narrative, pre-language, extra-language, and yet suffused with human presence, to create visual music that is affective.


INTRODUCTION
Visual Music is a creative practice that Friedmann Dahn asserts (Lund & Lund 2009, p.149) is: [A]n equal and meaningful synthesis of the visible and audible and is therefore ultimately its own art form.
The impulse to find correspondences between music and visuals and use these to build a new mode of creative activity, indeed a new art form, has a long history.The seminal historian of avantgarde and experimental cinema William Moritz (1998) stated: [N]on-objective animation is the purest and most difficult form of animation.Anyone can learn to "muybridge" the illusion of representational life, but inventing interesting forms, shapes and colors, creating new, imaginative and expressive motions "the absolute creation: the true creation," as Fischinger termed it (Fischinger, 1947) requires the highest mental and spiritual faculties, as well as the most sensitive talents of hand.
It could be argued, as 'the father of computer art' John Whitney (1980) stated that much visual music relies on Pythagoras's mathematical ratios.The historians of mathematics John Fauvel, Raymond Flood, and Robin Wilson (2006) also suggest that the idea of visual music could be said to start with Pythagoras, in about 500 B.C., investigating the musical scale and discovering the ratios that give a relationship between the frequencies of the notes.By exploring the octave and interval of fifths, arguably, he described the first scale completely based on mathematics and these ideal whole number ratios extended into the visual world, into architecture and sculpture.A close relationship of the visual arts and music gave impetus to artistic development in European and Islamic art into the Renaissance.
In the 18 th century, Newton produced a mapping of colour to musical notes.The historian Kenneth Peacock (1988, p.398) states: Sir Isaac Newton was the first to observe a correspondence between the proportionate width of the seven prismatic rays and the string lengths required to produce the music scale D, E F, G A B C. Indeed, he may have decided on white light dividing into seven colours in order to create this correlation.He divided the colour circle into seven unequal segments, each arc length relates to the note's position in the scale.But, as the physicist Patricia Fara (2015, p.9) clarifies: Newton never specified exactly what those quantities might represent in terms of light and pigments.
Even after three centuries no one agrees on a musical note to colour mapping, as the designer and innovator Fred Collopy (2006) states, see Figure 1.(1970) is more visually complex than just using the analogue film's optical sound track in vision, but nevertheless gives a good idea of this pure fusion.This 1:1 mapping can be cold and mechanical.A response is to stay 'over' technology (Kuivila & Behrman 1998), which is the obverse of the 1:1 mapping of strand 2. John Cage's Variations V demonstrates staying 'over' the technology; the composer creates the framework that allows the performer's 'in-themoment' sensibilities to come to the fore, freed from following exact instructions.
The area of visual music under exploration is strand 1-A; visual structure informs both image and sound.There are temporal differences between composing strand 2, in which the 1:1 mapping may well happen in real time, strand 3 in which either the images are made in response to completed music or music is created in response to completed images and strand 1, in which there is an interleaving of the creation of images and music.Strand 1-B is underpinned by a long legacy of musical structures.Strand 1-A is underpinned by the work of individual artists and animators and their exploration of visual structures.It includes mute works such as Thomas Wilfred's lumia and static works such as rhythmic paintings by Paul Klee.
The intention of the creator and purpose of the piece of strand 3 decides whether the piece is visual music or not, Len Lye's Trade Tattoo (1937) is an example of strand 3-A.Made with the intention of creating visual music, the animator visualises specific sonic elements in motion and the audio-visual sync is tight at several points.When the driving purpose is not visual music, for example LA Philharmonic's 'Orchestra VR' the result will be first and foremost its function, in this case an educational tool.Strand 3-B will also be made with the intention of creating visual music, the musician sonifies specific visual elements, and audio-visual sync is tight at several points, for example Tom Reid's score to Walter Ruttmann's Opus III (an absolute film from 1924).When the driving purpose is not visual music, for example a BBC One 'ident', the result will be first and foremost its function, in this case a piece of branding.

VISUAL MUSIC, MUSIC AND AFFECT
The pieces created for this research into affective visual music are underpinned by wordlessly sung music and other human traces; some human traces are visual, some auditory and some are crossmodal.The affects of music, including human traces in music, have been widely studied for centuries.Therefore, affect in music as well as visual music is considered.In this paper the term affect is used in its philosophical meaning, to emphasise embodied experience.As Massumi writes in (Deleuze & Guattari 1987, p.xvi): L'affect (Spinoza's affectus) is an ability to affect and be affected.It is a prepersonal intensity corresponding to the passage from one experiential state of the body to another and implying an augmentation or diminution in that body's capacity to act.
The body responds with an analogue of the intensity of the affect, as Tomkins and Demos (1995, p.19) state: [In] the facial muscles, the viscera, the respiratory system, the skeleton, autonomic blood flow changes, and vocalisations.
Whereas, affects are non-verbal, emotions are, as Eerola et al. (2017, p.3) state: 'complex, constructed experiences'.As Shouse (2005) identifies, emotions are socially displayed feelings.Feelings are personal because each individual has a distinctive pool of memories of sensations to draw on when interpreting and naming their feelings.Shouse (2005, p.3): The power of affect lies in the fact that it is unformed and unstructured (abstract).It is affect's "abstractivity" that makes it transmittable in ways that feelings and emotions are not, and it is because affect is transmittable that it is potentially such a powerful social force.
The affect of arts is being increasingly studied using interdisciplinary methodologies.Human sounds such as breath, vocalisation, tone of voice, and images of humans showing facial expression or posture can transmit affects that resonate with others more powerfully than the conscious message delivered by the work of art.
Both visual music and music can connote human emotional expression on many levels, some of which are affective: embodying or recreating the expressive quality of human vocalisation, embodying the emotion in human motion, and in moving image a sense of animacy (Heider & Simmel 1944;Thayer & Levinson 1983) or, within music, in Juslin and Laukka's term (2003), a 'virtual person'.Cox (2001) explains how musical meaning is tacitly understood in an affective, embodied manner as the listener, often unconsciously, subvocalizes the melody, imagining the muscle tension of the performer.
I would argue that these 'gestural images and concepts' are also encountered in animation and moving image.Freedberg and Gallese (2007, p.202) include affective response to images and works of art, which: [E]nables the direct experiential understanding of the intentional and emotional contents of images.
Human responses to music and visual music are complex.Eerola et al. (2013, p.487) found that though music expressing sadness is often analogous to a human voice expressing sadness on an affective level, emotions induced by music may be different from those expressed by the music.Eerola, et al. (2017) have interrogated the paradox of enjoying a negative emotion such as sadness in music from a stance of scientific realism.They found the enjoyment of sadness in music has a moderate link to psychological and interpersonal mechanisms underpinning the recognition of feeling.The link to cultural factors, for example history, ethnography, and empirical studies, is suggestive; it fits with a concept of music as a Western form contemplative experience.Memories, the present context, reflective processes such as taking a perspective and imagination also play a part in forming feelings and emotions.However, although music has a measureable effect on a number of hormones linked to positive and negative moods, the link between enjoying sad music to biological factors such as neurological, psychophysiological, or endocrinal responses is weak.
Affective gesture may be accompanied by overt gestures, such as tapping, swaying, dancing, or marching.And of course, there are image-making and music-making gestures too.Additionally, as Morita et al. (2013, p.6) state: [T]here is no one-to-one correspondence between body movements and emotional states.
Taken as a whole, this research underscores a human 'peculiar sensitivity to music' (Sacks 2006), and I would argue affective visual music.It highlights how different individuals can respond so differently to the same piece and the same individual can respond differently on different occasions.It also indicates that music and visual music have the potential to create more unanimity of affect than emotion.This is corroborated by research demonstrating that affective cues may convey expressing sadness across cultures (Laukka et al. 2013), unfamiliar music can induce sadness via 'emotional contagion' a mechanism that relies on affective cues, but sad music can induce joyful emotion (Eerola et al. 2017).

An affective visual music
The lumia, or projected light seen as form and colour in motion, of light artist Thomas Wilfred, though mute, are seminal examples of affective visual music.Cheney praised Wilfred's lumia in (Wilfred 1947, p.255): [A]n art as primitive, as complex, as capable of varied emotional beauty as music; and its medium is light.
Wilfred's work has been profoundly influential and forms links between 19 th -century installation works, such as dioramas, later visual musicians, such as Brakhage, the Whitneys, Belson and Turrell.Turrell in (Orgeman et al. 2017, p.19) recognises that Wilfred's work: [S]peaks to the long tradition of direct perception through the use of light in art.
Turrell cites him as a major influence on his work.Wilfred's compositions have a mesmeric flowing character; the undulating, vibrant, sculptural light shapes evolve out of the darkness and into each other.Deems Taylor quoted in (Orgeman et al. 2017, p.78): These colors, these forms, utterly unconnected with anything we have known heretofore… set the imagination free and they are by turn amusing, exciting and menacing, with flashes of quite unearthly beauty.
To realise his lumia Wilfred invented the Clavilux, a light organ, which he played.The Clavilux was analogue and mechanical; it created refracted and caustic effects from light emitted by light bulbs using transparent coloured records and reflective materials, see Figure 2. In Wilfred's work there is no narrative but there is visual structure.See Figure 3. His mapping of light to lumia stays 'over' technology, and affords huge variation of composition.Wilfred (1947, p.252) identified form and motion as absolutely key, even more so than colour.His sense of light having form as well as motion clearly differentiates his work from painting and film and allows a more threedimensional experience.
Wilfred aimed to create transcendental experiences, quoted in (Youngblood 1970, p.345): Shall we... express the human longing which light has always symbolized, a longing for greater reality, a cosmic consciousness, a balance between the human entity and the great common denominator, the universal rhythmic flow?
Figure 3: Based on (Orgeman et al. 2017, p.40)This closely aligns his work with a 'transcendental field' (Deleuze & Boyman 2002, p.25): [A] transcendental field… a pure stream of asubjective consciousness, a pre-reflexive impersonal consciousness, a qualitative duration of consciousness without a self.
Visual musicians have long sought to create transcendental experiences, beyond normative cognitive experiences, beyond the dualism of eye and ear.They turned to light, to colour and to synaesthesia.Kandinsky (1977, p.25) writing in 1911: Colour is a power, which directly influences the soul.
Brakhage, the Whitneys and Belson also aspired to create transformative, transcendental visual music, underpinned by the concept of synaesthetic experience (Brougher & Mattis 2005, p.145): Their films do not refer to the actual world but instead use optical effects to pluck at the musical inner mind.
The desire to create Deleuze's 'transcendental field', to create experiences that are both prelanguage and extra-language, comes from the desire to fill a vacuum.As Elder (2010, p.xxvi) asserts: The experiences of prayer, meditation, contemplation, trance and dream are not incorporated into modernity's model of normative cognition.
This leaves space for art that is transformational.

Visual structure in visual music
A seminal view of visual structure and visual rhythm is provided by the artist Paul Klee's practice-based research, which was informed by psychology of form, theories of visual perception, psychoanalysis and phenomenology.His writings, such as The Thinking Eye, are contextualised by the framework of his teaching at the rationalist, utopian Bauhaus.His writings stay 'over' technology, they do not form a set of aesthetic or technical rules to be followed, demonstrating (Spiller 1964, p.18): [The] idea of rationality without formulas, rooted in experience and aimed at redeeming the shapeless contents of the unconscious.
Klee collected natural forms.He explored abstract forms as if they were plants or creatures: defining their possibilities for motion, personifying their kinetic changes using animacy to define their essence.He defined two modes of experience: constructive-geometrical and metaphysical.These modes are not discrete; they are interrelated and inform each other.
John Whitney (1990) explains how Klee's concepts informed his thinking and the fundamental building blocks underpinning his work: [A] point of light that moves draws a line and the line in turn moves and draws a plane.
Inspired by Klee, Whitney's methodology for animating was to form Klee's elements into a composition, which became a keyframe, which he rearranged so that it became a new keyframe; the computer interpolated the motion.Whitney noted (1990): [I]t had extreme limitations but it made it possible to generate very fluid dance like patterns in space and it is what I still work with today.
Additionally, Klee identifies how visual structures can demonstrate the dualism of repetition and change.He identifies two types of structural elements 'dividual' and 'individual'.'Dividual' are repeated structural elements that can be divided or added to without changing the structure they form.In contrast 'individual' structural elements are patterns, such as a cross, in which if a part is added or subtracted it no longer forms the same pattern, see Figure 4.

Dividual Individual
Figure 4: Based on Klee in (Spiller, 'Individual' structural elements can be combined with 'dividual' structural elements.Visual composition may be predicated on a constructive geometric basis (even algebraic), a graphic basis, or a pictorial basis Klee in (Spiller 1973, p.189): The structural and individual elements might also be located by a process in which the individual experiences structural articulation in his own body.
For example: the geometry of a cross in a grid, the graphics of text in a pattern, or the pictorial representation of a fish's embodied experience of articulation, see Figure 5.  (Spiller 1973, pp.187-189)When structures are combined visual rhythms are created, see Figure 6 and Figure 7.  (Spiller 1973, p.196) Figure 7: Based on Klee in (Spiller 1973, p.209) Klee notes (1973, p.193) that the 'dividual' structure is unaccented and forms a rhythmic base, whereas the 'individual' structure (which does not have to connect up with itself) is accented 'rhythmicised' and has a higher order of articulation.

METHODOLOGY
This ongoing practice-based research uses methods similar to soundscape composition; visuals are recorded from the real world, sampled, and changed in order 'invite' the audience to reexperience their responses to the real world.The soundscape composer Barry Truax (1996, p.57) explains: [M]y Soundscape Study takes a set of sounds with archetypal imagery …subjects them to a series of transformations in speed, pitch, and textural density (usually independently).The piece invites the listener to follow the resulting changes in morphology and imagery that the transformations produce, and hence to become more aware of how these variables condition our habitual responses to environmental sound.
Similarly, visual elements, derived from photographic or videographic imagery, are transformed and melded with each other to form a composition with a loose, evocative progression or journey, such as perceiving time passing through the changing colours of the sun setting.This 'abstraction' Le Grice (2009, p.290) defines as: [T]he process of separation of the component features and qualities from the 'whole' of an object and as such it [non-narrative cinema] is not synonymous with non-representation.
Ambience-2 (Figure 10; https://vimeo.com/181305587)uses human traces of the singer to direct particle flows in an affective manner, while Shadow Sounds (Figure 11), fuses image and audio together so that 'each sound and animation is consistently used together, as one audio-image unit' (Watkins 2018).Scattered (structure) as against 'compact' always creates a strong contrast, especially when the consolidation is very dense.It is then easy to take the step from an alternation of distinct and blurred structures to an alternation of structure and absence of structure.
On reflection it was felt that though the amorphous shapes in Shadow Sounds to some extent pleasingly echoed Wilfred's amorphous light play in form and colour the lack of 'dividual' structure resulted in an overall lack of visual rhythm, i.e., the individual 'audio-image units' created one surprise after another, rather than built to a whole composition.A new method was used for Sky (Figure 12; https://vimeo.com/237235704).A library of 'audioimage units' based on distinct, but related, sung phonemes combined with particles was created (Watkins 2018).Arguably data-driven transpositions of pitch/frequency and or volume-to-image (strand 2 of visual music) are very briefly engaging, and then become overly predictable.Finding a balance between a purely metaphorical visualmusic sync and too much sync causing redundancy is key to creating engaging visual music (Garro 2005;Kaduri, 2016).Effective use of sync, or how best to match the timing of visuals with wordless song to create affective visual music, was explored by Watkins (2015;2016;2018).It was found that creating 'absolute sync points' by matching the onset of the sound and visuals whilst allowing the rest to follow more loosely created an engaging sense of sync without redundancy (Watkins 2018): The animations are inspired by the impetus and 'spectromorphological life' of the sounds.The singer, Martin Nelson, asked if the shapes were programmed from the sound data, as they seemed to fit so well.It was pleasing that using the sensibilities of an animator resulted in animations that felt so right to him.This library of 'audio-image units' was layered into gathered data.Affective human traces were evident in the vocals and the animation.The restricted frequencies (approximately two pitches) of the wordless singing created a resonant human dissonance, which had a visual analogy in the use of displaced visual sync giving echoed edges and the use of blur in the background.The slowly turning background is analogous to the 'dividual' structural; the whole background is altered by light passes sweeping over it and speed changes.There is interpenetration of the 'individual' 'audio-image units' and the 'dividual' as the particles burn through the background to reveal brighter areas.Ambience-1, Ambience-2, and Ambience-3 were displayed looping, on three screens (Figure 13).This work was inspired by Wilfred's lumia in several ways.The third transparent screen adds a sense of depth to the coloured light in motion that corresponds to the three-dimensional depth of Wilfred's lumia (v) The installation empowers individually constructed subjective experiences in a manner akin to Wilfred's performances.Each of the three screens started on a different section of Ambience.The viewer is free to focus on a large single screen, the interplay of two or three screens, overlapping or not.

Figure 1 :
Figure 1: Based on(Collopy 2006, p.64)Many attempts have been made to make a synthesis between light and music, through colour organs, painting and painting onto film as well as making sound visible by showing oscillations caused by sound patterning sand (Ernest Chladini) or water (Alexander Lauterwasser), or an oscilloscope's display.A direct translation of sound to image can be achieved through algorithmically transforming sonic or visual data into a data set that can generate or manipulate images and sound in real time.Visual music has been very widely defined; the diffuse nature of visual music gives rise to many different outputs from abstract animation to VJing.This paper proposes that visual music has three main strands that are mirrored:  1-A Visual structure informs the composition of both image and sound. 1-B Musical structure informs the composition of both sound and image. 2-A Visually representing sonic data. 2-B The sonification of images. 3-A Moving image created to pre-existing music. 3-B Music created to pre-existing images.This has led to strand 2 being identified as 'pure' visual music, literally what you see and what you hear are one and the same, a 1:1 mapping.Norman McLaren's Synchromy(1970)  is more visually complex than just using the analogue film's optical sound track in vision, but nevertheless gives a good idea of this pure fusion.This 1:1 mapping