KIMA – The Voice: Participatory art as means for social connectedness

Oliver Mag Gingrich Urszula Tymoszuk Evgenia Emets Alain Renaud David Negrao National Centre for Computer Animation Media School, Bournemouth University, UK olivergingrich@gmail.com Centre for Performance Science, Royal College of Music Imperial College London, UK urszula.tymoszuk@rcm.a ac.uk Artist Lisbon, Portugal emetsjane@gmail.com Bournemouth University Geneva, Switzerland AAlain.renaud@ mintlab.ch Artist Lisbon, Portugal davidnegra@gmail. com


INTRODUCTION
With this development, we are looking at mathematically meaningful interrelations between human voices (harmonies, intervals) and what the experience of such harmonies can mean for the individual in a social context. Creating a sonic and visual composition with the raw material of the recordings of a simple sequence of sounds from the human voice and real time voice of the audience, sonic harmonies as represented visually. For the audience, this facilitates musical collaboration, and social engagement. Working together with researchers from the HEartS project investigating Health, Economic and Social impact of the ARTs engagement led by the Centre for Performance Science (a partnership between the Royal College of Music and Imperial College London) we will explore the role the arts can play in increasing the understanding of experiences of social connectedness. Specifically, we will investigate strategies to combine the art piece with measuring experiences of social connectedness and wellbeing.
KIMA: The Voice focuses on the human voice as a tool for social engagement. Each 'vocal signature' and its interaction are captured -an intelligent machine learning algorithm "trains" the system. The more vocal signatures KIMA captures, the more precise the vocal analysis and visual interpretation becomes. The system is capable of learning to differentiate between intricate nuances of the human voice and the pure tones, thereby improving its visual response to sound by offering an increasingly more meaningful and exact visual expression of sound for its audience. At the same time the system is looking at the intervals between the human voices and pre-recorded sounds, creating a visual interpretation of this complex realtime composition. Each member of the audience can contribute with a tone, a sound to be woven into the overall sonic composition, where all the voices of the audience will collide sonically and visually.
This paper discusses the project KIMA: Voice on a conceptual, technical level and explains how we are looking to integrate novel ways of measuring audience experience, wellbeing and perceived social connectedness throughout the experience, as well as presenting the past, present and future of the project.

KIMA -PAST, PRESENT AND FUTURE
The art collective Analema Group has worked on their signature project since 2011. First conceived as an art and research project on physical properties of sound also known as cymatic patterns, this project has seen several phases of intense research with the subject matter. Integrating practice-based research with a thorough investigation of mathematical representations of sound waves, the development can be structured into four following phases.

Cymatic patterns & motion control
The relationship between sound and vision has multiple tangential points, not at least in physics, mathematics, and art. During the initial research phase on the subject matter, the art collective looked at the phenomenon from a multidimensional point of view, including experiments in media art, an exploration of contemporary art strategies in visual sound and first steps towards an integrated interface for KIMA for motion control and visual sound interaction. This first iteration of KIMA -was first premiered at Kinetica in 2012 with performers Victoria Cooper, Anna Buonomo and developer Johnny Stutters. The project presented real-time visualisation of cymatic patterns controlled by a live performer through motion interaction (Gingrich, Emets & Renaud 2012). Although effective in its performative qualities, the piece lacked an interface for its audience and visualisations remained rudimentary and only came to realise its potential at the International Youth Art Festival Kingston through the input of visual developer Joe Pochciol, in a performance by Satoko Fukuda and Dane Hirst (Rambert). KIMA interpreted real-time visualisation of a violin performance as a cyclical representation of frequencies.

KIMA -a telematics environment
With KIMA, Analema Group mapped basic sound parameters onto a rectangular plate, to be controlled and explored by the audience -either through voice and motion (Festival of Learning), other music instruments (Kinetica 2014) or live performance (International Youth Art Festival Kingston 2014). The collective implemented a series of activities that explored strategies for effective audience engagement, while exploring visual expressions of cymatic patterns that are both intuitive and artistically compelling.
As part of this development, we brought the activity to the UK's biggest Deaf Arts Festival, to see how this particular community engages with visual sound as an innovative form of sound experience. The work was represented in collaboration with artist Robert Jack (Queen Mary University of London) who developed a bench that provided sound feedback on resonance in form of vibrations. Our goal, an accurate representation of sound waves as mathematical correct simulation of cymatic patterns on 2-dimensional plates was only implemented during our performance at the Union Chapel which included an array of diverse artists and performers.

Mathematical representation of sound
Whereas previous activities succeeded in showcasing visual sound in unexpected and challenging new ways, a literal transcription of physical phenomena in the digital realm, had remained unresolved. Together with visual developer and artist Marcel Schwittlick, the team tackled this challenge at the Union Chapel. The famous Henry Willis organ, is hidden from the audiences view, but provides an incredible soundscape for music performances. For their takeover of the Union Chapel, the team wrote three conceptual scores for an array of artists to perform as real-time improvisation performing with mathematically accurate visual representations of their sound on the ceiling of the Union Chapel.
The performers including Robert Ames, the artistic director of the London Contemporary Orchestra, Claire M. Singer (Union Chapel) and Estela Merlos (Rambert Dance Company) contributed to the immersive performance of real-time cymatic patterns on the ceiling of the Union Chapel. Emily Burn's vocal performance of the final piece provided context for the direction of the piece.

Machine learning & immersive sound
The collective decided the further explore two vital aspects of the project: concentration on the human voice and participatory nature, at an interactive performance at the Roundhouse London. For the 50th anniversary of this important London art and performance venue, the art collective was invited to perform a new iteration of the piece "KIMA: The Wheel" on acclaimed artist Ron Arad's Curtain Call -an 18m wide circula installation of silicone rods forming an immersive digital canvas for the piece. The audience was invited to explore their voice across six visual zones, in the art piece that represented nuances of the human voice (timbre) through a combination of sound analysis, machine learning algorithms and immersive sound. The piece invited around 250 participants to explore their voices individually and to collectively contribute to a collaborative audio-visual composition. Contextualised through spatial pattern poems by Evgenia Emets and a performance cowritten by Nao Masuda and Ben Zucker and a spatial sound design by Dr. Alain Renaud, the piece set nice benchmarks for real-time machinelearning based representations of sound.
The follow-up performance of KIMA: The Wave, concentrated on a multi-dimensional approach, contrasting these sound visualisations with interference patterns that could be felt as deep frequencies at the L'Abri venue in Geneva. A new iteration of the piece could be experienced by playing with singing bowls and seeing a representation of wave interference phenomena as projection piece on the stone walls on the walls of the bunker.

KIMA -THE VOICE
With KIMA: The Voice our focus shifted from one human voice to interactions between human voices. In this development, we look at harmonic scales and simple mathematical relationships between sounds i.e. intervals. Preference for harmonic or consonant tones can already be measured in human infants (Perani 2010, Trainor 2002, Virtala 2013 but are also linked to their preeminence in Western music (McDermott 2010).
Pythagoras is widely credited as being the first one to connect musical harmonies with mathematical ratios and to this day, we relate harmonies to integer multiples of fundamental frequencies in contemporary acoustics. Research by Shapira Lots and Stone suggests that humans prefer consonant pitch intervals of an octave or a fifth rather than 'dissonant' intervals such as a tritone, originates in the simplicity of the ratio of these frequencies. The ratio between 1:2 for the octave or 2:3 for the fifth is both mathematically as well as sonically elegant. With KIMA: The Voice, Analema Group created an interface for audience participation, where the audience is invited to create harmonies between sounds.
The project draws inspirations from the presence of sound art such as the works by Pauline Oliveros, her focus on deep listening, collaborative and participatory approach to engage audiences resonate with our work. 'Raw Materials' by Bruce Nauman engaged with voice samples, creating an immersive sonic environment in the space, all the while working with the voice differently.
Recent inspirations for the piece include "The Voice and Everything else" by Mikhail Karikis, whose work reflects on voice as creative tool inspired by the Futurists. Artists have long investigated the spatial dimension of sound, and its effect on the experience of the audience -such as Janet Cardiff with her immersive sound installation 'Forty Part Motet' or Tarek Atoui's recent accent on the physicality of sound and Bernhard Leitner's sound spaces.
Quoting Luigi Russolo 'The Art of Noises: Futurist Manifesto', 'vowels represent sound in language, while consonants clearly represent noise' -we embrace both. Through our experimental vocal work, we explore what falls between the cracks in language. We already developed a visual representation of sound for our piece 'KIMA: The Wheel' at the Roundhouse, which offers the audience a clear clue of the difference between vowels, consonants and timbre of each voice. We invite the audience to explore textures and phonetic qualities of language, its multitude of sounds, to find the hidden potential in their own voices, while co-creating a sonic, poetic and visual composition. But with KIMA: The Voice, we are not only focusing on sounds emitted by the audience, but also on the harmonies that evolve between them.
Mathematically speaking, we look at incoming frequencies and analyse the ratios between sounds to look for consonant harmonies such as unison, octaves, perfect fifth, perfect fourth, but also more abstract ratios such as the golden ratios are visually represented. The audience thus explores their voices visually as well as the space between them. A study by Bidelman and Krishnan points to the idea that brainstem responses to consonant intervals were more robust and their 'pitch salience' was stronger than in consonant ones (Bidelman & Krishnan 2009). This approach is consistent with the model of neural synchronicity proposed by Shapira Lots and Stone (2008). The researchers from Tel Aviv University argue that our response to pitches is governed by neural responses oscillating when activated by the cochlea. When two incoming frequencies are coupled, they can become 'modelocked', thus reinforcing their response. This modelocked synchronisation can happen at any meaningful intervals or frequency ratio such as 1:1 (unison), 1:2 (Octave), 2:3 (Perfect Fifth), 3:4 (Perfect Fourth), 4:5 (Major Third) or 5:6 (Minor Third). With KIMA: The Voice, we represent meaningful intervals visually, guiding participants towards meaningful, harmonic sound intervals.
KIMA: The Voice acts as a visual tuner, providing a visual reference point to harmonies, an instrument that results in the exploration of consonant sounds. With this development, we leave the Western conception of musical scales towards a purely mathematical reading of sounds. Every emitted sound frequency can act as a reference for a harmony, no matter whether this represents a musical note in itself or not.
KIMA is the first visual sound art piece that focuses on vocal harmonies in creating sonic, visual and social connections between participants. These meaningful interactions translate into immersive visual sound experiences, that may alter the audience's perception of social connectedness. The piece has been exhibited at ART IN FLUX 2018, By Other Means Gallery in June 2018, at the CDE showcase in London's Digital Catapult as well as at Maxilla Space and will now be exhibited at the Great Exhibition Road Festival, and evaluated in terms of benefits experienced by the audiences.

PARTICIPATORY ART AND WELLBEING
The visual representation of sound for our piece 'KIMA: The Wheel' at the Roundhouse offered the audience a clear clue of the difference between vowels, consonants and timbre of each voice. Through a machine learning algorithm, we collect a library of public voices. Machine learning algorithms learn through the richness of input of sonic data, to interpret sound visually. We invite the audience to explore textures and phonetic qualities of language, its multitude of sounds, to find the hidden potential in their own voices, while co-creating a sonic, poetic and visual composition. For the past two years, the collective focused on various forms of audience participation. Whether through music instruments or human voice input, the audience experienced a visual response to sound in realtime. Iterations exhibited at 'Incloodu Deaf Arts Festival' -the largest arts festival for the Deaf Community in the UK, 'Kinetica Art Fair' and Union Chapel. We bring the audience into the centre of each performance through spatial arrangements of sound as well as interactive visual elements enacted by the audience. The piece is conceived as participatory, in the tradition of an 'open work'. Such an 'Open work' suggests various open ended possibilities for the actual final composition, where every participant, performer or member of the audience, contributes and therefore changes the course of the performance.
Analema Group is currently collaborating with the HEartS research team at the Centre for Performance Science (an interdisciplinary research centre with an award-winning track record of research and knowledge exchange in the arts and wellbeing) to understand the effect of participatory artwork on audience's perceptions of social connectedness and wellbeing. The role of the arts in promoting health and well-being throughout the life course is increasingly documented (All-Party Parliamentary Group on Arts Health and Wellbeing, 2017;Lomas, 2016;Royal Society and Public Health Working Group, 2013). There is now evidence to suggest that across art forms (e.g. music, literature, the visual arts, and drama), engagement with arts-based activities can enhance various aspects of well-being and subjective experience, for example enjoyment, meaning, bonding, and aesthetic appreciation.
The role of participatory arts in promoting feelings of social connectedness or preventing feelings of loneliness has been relatively overlooked in empirical research. Previous research body from the Centre for Performance Science using a range of methodological approaches suggests that engaging with music, often a social activity in itself, can enhance social wellbeing and facilitate creation and maintenance of social connections. For instance, music-making has been found to enhance social interactions in older adults (Rhythm for Life, Perkins and Williamon 2014), mental health service users (Making Music for Mental Health, Perkins et al. 2016) and new mothers (Music and Motherhood, Perkins 2017, 2018). Nonetheless, the few studies which explored social benefits of arts engagement focused on participation in single, traditional art forms, and the benefits experienced at the individual level.
With KIMA: The Voice, we want to take it one step further and discuss whether we can actively measure how art can facilitate social interactions and allow our audiences to connect with one another in meaningful and culturally enriching ways, while furthering social connectedness, social inclusion and creative engagement. Our research will explore whether participation in KIMA: the Voice affects participant-level and pair-level perceptions of connectedness, closeness, loneliness, and happiness.
We will also pilot an innovative way of synthesising participant-reported experiences with artworkcaptured data. With KIMA: The Voice, we are now looking at various strategies to measure the duration and the intensity of audience interactions (e.g. different age groups or people with lived experiences of different mental health conditions) by focusing on how long audience participants are harmonising for. This data in combination with selfreported measures will help to relate audience participation directly to its effect on audience interactions. We further hope that by integrating very specific measures on meaningful interactions (such as the duration of engagement, duration of vocal harmonisation) and by correlating this data with participant self-reported, we will address some challenges of measuring social impact of participatory art caused by methodological issues, complexity of art practices, as well as the heterogeneity of participants.
In the long history of the project, KIMA has evolved into a constantly changing art and research piece that provides our audiences with ever-new ways of relating sound to vision. Whether by relating sound to physics and mathematics, as a real-time performance piece, or by inviting our audiences to explore the power of their own voices -KIMA is an example for how participatory art can play an active role in connecting humanity through the creation of culture. The collaboration with HEartS will further highlight the effect of participatory art for health and wellbeing over the coming years.