Form and Function: Examples of Music Interface Design

This paper presents observations on the creation of digital music controllers and the music that they generate from the perspectives of the designer and the artist. In the case of musical instruments, what is the role of the form (the hardware) where it concerns the function (the production of musically interesting sounds)? Specific projects are presented, and a set of operational principles is supported from those examples. The associated encounter session will allow delegates to experiment with the interfaces exhibited, further informing these principles.


TRADITIONAL MUSICAL INSTRUMENTS
With a few notable exceptions (Wagner tubas and saxophones, for instance), most acoustic musical instruments have developed gradually over time.There are now very few significant differences between available forms of most standard musical instruments, and it is clear that there is little chance of significant further development of these instruments.The reasons for what might be seen as this 'stagnation' are many and various and in part rest on the petrifaction of musical development that seems to be enclosing western 'classical' music.This lack of change also arises from the progressively specialized nature of the genre: the manner in which musicians have approached performance has itself changed over time; today there is much emphasis on virtuosity and precision (also see section 4.4).
These factors may have led developers of music technology instruments to view their interfaces with similar conservatism, although commercial factors have also been important, for instance the basic conservatism of their main market.The significant part of commercial research and development has been in areas other than the development of novel interfaces, and musically it remains unadventurous; typically maintaining links with established harmonic and melodic practices.What effect of these developments will have on the future of musical instruments remains to be seen, as it is clear from the development of 'conventional' instruments that there is little conscious control or planning here.In fact, the evidence of the last fifty years suggests that attempts to develop 'new instruments' are very likely to fail.
Attendance at any live music event demonstrates the importance of the physical appearance of musical instruments.Professional players, pop groups and orchestras know that appearances can be crucial, although not all might agree on a particular style.While function is of crucial significance, form is important, if only from a cosmetic or stylistic view.The fact that a trumpet utilises valves in a metal superstructure separates it fundamentally from the violin with its catgut strings.Although it is often not considered in this way, the summation of the form really does fully describe its function; the music that is created is comprised of the subsequent interaction between performer and hardware.

OTHER INTERFACE EXAMPLES
Another result of the above mentioned stagnation in hardware development has been the increased efforts by some to move development on, even at that risk of appearing to be overly revolutionary and so alienating existing audiences.Early key figures in this process have been Leon Theremin, whose eponymous electronic instrument has some superficial similarities to one of the author's instruments described below, and Harry Partch.
It is important to note the different approaches existing in this work.Theremin is a good example of an inventor whose particular musical creations were tools created specifically as experiments in new interface design.Harry Partch, on the other hand, was clearly more of a musician and composer whose motivations for his development of new instruments such as the Quadrangularis Reversum [30] concern creative expression as much as they do experimentation.This emphasis in the work and in this paper is more similar to that of Partch, rather than Theremin.
There has also been a significant contribution to the field by popular music artists and fine artists with an interest in sound.Examples include Laurie Anderson, Janet Cardiff and Christian Marclay.In the wake of Partch, but utilizing electronics, musicians such as David Tudor and Alvin Lucier, amongst others, have also explored the interface from a creative and experimental perspective.
Over the last twenty years the investigation and implementation of physical computing and embodiment has become increasingly popular.Whatever the reasons for this -the familiarity of standard computer interfaces breeding contempt (although there already seems a role for nostalgia in some places [7]), the rise in interest in making, do-it-yourself and recycling which has also cultivated interests in hardware hacking and circuit bending [3], [9] -the number of performances and analyses of hardware has grown dramatically recently.Perry Cook's intriguing instruments [6] and some of the offspring of the MIT Media Lab [27], [28] have, along with contributions from the New Interfaces for Music Expression conferences and Make magazine [16] changed the academic status of novel interface development in music significantly.Amongst many significant contributions, issues of Computer Music Journal [4], (14:1, 14:2, 22:1, 26:3), Organised Sound [21], (7:2, 7:2), and Contemporary Music Review [5], (6:1).Wanderley and Battier [26] and Miranda and Wanderley [19] provide introductions to the area of physical control of digital methods in the creation of sound and performance.Specific to physical computing is O'Sullivan and Igoe [23] and specific to the Arduino is Banzi [2].
In addition to these there are a number of specific texts, which are of particular interest with regard to this project.Marrin Nakra's 'Conductor's Jacket Project' [17], [18] is particularly illuminating regarding her use of multiple sensors to create and analyse a total music output.Rovan et al., [25] and Hunt, Wanderley and Kirk [13] provide surveys of the general area of mapping.Orio [22] and Hunt, Wanderley and Paradis [14] are interesting in how such mapping effects 'real' instruments.
Probably the artifact closest to the current version of the unit called Gaggle described presently [10,11] is the 'Sound=Space' installation by Gehlhaar [8].This variable room-sized installation is described as having a number of configurations and purposes: including use as sound-art installation, use for dance and therapeutic use.Gehlhaar describes a series of possible topographies for use in different environments and for different purposes, for instance 'changes in themes and rhythms' (Gehlhaar: 68) or action creating a melody.The principal disadvantage apparent in this system concerns its lack of flexibility.The installation is based around a number of units each of which used a single pair of ultrasonic sensors (to a maximum of 48 at the time of the article's writing).These are set up around a space (rather than in it) and the topography is put in place to express particular kinds of activity and in order to obtain particular results.
My own experience with Gaggle is that while further developments certainly involve more work on different topographies, significant areas for development lie in other types of interfaces made from clusters of varying sensors, materials and environments.The realization of a monolithic scene that is capable of being flexible enough to display sufficient quantity and quality of expression is, I think, optimistic.I would suggest that future expression would be small, flexible and heterogeneous.

PRESENTED EXAMPLES
The examples presented here are those that have been developed to a practical, if prototypical, level.They are not proposed as being in any way definitive in either form or function; several others are currently in the planning stage.They are experiments in the influence of design on the creative act: both compositional and performance-based.

Gaggle Figure 1. Gaggle Prototypical Interface
The Gaggle interface [10,11,12] was originally conceived as an improvisatory interface for the control of generative, automatic music for use at the HCI 2009 conference in Cambridge UK, where my colleague Tom Hall and I were invited to contribute to the Open House Festival [20].Generative aspects would control most aspects of the music including pitch, duration and timbre.This in turn was the result of a number of years working in the area of generative or algorithmic composition.The purpose of this was almost entirely in order to help me understand the creative process itself through developing software that emulated it, and a very clear part of that emulation has always been recreating, or at least taking account of those elements of 'liveness' that inevitably make live performance so satisfying.These elements, investigated in depth elsewhere [15], include aspects of indeterminacy, most obviously the repetition of melodic, rhythmic and timbral material with variation and the software encapsulation of various global structures such as the length and order of particular groups of material.
The Gaggle interface comprises nine ultrasound sensors: in this case 'Ping' units manufactured by Parallax [24].These units work by instructing an emitter to output a 40 kHz frequency sound for 200 µs.The pulse is then read on its return and duration of echo calculated.This provides a quite precise indication of the distance of any solid object positioned directly in front of the unit to a manufacturer's limit of 3m.The intention is to custom build these units in future to continue to develop precise control and understanding of the process.The ultrasound units were held in place with 'goose-neck' stay-put tubing (hence the name Gaggle) potentially allowing for significant freedom and customization of placement of the units.This was established initially when I was planning to demonstrate the unit using my own movements.
In this case, the data collected from the sensors was used to control sounds generated in the SuperCollider audio environment.The parameters were manipulated in a number of ways: the 'pitch' directly, the modulation index of a frequency modulated sound, the amplitude of a sounds component, the data triggers certain algorithms that control a number of these parameters.Of course, as the environment is software controlled, these arrangements themselves can easily be changed or controlled.

Figure 2. Dancers interacting with Gaggle: open hands, pulsing
Gaggle was first used publicly at HCI2009, during the Open House Festival where we were commissioned to collaborate with the choreographer Jane Turner and six of her troupe of dancers.It was the way that the dancers interacted with Gaggle that was of particular inspiration in the further development of these units.Figure 2 is a photograph taken of one particular interaction.In this case, the dancers, through structure improvisation, arrived at the point where they found that this sort of pulsing movement provided a rewarding sonic/movement-based result.
It is important to appreciate that the parameters controlled were not straightforward.A particular movement could produce a variety of controlled but relatively unpredictable outcomes.Working with the level of predictability -is an important issue in this work (see 4.5).
The relationship that could be seen developing between the dancers and Gaggle puzzled and inspired me; in particular it made me consider other aspects of interaction that might occur.Without any special prompting from choreographer Jane Turner [31], they moved around the space, utilizing its features, such as a large tubular pillar and during the actual performances the Festival attendees themselves, building up an interactive space in which their relationship with the unit and their environment could develop.
Figure 2 also illustrates an example of one of the key motivators in the performance: the attempted control of the sound by physical gesturing.A primary use of the analysis of the dancers' behaviour in this situation for a composer is the use which subsequent designs of both interfaces and algorithmic responses make of the various physical gestures.It is this mapping which is the metaphor itself and the extent to which actions should correspond or contradict the relevant metaphor is of crucial significance in determining the nature of the resulting work.
At one point the dancers circled Gaggle with some velocity, sweeping their arms up and down outlining 'waves' around the unit.This suggested that the dancers felt (correctly) that more movement indicated a greater number of audio events.So, the metaphor used in this particular case was that greater movement means greater sonic activity.The movement reflects the 'design' of the unit in that circling it is the best way of creating movement near it and so generating the movement required of the metaphor.On another occasion the dancers utilised the same movement, but in a different location: a circling motion conducted away from the device.The sonic result here was that only a part of the audio material was created in the way that it was from the movements previously described, resulting in a form of echo of that material, but with aspects missing and others radically altered: there was significantly less timbral modulation, for instance.This was quite appropriate metaphorically and worked well in this sense.
There are further details and analysis of these movements in Hoadley [11].

Wired
Wired is a prototypical interface utilising properties of touch.This circuit is realized by utilizing the high input impedance of the trigger pin of the 555 Integrated Circuit (IC).When the IC is triggered by the induced voltage of human body the output goes high for a time determined by the values of a resistor and a capacitor.This enables a simple structure such as a wire to be used as touch sensor.

Figure 2. Wired Prototypical Interface
The design has been very much influenced by the use made of Gaggle.The experience of working with dancers and their freedom of expression through physical movement showed that they very much enjoyed interactions with and investigation of interesting and novel objects.The device has been demonstrated in highly prototypical form at the Museum interfaces, Spaces and Technologies (MIST) workshop in Cambridge in March 2010 and while there were certain technical issues showed again how eager delegates were to experiment and investigate unusual devices.The device was praised for ease of use and for enabling those with little or no musical experience to 'perform' in a pleasing and expressive way.The value of this quality is another matter that will be touched on below.

Touchable/Approachable
The Approachable is a unit currently in preparation; no actual prototype exists.The impulse is, again, experience of work with dancers as well as from seeing interactions with the public.It seemed clear from these experiences that approaching a unit was one of the most natural of actions, as shown in Figure 2.However, when interacting with the Gaggle, they were not enthusiastic about touching the unit, and to be fair, I wasn't at all happy with anyone actually touching the unit.However, it seemed that it would be one of the more natural behaviours and so the idea of the Approachable is that it should react to both proximity and pressure.Performers could investigate positions in the continuum between a point of significant distance from an object and a point allowing lots of physical contact and pressure.
An example might include use by multiple performers from multiple perspectives.The position of a performer on one side of the Gaggle might determine the nature of what might happen with the data from a performer on the other side: so, the closer the first performer was, the more violent and active the general algorithms were (although the detail of the movement might be controlled by the second performer), until the first performer reached a certain point, when the whole texture might change to something very gentle and soft.

ISSUES
In this section I attempt to tease some generic, work-in-progress questions and issues from the experience of the implementation of the above devices.
Cook in an entertaining paper [6] suggests that programmability is a curse, and that one should write music, not develop controllers.He also gives an indication of some of the pitfalls of interface design and suggests principles that might be considered.
Ten years after the publication of Cook's paper it might be possible now to consider again issues and questions that might be used as starting points in the continuation of this research.

What is the relationship between function and form?
How feasible is it to discuss how Beethoven's Violin Concerto is influenced by the design of the violin?On one hand, the relationship is fundamental but on the other, the two are so intimately linked that to draw attention to the link seems almost tautological.But is this not equivalent to saying that function and form are in practice the same?We have been so used to the idea that electronic replicas or extensions of instruments, (even laptops themselves) are musical instruments, perhaps we find it hard to see clearly in this subject.
Existing instrumental designs also effect the way one wants to play them -so the typical use of a keyboard to control synthesisers encourages users to play them like pianos, whatever the sound or texture is being played or is most appropriate.

'The instrument is the composition'
While there have been many attempts at making new instruments to replace existing ones, so far the replacements have in general signally failed to make much impact on the usual selection of 'standard' instruments.
This may be because performance on a musical instrument is the totality of the experience of a real human manipulating a real object.What, from this continuum, is it possible to use in the HCI? Devices such as the iPod show that it is not necessarily the total functionality of any particular device that is important, but the balance between capability and ease of use.The latter might quite explicitly require what might be interpreted as a reduction in functionality.
The third option is both a combination of these and a rejection: the instrumental design becomes a part of the creative process itself and is no longer assumed to be an independent item (although this possibility doesn't need to be ruled out).

Is programmability a curse?
Programmability is one thing that is not possible in the domain of the 'real' musical instrument.One has only 'real' options: physical interferences such as muting, mutating and hacking (sometimes literally).Things that are programmable do not possess that boundary of solidity beyond which we cannot go.
We have either a flute or a clarinet.Replace a flute's mouthpiece with a clarinet's and what do you have -a soprano saxophone?Maybe, but not a flute.However, your (Yamaha/Roland/Akai) 'hyperflute' can be anything you wish -a flute, a trumpet; even a drum machine!So what is it exactly?A synthesiser.

Performer, composer or improviser?
How have attitudes towards instrumental playing changed and how have these changes resulted in practical changes in performance?Reports suggest that performances before the twentieth century could be very different in quality, tone and content from what we might expect [29].Today there would appear to be a much more acute interest in precision, virtuosity and exactitude in approaches to 'classical music'.This interest may have isolated this genre from other more popular and experimental activities.
There is a difference between 'interaction with things and the creation of music' and 'a musical instrument'.
Aesthetic design plays a significant role in defining how a 'performer' might interact with any given object.
What effect does the perceived 'role' of the subject play here?
In a gallery-type environment, does the visitor become a performer?What do they want from such an experience?How much prior experience/learning should be involved?If a musician or other deliberately chosen performer is the object, do they require more from a unit?Would they want more potential for control in order to increase possible expression?A traditional instrument like a violin might provide a useful model here: most people are unable to coax a particularly pleasant or musically sophisticated sound from a violin on first encounter, but the violin has, of course, enormous potential for expression.
Finally, how feasible is it for a unit to 'perform' well in these different contexts?Initial experience with these units suggests that while initially interesting, this level of interest soon wanes without an intriguing structure or some other activity to engage attention.

Multiple parameters and conscious control
One of the features of acoustic instruments is that, while in comparison to their technological counterparts they may seem simple, in reality they are not.We have become used to these interactions and tend to ignore their most important featuresmost obviously, the quantity of information available from any 'simple' expression.This information comes about through the use of continuous control information on a set of simple but continuous parameters.A flute has a fixed number of finger holes, but the breath control is continuous and infinite: there are unlimited ways of controlling a flute's tone through breath (also finger holes need not be fully opened): it is certainly the most significant factor in expression on the instrument.Any 'standard' acoustic instrument has similar factors.A musician practices using these continuous controllers and if they becoming a good player usually means no longer needing to utilise conscious control; lower level activities such as fingering and breathing become automated, allowing more concentration on higher level tasks such as musical expression.
One of the main experimental strategies in developing new units is in concentrating controllable parameters in particular areas and using particular sensors so that conscious control of all parameters is difficult or undesirable.This itself is a stopgap solution to a larger problem: eventually users get used to any particular set of controls and as this point is approached unless some other goal is identified boredom may result.

Latency and responsiveness
This is a technical issue involving the speed and quality of response.Imaginative users often first test for this.They frequently gesticulate as in Figure 2, starting slowly in order to ascertain how the unit will react initially, but soon after they will test the speed and abruptness of a response, using sharp and sudden movements and seeing how the unit responds.Gaggle has not been set up to respond in this way, although it could be programmed to be more immediately responsive.One of the reasons is that the Arduino card used in the current manifestation is not enormously fast, and although a fairly large amount of quite high-resolution data (0-1023) is transmitted, with nine sensors there is inevitably some latency.This matter can easily be overcome with cash, by using a faster interface such as the Teabox [1], which operates at audio rate.As this product is significantly more expensive that the Arduino board developer may wish to be meticulous in deciding when to use each board.It is also sometimes possible to overcome such latency with specific programming.

CONCLUSIONS
This paper briefly discusses work and observations made during the development, implementation and use of a variety of hardware interfaces for the control of audio software.It outlines the complex line of interactions that occurs and needs to be taken account of when undertaking this work, and proposes some principles, questions and issues that might direct future research.Each individual's 'encounter' with the units provides a unique insight into that person's previous experience and methods of interaction, as well as into their 'requirements' in creative terms.
Aesthetic design ultimately affects the way in which the user encounters the unit, and this experience plays a fundamental role in that users understanding of that encounter.
How this understanding changes and develops in time is one of the most basic questions in music.