MiMic : a motion control interface for music

The goal of this project is to provide an interface that frees the ‘laptop musician’ from the laptop and in doing so, encourages more interactive performances within the genre of electronic music. This paper examines interactive systems that use motion and other sensors to control computer music applications and discusses interface design in this context. The various interfaces already tackling this goal demonstrate issues with regard to complexity, affordability and usability. We are particularly interested in interfaces that utilise simple design concepts to provide a flexible, intuitive controller that is appealing and affordable to a broad range of electronic musicians.


INTRODUCTION
A number of interactive systems have been developed to use motion tracking and other sensor technologies to control electronic music.This approach has obvious advantages over the standard computer user interface (mouse and keyboard), as it allows the performer to be free of interacting with computer screens or hardware, enabling greater interactivity with the audience.These range from glove-based interfaces like the Fortier Marshall Gloves (Marshall, 2009), to elaborate whole-body suits, such as Sonalog's Gypsy MIDI.Many of these attempts are composed of a complex array of components, thus offering an extensive range of control options.This has led to a number of impressive yet somewhat complicated systems.This paper looks to analyse the limitations that arise from such complex approaches, and to propose an alternative approach that aims to provide usability through simplicity.
In consultations and discussions with performing artists, a list of user requirements has been assembled defining an 'ideal' multi-purpose interactive performance controller interface: • Provides the user with greater physical freedom.• Adaptable to a range of computer music applications.• Provides the user with an intuitive and engaging interface.• Capable of effectively controlling multiple parameters simultaneously.
• Offers both effective preset control designations and the option for user assignable controls.

EXISTING DEVELOPMENTS
Interactive interfaces with sensors as a form of controlling music have been in existence for a number of years.Early uses include The Hands, (Waisvisz, 1984) hardware from which utilises a variety of motion sensors to control synthesis parameters via MIDI.The advent of software such as Max/MSP and Pure Data brought about new control methods, such as the Lady's Glove (Sonami 2010), which features multiple accelerometers, ultrasound distance sensors, and pressure sensors.
Other approaches including the use of electrical signals from the body itself: The BodySynth (Severinghaus & Van Raalte, 1998) utilises up to 12 EMG sensors to collect muscle contraction data, and converts them to MIDI data streams.IBVA Technologies also produce a system that measures brain wave data directly from the user's scalp via EEG, and implement them in musical applications for hands free control.
The FM Gloves (Marshall, 2009) were originally designed by Pierre Yves-Fortier, and later refined by Mark Marshall.The goal was to create a glovebased controller for synthesizer parameters.The unit uses a combination of force sensitive resistors, accelerometers and an IR sensor.These are configured as follows: • Left hand: Force sensitive resistors (FSRs) on each fingertip and a 2-axis accelerometer.Striking the FSRs triggered notes; the force was used set the velocity, with each sensor assigned to a different pitch.Accelerometer data controlled timbre and filter parameters.
• Right hand: An IR sensor measured the distance of the hand from the body, data from which was used to control volume.FSRs were utilised in the same manner as the left hand.
This system was created solely as a synthesizer controller.As such, each control is designated a specific parameter.These gloves proved successful for their purpose, with test subjects attesting to their degree of familiarity and intuitive control.Furthermore, as the system is wireless and all components used for data capture are built into the gloves, the user is not limited to performing in a specific location.
The Scangloves (Kessous & Arfib, 2004) were designed more as a research piece than a performance instrument; they demonstrate the application of gesture control to sound and video synthesis.Each glove is unique -one is based on a 5DT Data Glove, a commercially available product equipped with inclinometers and bend sensors to detect flexion of each digit.The other is purposebuilt for the system, incorporating two bend sensors and two force sensitive resistors (part of an I-CubeX system).The data from these gloves is sent to Max/MSP, where it is used to control variables within a scanned synthesis system.
VAMP (Jessop, 2009)  • A pressure sensor on the index finger.
Pressing this 'freezes' the note currently being sung via processing in MSP.• A 1-axis accelerometer on the hand (aligned to collect data in the direction that a conductor would perform a downbeat).
Peaks in this data stream trigger the sustained note to be 'pulsed', created a beating effect.• Two bend sensors, in the elbow and wrist.
The elbow sensor controls crescendos/ diminuendos according to whether the joint is extended/flexed.The wrist sensors control harmonisation; the angle of the wrist corresponds to the amplitude of an additional sustained note.
• Maxstream Xbee RF Modules to transmit the data from these sensors to Max/MSP.
One of the key advantages of this system is the consideration with regard to sensor mapping.This makes the system particularly intuitive, as sensor placement is sympathetic to natural actions, such as the motion of extending the arm to implement a crescendo.
Each of these glove-based systems proves that motion control is capable of providing innovative and intuitive techniques for the performance of electronic music.However, as they only control specific parameters, they do not demonstrate that this approach can provide a flexible platform to control a range of computer music applications; each system has been carefully configured for a specific purpose.
The Body Suit (Goto, 2006) is a more elaborate control system with a broader range of applications.
The suit consists of bend sensors on each joint that are used to detect the motion of the user.It has been used in a number of scenarios, including audio-visual performance and in conjunction with Goto's percussion robot, Robot Music.For this, the suit triggered sequenced percussive events as well as simultaneously controlling other parameters, such as the delay time between hits.
Just as the glove-based controllers, this system demonstrates the effective use of motion sensing to control a number of parameters simultaneously.However, the greatest advantage of the Body Suit is that it proves that a single motion control system can be adapted to a variety of applications.
Sonalog's Gypsy MIDI is a commercial product that uses a system of gyroscopes and potentiometers to convert the performer's movements into digital information which is then transmitted wirelessly and interpreted by eXo, Sonalog's MIDI mapping software.Key advantages of the system include: • It does not require calibration for use in different environments.• As the resolution of control data remains consistent irrespective of distance from a base station, it is limited only by the range of the transceiver system, in this case 180m.
There are also several drawbacks to this design.Firstly, the hardware, though light, is awkward; limiting usability.The mechanism offers 27 assignable controls per arm, which can then be mapped to destinations of the user's choice.However, a default mapping calibration is not supplied, so the user is left to experiment for themselves.Although it is important for the product to be user customisable, many electronic musicians want something that they can use 'out of the box'; a range of preset examples that demonstrate how to make the most effective use of the product.This provides essential guidance, helping the user to familiarise themselves with the system before experimenting with their own configurations.
Jazzmutant's Lemur is a multi-touch and modular controller.It incorporates a 12" multi-touch TFT screen, which can be used to display and edit parameter values within the user's computer system.A primary feature of the system is its adaptability to user requirements; the controller destinations are fully assignable and the user interface is also entirely customisable.Furthermore it is compatible with Ethernet, OSC and MIDI, making it compatible with a broad range of setup configurations.As the controller is multi-touch, the user is able to edit multiple variables simultaneously.Another advantage of Lemur is the ability to allow the user to change screen in accordance with the function that is being controlled.This enables the same interface to be used for a variety of applications.

Analysis of design concepts
Each of the interfaces discussed have demonstrated a number of key concepts to include, as well as potential issues to avoid with regard to developing a multi-purpose interactive system.
As the glove-based controllers were all built for specific purposes, their complex array of sensors cannot be easily adapted to a range of applications.
On the other hand, the open-ended approach adopted by Gypsy MIDI is limited by the overwhelming number of control sources (54), coupled with the fact that little guidance is provided to enable the user to fully exploit the capabilities of the system.It also lacks the insight with regard to sensor placement demonstrated in projects such as VAMP; whose intelligent sensor placement allowed for a highly intuitive system.
Non-motion sensitive interfaces have also provided a number of valuable design concepts.Another key feature of the Lemur is the ability to use different modes to enable the user to change the function of the device.

DESIGN AND DEVELOPMENT
This section discusses the design of our system, employing concepts that we believe to demonstrate an intuitive, reliable, and flexible solution.Many of the core concepts are based on the 'KISS' engineering principal -'keep it simple stupid'.Using this approach, a design has been developed with simplicity, efficiency and usability in mind.

Goals
There are a number of key goals we have aimed to achieve through this project.The first is to enable performers of electronic music to control samplers, synthesizers and sequencers through the use of motion, to an extent freeing them from the conventional confines of hardware.As such, performers will be more easily able to engage the crowd and add a further dimension of interaction to their performance.The second is to do so by designing a simpler, more intuitive interface that is as or more effective than existing motion tracking control systems.A third objective is to provide something affordable and thus accessible to a broad range of electronic musicians.

A user's perspective
From the user's perspective, the system will consist of the gloves (fitted with the sensors, transmitters and LEDs), a base station receiving incoming wireless data, and the software that they are controlling.When the user moves their hands in view of their webcam, corresponding cursors move onscreen, providing visual feedback (see Figure 2).According to the user's preference, either the right or left hand can be used as the primary controller.This hand will then perform higher level tasks, such as navigation and selection within the software.
The secondary controller provides specialist control over certain software parameters.For example, when editing the reverb effect, the secondary controller becomes active, and the horizontal distance between the two hands sets the reverb's room size parameter.The vertical axis of each cursor is also utilised to control effects parameters, in this case with the left assigned to reverb colour and the right to wet/dry mix.Each finger also provides the user with a further level of control.As with the hands, the fingers also follow a hierarchical layout as demonstrated below.
For the second hand, the functions of the second, third and fourth finger will vary according to the current software mode.

Hardware design
The physical controller is centred around two gloves, each fitted with four bend sensors, which continually monitor flexion of each finger.The sensor data is encoded by a PIC microcontroller, serial data from which is transmitted to the user's computer via a wireless network comprising of three Maxstream Xbee modules.The independent motion of the two gloves is tracked via webcam, and by utilising colour tracking algorithms within Max/MSP Jitter.
This system detects coloured LEDs built into the gloves; different colours have been used for each glove to allow the software to differentiate between hands.Max/MSP then uses this data to control parameters within a number of custom-made software environments.

Coloured filters: refining LED detection
In order to filter out unwanted light sources and limit the brightness of the light from the LEDs, coloured filter gels are used.These act as bandpass filters for the colour of the LEDs used, allowing their light through whilst dramatically reducing interference from other sources.Given that the LEDs are extremely bright anyway, this does not present a problem.In this manner the two desired light sources are easily isolated, making them substantially easier to track.

Software design
The software interface consists of a core screen where up to 16 samples/loops can be assigned by the user.From here the user can toggle loops on and off and control a number of effects in real time.
For each loop, a series of sub menus can be accessed, allowing further control over looping and effects parameters.In addition, a simple 'MIDI out' mode also allows the controller appear as a MIDI device on the host machine.

HCI
In order to enhance both the impact of performance and the intuitive qualities of the product, the motion tracking software has been designed to support perceived correlations between types of movements and the resulting variation in sound.Furthermore, the core component of the sampling interface has been designed to resemble existing hardware to provide a degree of familiarity.
Considerations have been made for finer control functions of the system: the right and left hands are independently assigned to general areas of functionality, meaning, for example, for a right handed user, the right glove would be used for fine controls, such as navigating sample matrices, and the left for less specific applications, i.e. effects parameters (filter cutoff, delay feedback etc.).

PROTOTYPE
These design concepts have been implemented in order to encourage the user to realise logical relationships between their motions and the respective functions of the software.In doing so, the user will be able to use the system with a range of software and be able to anticipate how their movements will be interpreted.This approach to mapping presets allows the user to focus on augmenting their current setup with the system, rather than needing to familiarise themselves with an entirely new instrument.This minimises training time, enabling the user to quickly and effectively implement the system into their current setup.

User testing and validation
User testing is currently under way, with results already having an impact on the project.Initial hardware tests revealed that the latency of the momentary triggers was too great; with delays of over 80ms.Refinement of the ADC's and of the implementation of the wireless protocol brought the latency down to an acceptable range of between 7 and 15ms.Following this, user testing has yielded positive results, with users quickly becoming accustomed to the control layout and demonstrating proficiency with the system in a short timeframe.

CONCLUSION AND NEXT STEPS
Based on the user requirements, we have designed and developed a product that utilises simplicity to provide an intuitive and flexible control interface.By primarily utilising equipment that is already readily available (e.g.webcams) the solution is highly affordable We believe that the simplicity of this design makes it an excellent basis on which to develop; it utilises hardware that the user has access to and is familiar with (i.e., a webcam) and augments it with a reliable, simple, yet versatile array of controls.In addition, the fluent and precise nature of moving one's hands through the air provides an intuitive method to control continuous data, whilst the motion inherent in using the system provides something visually engaging for the audience.
One obvious limitation here is that the user will have to remain within a reasonable distance of their camera; but tests have shown that effective tracking is achievable up to 4-5m from the camera (depending on the features to be tracked/ recognised), which should be a significant amount of space for most performance purposes.
We are interested in examining other applications for our prototype.Current ideas include using the system as a games controller or as a tool for controlling graphics software.As such, ongoing investigation continues into the use of the system in a broader context to provide users with remote control over a range of software, outside the scope of purely musical interfaces.

Figure 1 :
Figure 1: An overview of the system architecture

Figure 3 :
Figure 3: Prototype glove with wireless transmitter circuit

Figure 5 :
Figure 5: Playing with the gloves

Table 1 :
Momentary control assignment table