Configuring a Haptic Interface for Music Performance

Our daily interactions and perceptions involve multiple sensory multimodalities; most exchanges are inherently multimodal, playing an important part in our decoding and understanding of environments. Events can often engage multiple senses. Furthermore, musical experiences can be highly multisensory, with obvious auditory stimulation, the visual elements of a live performance, and physical stimulation of the body. In this paper, we will propose a means of incorporating an additional somatic channel of communication into live performances and compositional practice to further augment the physical nature of live performance. This work explores the integration of augmented vibratory, or haptic stimulation for audiences in live performance. The vibration interface is presented as an expressive and creative live performance-based tool for composers. Vibrations, or haptics, are implemented as an additional instrumental line, alongside auditory musical gestures, to expand the composer’s palette of expressions through augmented somatic engagement. This paper will describe the background, and design and development of a haptic interface for the purpose of audio-haptic listening-feeling. The focus of this paper is to describe a study into motor latency for informing multimedia simultaneity.


INTRODUCTION
In a previous EVA paper (Armitage & Ng 2013), we proposed a multi-actuator interface for use in compositional and performance practice. The paper focussed on design requirements and preliminary designs of the haptic interface and how it fitted within the larger remit of the Interdisciplinary Centre for Scientific Research in Music (ICSRiM) research concepts and wider work in multimodality in music practice.
The use of haptics in music performance and composition practice is a gradually expanding field, particularly with regard to the design of music controllers and interfaces and digital musical instruments (Birnbaum 2007, Verplank, Gurevich & Matthews 2002; other work is exploring its use in artistic practiceto guide improvisation (Hayes & Michalakos 2012), for installation (Gunther 2007) and to aid synchronisation and ensemble communication (Armitage et al. 2012). The potential for haptics in such applications is an exciting prospect, providing a means of communicating information via non-audible channels. Much research is required to develop protocols and standards in the design of such systems, currently an increased modularity and open source approach is the most effective way of ensuring the accessibility of haptic systems for creative applications.
Overall the aim of this project is to develop approaches, frameworks and devices for creating integrated auditory-haptic experiences. An appreciation of system latency and jitter is required to firstly, synchronise fixed media haptic and auditory stimuli by adding latency to the audio system, and secondly to allow an appreciation of whether the haptic interface could function in realtime applications. These results provide a test-bed for further investigations into the human perception of haptic-latency in time-based applications. This paper considers the necessary requirements and configuration of a haptic device for music listening/feeling purposes. As this paper focuses on exploring the mechanics of such devices, particularly in evaluating hardware, we first consider the perceptual thresholds of our skin including its 'bandwidth' and 'sensitivity'. This informs the development of a haptic interface in relation to requirements of motors (type and vibrational amplitude). In particular, this paper discusses the results of a quantitative evaluation of three motors proposed for the project and the mechanics for driving them.

Vibration thresholds of the skin
To ensure characteristics of vibration are not only perceptible, but effective and accurate, we need to ensure that they work within detectable and sensitive 'ranges; of the skin. Vibration parameters primarily explored through vibrotactile perception research are: frequency, timing, amplitude and location.
Akin to the auditory system, tactile sensation is achieved when vibrations surpass various thresholdsthis is dependent on several factors: area of skin, hair density, frequency, duration, mass and waveform.
In terms of the skins vibration sensitivity range, the upper limit is often considered to be 1 KHz, although Wyse et al. (2012) report that the use of more complex vibrotactile signals applied to the hand can be felt up to 4000 Hz at when produced at increased amplitude. Interestingly, this study relates speech information, normally within the range of 300 to 3000 Hz, to the similar bandwidth of the skin. Generally, it is agreed that the greatest sensitivity is felt at around the range of 200 -250Hz.
When coding discrete information through the tactile channel it is generally considered that only 4 levels of amplitude are used between the pain and comfort threshold and 9 levels of frequency with a different of around 20% (van Erp 2002). Similarly to the frequency and amplitude relationship in auditory sensation, haptic signals vary with relative perceived amplitudes in relation to frequency. As this work is interested in expressive haptic characteristics as opposed to information coding, the amplitude will be considered continuous. The minimum acceleration detectable as a tactile sensation is generally considered to be 0.04 G (Precision Microdrives 2015).

Perceived multimodal simultaneity
This work approaches an exploratory multimedia, presented in a continuous form with two primary data streams; one consisting of data to render haptic actuation and one to produce sound. As the audio-haptic installation is presented in a temporal format, it is considered that the data streams should be in synchronisation. As this application considers passive listening-feeling activity, the primary concern is that auditory and haptic stimuli achieve the desired effect, and also a more natural and intuitive experience. Multimodal simultaneity can be difficult to achieve, affected by processing time and hardware limitations. As the haptic composition is played for example, data is sent to the audio and haptic renderers and particularly in regard to the haptic element there is a delay time whilst threshold current is reached before the user feels it.
A perceived synchronisation threshold could be described as the maximum time at which two stimuli appear in synchronisation. Researchers have employed various psychophysical approaches to measuring this threshold, with variable results (Levitin et al. 2000). Perception of skew, or latency, varies between individuals, whilst also being affected by sensory domain, the distance between the stimulus area and the brain, and the thought process required.
In auditory-tactile research it has been previously been reported by Altinsoy (2003), that a study sending tactile impulses to the tip of a participant's index finger alongside auditory white noise. Delays of auditory noise were presented randomly before and after the tactile stimulus between ±150ms, and participants were asked to report whether the stimuli were synchronous or asynchronous. It was found that synchronisation was reported when the audio appeared 25 ms ahead of the tactile impulse, and within 50 ms prior to it. Levitin et al. found that in active touch stimuli between -25 and +42 ms are found to be asynchronous. Altinsoy (2012) points out that although these thresholds are given, many experiments report participants with significantly higher sensitivity of up to 10 ms. Of particular relevance to this work they that "musicians have smaller thresholds than the general population, possibly because of their training."

Overview
A haptic interface has been designed, consisting of 16 eccentric rotating mass (ERM) motors that are aligned in a grid array. By switching between motors on the grid different tactile effects can be imparted on the body, exploiting illusions of the sense of touch (Lederman & Jones 2011). This has been integrated into a portable seat cover that can be positioned on the back of chairs.

Motor types and considerations
Eccentric rotating mass motors are DC motors that have been most commonly used commercial devices and vibration alerting systems. Their construction consists of a shaft with a nonsymmetrical, or offset mass attached to it. As it rotates, the mass displaces the motor causing it to vibrate on 2 axes. To achieve basic control of the motor, a constant voltage from a DC source can be applied to the motor terminals. Linear resonant actuators (LRA) are driven similarly to a speaker, but differ that instead of driving a cone to produce sound they drive a mass to produce vibration on 1 axis. LRAs could be considered advantageous over ERM motors in terms of their reliability and longevity.
We have investigated three different motors for this project (see Table 1), all from Precision Microdrives (http://www.precisionmicrodrives.com/), a small ERM coin-type motor (310-101), a larger ERM motor (308-102) and an LRA (C10-100). Important considerations when designing haptic devices are the vibration amplitude and lag/rise/stop time. Particularly for musical, or other applications that require synchronisation a particular requirement for an understanding of the latency inherently caused by moving parts is fundamental.
The lag time is the time taken between the voltage being sent from the controller and the motor achieving vibration amplitudes of ±0.04 G, or 0.08G peak-to-peak, are the minimum force detectable by naked skin. The rise time is shown to be the time between the voltage being applied and vibration amplitude reaching 50% of its maximum. Whilst breaking time indicated the time taken for vibration amplitude to drop from its maximum to below 0.08G when voltage is removed.
The vibration amplitudes of all three motors are tested using a test sled mass of 100g, this is used to normalise the motors and reduce chances of the measurement accelerometer clipping. For the purposes of this study, larger vibration amplitudes are deemed to be preferable when considering the mass of and force exerted by the back on a chair. Haptic motors are generally driven by additional circuitry; most simply a single transistors to control power when motors are used in conjunction with microcontrollers. H-bridges allow bi-directional motor control, and purpose built haptic drivers. Access to dedicated haptic motor drivers can be problematic, as the commercially available ICs only come in surface mount packages, presenting a difficulty in prototyping.
A previous design using the 310-101 drivers used a TLC5940 LED driver package with controllable PWM. This initial prototype proved effective for driving the 310-101 motors, but its maximum current output of 120 mA wasn't great enough for 308-102 motors, with a typical rated operating current of 145mA. Furthermore, it could not be interfaced with the LRA motor that is driven by an AC signal. Thus, the most recent prototype employs the Texas Instruments DRV2605 driver was selected for two main reasons; firstly, it interfaces with both ERM and LRA motor types, at a range of operating currents and voltages, which is advantageous for testing and comparison purposes; secondly, it is available as an economical breakout board from Adafruit (https://www.adafruit.com).

CONFIGURING MOTOR SETUP
The initial stages of configuring the motor grid setup were to establish what motor would be most appropriate in the context of this setup. As previously mentioned, several factors influence this from a technical perspective. Alongside the aforementioned requirements of rated vibrational amplitude and response, low noise and ease of integration are important considerations. Due to this being a musical application minimal noise is beneficial as it reduces the overall auditory distraction effect of the motor noise. Another important consideration that is connected to the response of the motor (lag, rise and brake times) is the consistency of the frequency of successive pulses, or jitter, when considering music, a time based medium.
This study measures the latency apparent in the motor hardware to better understand how it can be compensated for in fixed media and determine the possibility for it to be used in real-time applications. Although there will be latency in other areas of the system, it is first necessary to consider it in the hardware element of motor activation.

Motor latency test setup
In this test each of the three motors were measured in an experimental setup influenced by that described in the Precision Microdrives (2014) application notes. The motors were secured to a 102g-test sled (see Figure 1). An MMA7361 accelerometer was used to measure the motors vibration on one of their vibration axes, the sensitivity of the accelerometer was set to ±6g. An Arduino Uno was used to address the DRV2605 driver, triggering Immersion's pre-programmed effect 'Strong Buzz 100%' (ID #14) on each motor and using different libraries dependent on motor type as indicated in the datasheet.

Motor latency results
Approximately 630 motor onsets were recorded for each individual motor with the data being collected at a rate of 400Hz. Latency was measured as being the time between when the voltage was initially sent from the microcontroller to the first peak (+ve or -ve) in the accelerometer data. The average time between peaks was computed alongside the standard deviation. The average latency for the 308-102 was 24.76 ms, the 310-101 averaged 137.75 ms and C10-100 28.20 ms. These results, together with outliers, are shown in Figure 2.

Discussion
When considering these results it is clear that the 308-102 motors offer both the lowest latency and considerably less jitter than the other motors. At this time, they are considered most appropriate for use in this project. Whilst providing the greatest vibration amplitude, they will also add the least amount of latency to the system. Though, the use of this motor type does present challenges to the design as they contain external moving parts that require a more complex enclosure. While it has not yet been experimentally measured, it is also apparent that the 308-102 motors will add significantly greater noise to the haptic array compared to the near-silent C10-100.
In relation to the values suggested earlier it is debatable as to whether the system would work in effecting synchronisation real time applications. Although some feelers sound not notice an auditory-tactile delay of 24.76 ms, others would be sensitive to values of less than 10 ms. Due to the passive act required, if using an online or generative system the latency can be added. For use in live instrumental performance, this would make more of an impact. The results of previous studies into the perception of auditory-tactile simultaneity are not conclusive, and in the case of this study, require further investigation.

CONCLUSIONS
This paper has described aspects of the development of a new haptic interface for music listening purposes. It has considered thresholds of the skin in terms of motor selection and provides as quantitative evaluation of motors tested for use in the system. The interface is currently being assessed in terms of its human factors, using perceptual experiments to configure and define levels of vibrational amplitude and the haptic-audio relationship. As part of Digital Futures, the interface has been used in an installation exploring amplitude relationships between sound and the somatic, 'Unheard' Sounds.