Interactive Multimedia Rocks for Geology

This paper presents the Ruskin Rocks project which is developing a rock instrument, making use of interactive multimedia to augment the musical characteristics of stones. Several historical rock instruments, which inspired the building of this instrument – iRock – are discussed. Fundamental vibration theory is applied to understand the musical nature of the rocks, and empirical data presented to support the decisions made for the instrument design. Several new concepts based on user-centred design have been introduced to study the effects of interfaces and interactivities. The paper also outlines the iRock system architecture and discusses augmentations to the instrument and signal processing techniques.


INTRODUCTION
The overall aim of the project is to use the musical characteristics of rocks (mineral stones) to introduce knowledge about sound, its relationship to the natural materials and geology of the region.The project is conducted in collaboration with aggregate quarries in Cumbria, and local schools around the region, to identify musical rocks from the rich geological composition of the fells and dales.
This paper focuses on the interactive rock instrument that is currently being built, as a vehicle to introduce children and the general public to the musical potential of rocks from the region.While the physical appearance resembles that of a traditional lithophone, the instrument design integrates interactive multimedia technologies and digital signal manipulations to provide real-time multimodal feedback, thus enhancing user interaction with the rock instrument and at the same time offer related information on geology and sound.This augmented lithophone -interactive rock instrument (iRock) -offers a set of multimedia functionalities including: virtual visualisation into the mineral rock structure with associated geological information (e.g.including visual representations from Scanning Electron Microscopy (SEM)), and digital signal processing, both for audio effects and to provide the performer real-time visual feedback.It shall be built using a variety of rocks from the region to illustrate igneous, sedimentary and metamorphic rocks.
In this project, we plan to produce two instruments: (i) a larger instrument (up to four octaves) for performance with optional sensor/electronics attachments; and (ii) a smaller setup (with one example rock type from the region per key) for interactive multimedia edutainment.The multimedia attachment from the smaller setup can be transferred and used (with additional sensors) on the main instrument for augmented multimedia performance.The interfaces adapt traditional percussive instrument design while experimenting with new concepts to further evolve musical possibilities.
In addition to the new musical interfaces for edutainment, other explorations include composition, performance technique and performance interface studies based on the iRock instrument.

STONE INSTRUMENTS AND MUSIC INTERFACE
There are numerous examples of stone instruments from all corners of the world, dating from ancient times to the modern day.One of the oldest examples is the Nubian Rock Gongs from Sudan (BBC documentary 'Lost Kingdoms of Africa').In the Far East many instances of suspended chimes exist, with one or many keys, which are struck with wooden mallets or padded sticks.For example, the 'Qing' from China (see Figure 1, top) has a non-symmetrical chevron shaped key with a curved lower edge.Similar instruments are found in Korea (p'yŏn'gyŏng) and Japan (Bien Chung).In addition to the instruments from Africa and Asia, examples are found in the Americas and Europe.These lithophones are typically organised in pitch from low (left) to high (right).The keys rest on straw/rope and are not fixed on to the body (typically a soundbox) in order to allow maximum freedom for vibration.There have been several unfortunate occasions when keys have been stolen, resulting in the addition of screw fixtures that are not at nodal point.This has resulted in a strong dampening to the vibration.
For this project, the instruments have to be fully accessible for public visitors and at the same time a playable musical instrument and hence we are particularly interested in different approaches to hold the rock with minimal damping factors.
The iRock instrument is based on traditional xylophone/lithophone design in order to minimise the learning time for a percussionist to play the new instrument.

INSTRUMENT
The iRock instrument will continue the long tradition of making music from stone, with the inclusion of interactive multimedia to heighten the experience both for performer and audience.
The completed instrument will be installed in Brantwood (http://www.brantwood.org.uk), the former home of John Ruskin.This will provide an educational opportunity to develop greater awareness of Cumbria's natural environment, and the importance of local aggregate quarries.
Aggregates remain one of the major industries around the Lake District, following a long tradition of mining and quarrying in an area that has played a pivotal role in the development of modern geological understanding.We intend to develop and celebrate the link between industry and the landscape, exploring interactive multimedia for edutainment using music and sound to disseminate knowledge and understanding of the local geology.

Design considerations
Classical tuned percussion instruments such as the xylophone have been stable in terms of design for a long period of time.Meanwhile, research on ergonomics, gesture interface, and human performance has progressed greatly, and been the focus of much attention the information age.
The iRock instrument's design will mirror that of a traditional xylophone in that it will adopt an equally tempered chromatic tuning, have lithophone bars comparable in width to traditional xylophone bars, and be arranged in a piano keyboard style layout.The intention is to give the instrument a degree of familiarity.This is the point at which physical similarity to the classical xylophone ends.
Typically, playing a musical instrument involves physical activity, during which the performer expends energy.We wish to optimise the design of the iRock instrument to minimise the effort required to perform.To achieve this we must understand how the player interacts physically with the instrument.Factors such as the player's height, and range of hand movement, are of critical importance in designing an ergonomically efficient interface.
When playing a classical xylophone around the mid-range, typically the performer's upper arms will be almost vertical, with the distance to the striking point on the instrument determined by the combined length of lower arm, hand and beater.In addition the shaft of the beater will rotate in a plane almost coincident with the longitudinal axis of the xylophone bar when striking the note.If the player remains in a fixed position, and reaches for notes in the upper or lower range, the arm must be extended and there will be a significant angle between the rotational plane of the beater and bar's longitudinal axis when striking the note.This has the effect of reducing the accuracy of the strike point and increasing the chance of error.Adopting a curved instrument layout significantly reduces both the required arm extension, and angle at which the bar is struck.The curvature should be elliptical, with minor axis representing distance from elbow to index finger knuckle, and major axis the distance from shoulder to index finger knuckle with arm extended.This elliptical layout significantly improves the ergonomics of the instrument, as compared to a traditional linear arrangement of bars.

Figure 3: Arc shaped layout tracks loci of hand movements
A second major difference to the classical xylophone design is that the playing surface of the instrument will be slightly inclined away from the player, as depicted in Figure 4 (left).We believe that this will provide a more natural playing position and allow an audience to see more of the instrument playing surface during a performance.For the inclined surface to provide improved ergonomics, the height of the playing surface will be critical -too low will cause the player to hunch over the instrument, too high will require the shoulders and forearms to be raised, with the wrist bent into an unnatural playing position to compensate.If the keys are lower than the elbow the inclination should be away from the player (Figure 4, left).Conversely, when the keys are higher than the elbow, the inclination should be toward the player (Figure 4,right).It is also proposed that some height adjustment of the playing surface should be possible to allow adjustment to the player's height.

Rock types
As part of the design process, a variety of rock types sourced from local aggregate quarries were tested.These included: Hornfel, Ignimbrite, Limestone Carbonate, Metamorphosed Sediment, Metamorphic Slate, Metamorphic Volcanic Ash, Metamorphose Sediment, Volcanic Tuff.
Rock samples were grouped according to their source and type, and tested for their acoustic characteristics, such as frequencies produced when struck and amplitude decay rates.The results obtained during this experimental phase of the project are interpreted using a standard acoustical model, which describes the transverse vibrations of a bar.In addition, a scanning electron microscope was used to examine the rock's geological structure.The conclusions from the experiments conducted are being used to determine the rock types and physical dimensions of the individual iRock bars.
Different methods of supporting the bar are also investigated, as the support mechanism is one of the primary sources of damping for a vibrating bar.

PHYSICAL MODEL OF VIBRATING BAR
The bars of the iRock instrument will be rectangular, have a uniform cross-sectional area, and be free to move at both ends.Our aim is to determine the optimal dimensions and rock types to use within the iRock instrument.Firstly, we must consider the various types of vibration that take place within the bar.When striking the playing surface of a lithophone bar, the resultant displacement is in the vertical direction, and so transverse vibrations are excited (see Figure 5).
Examining the forces and bending moments in a small segment of the bar, then integrating over the bar's length, leads to a general equation of motion for a transverse bar vibrations (Fletcher 1990, Kinsler 1982): In the above equation, the x-axis lies in the horizontal plane, parallel the longitudinal axis of the bar, the variable y denotes the transverse (vertical) displacements, and t represents time.The significance of the constants κ and c will be discussed later in this section.
The equation of motion may be solved by separation of variables, and boundary conditions applied at the free ends of the bar.This leads to a simple relationship that predicts the allowed frequencies f of the transverse modes of vibration for a bar of length L: The first point to note from the above equation is that the frequency of each mode of vibration is inversely proportional to the square of the bar length.Secondly, we see that the overtones are not harmonics, as they are not integer multiples of the fundamental frequency.The above equation also allows us to predict where the nodal positions for each mode lie along the length of the bar.For the fundamental frequency, the nodal positions at which the supports should be located lie at 22.4% and 77.6% of the bars length.Nodal positions may be verified experimentally by sprinkling salt onto the bar and exciting vibrations of a particular mode, thus causing the salt to settle to the nodal positions (La Favre, 2007).
The other constant κ depends on the bar's crosssection.For a rectangular cross-section of thickness T, the constant is given by: 12 / T = κ A complete derivation of these formulae can be found in most acoustics textbooks.Here only the main points are summarised, which will allow us to characterise the rock samples tested and make suitable choices of rock type and dimension for the iRock instrument.
In addition to the transverse modes of vibration, torsional and longitudinal modes of vibration may also be excited, depending upon how the bar is struck (Rossing, 1990).Although some experimental work is being carried out to identify these modes, for now we will restrict our discussion to the transverse vibrational modes.

Frequency analysis
If the fundamental frequency of a sample bar may be measured experimentally and the rock density calculated by measuring the mass and volume of the bar, we may use this empirical data to obtain a value for the constant c.By this method we can deduce the Young's modulus of each rock type.
The constant κ is evaluated by simply measuring the thickness of the bar's cross-section.Once the constants have been determined, using a number of samples for each rock type, we may then predict the frequencies that will be produced for any other rock length and thickness with a reasonable degree of accuracy.
With the bar supported at the 1 st nodal position, sound recordings were made of each rock sample, with the bar being struck in a number of different positions along the playing surface, and with different amounts of force.Choice of beater was found to significantly influence the transient frequencies produced during the initial attack of the bar strike (Bork, 1990).Of those tested, the beater that produced the best sound with the rock samples was a cord wrapped mallet with birch shaft, from the Dame Evelyn Glennie (EG5) Autograph Series.From the sound recordings, a frequency analysis was obtained by performing an FFT on each bar strike.A typical frequency spectra, in which the fundamental and first two overtones are clearly visible, is shown in Figure 6.Our results indicate that the ratio between fundamental and overtone frequencies is indeed accurately predicted by the theory.Some additional peaks were present in the frequency spectra which may correspond to torsional or longitudinal modes of vibration.Values obtained for Young's modulus remain reasonably consistent within the groupings made according to rock source and type.

Temporal analysis -decay rates
In accounts given of musical rocks, we often hear of the rocks having a particularly resonant or pure sound.What does this mean in terms of our physical model for the vibrating bar? Figure 7  below has been produced from the same sound file used for the frequency spectrum in Figure 6.It shows the amplitude of vibration as it decays over time.The curvature of the amplitude envelope decay curve appears to be approximately exponential.If this is the case, then by converting to a logarithmic decibel scale, the dB level should decay linearly in time.
By examining the frequency spectrum at a series of snapshots in time after the bar is struck, a graph can be plotted showing the dB level decaying over time for each individual frequency component in the spectra.Figure 8 shows the results obtained using the same sound file as for previous examples.The graph indicates that the dB decays are indeed linear to a good approximation.It can also be seen that the fundamental frequency f 0 decays much more slowly than any of the overtones.This is expected, as the supports are not positioned at the nodal positions, resulting in heavier damping of overtones.The straight line equation is displayed for each linear dB decay, with the gradient giving a direct measure of decay rate, and the intercept an indication of the degree to which transverse modes are excited as a result of the bar strike.Strike position, force, and mallet type all influence the degree to which a particular mode of vibration will be excited -for example striking the bar at an anti-node will maximise excitation for that mode.The graph indicates that for this particular strike the initial excitation of the first overtone exceeded that of the fundamental, but within a few milliseconds the faster decay rate takes over -the fundamental always wins!

Effect of rock thickness
So far the analysis of a single rock sample (HF1.8) has been discussed.The series of rocks HF1.1 -HF1.9 are of the same rock source and type, with identical length and width, but with varying thicknesses ranging from 3.5mm to 35mm.By carrying out a frequency analysis on each sample, it was verified that fundamental frequency is directly proportional to rock thickness as predicted by the theory.For this set of samples, our results indicate that a 1mm increase in thickness causes a 125Hz increase in fundamental frequency.This illustrates the critical importance of bar thickness on the overall tuning of the instrument.While for a known rock type and thickness it is possible to predict the length required to tune to a particular pitch, any variation in the thickness will rapidly throw the tuning out.Analysis of the decay rates of samples HF1.1 -1.9 highlights another important factor in the choice of rocks for the instrument.The results indicate that the decay rate has a linear dependence on the rock thickness, which in turn is proportional to the fundamental frequency.This can be accounted for by considering how the vibrations within the rock are damped.

A balanced approach
In order to meet the instrument's educational requirements and impart knowledge about the rich geological diversity of the area, the instrument must utilise a variety of different rock types sourced locally.This in itself poses a challenge -how can we incorporate significant variations in timbre and decay rates into an instrument, while maintaining a musically pleasing effect when played?
Using the theory for vibration of bars and experimentally collected empirical data, it is now possible to predict with a degree of certainty how the different rock types will behave.Using this knowledge, an assessment can be made of how these factors will influence the physical design of the instrument.
For the size of the instrument to remain practical, the maximum bar length will be limited to one metre.To achieve the maximum possible frequency range within this constraint, it is apparent that the rock type having the lowest value for constant c should be used for the lowest notes on the instrument, and the rock type with highest value of c should be used for the upper notes (recall that this constant depends on the ratio of Young's modulus to rock density).To extend the low frequency range as much as possible, it will also be necessary to make the bar as thin as possible to achieve the lowest pitch possible.Clearly there is a trade-off between extending the low frequency range and maintaining sufficient mechanical strength in the bars.The iRock instrument is to be housed in a public space and will have to be suitably robust -this requirement must be balanced against the wish maximise the lower pitch range within the final design.To rephrase, how thin can we make the bars without them breaking when the instrument is played?
There is also a potential downside to using low c materials for low frequencies, and high c materials for high frequencies.By adopting this approach, rock types that sustain vibrations less well (due to their material properties) will populate higher frequency regions where damping of vibration is already more rapid due to the frequency of oscillation.The converse will true for the lower frequency regions.The resultant effect of choosing a rock layout designed to extend the available frequency range is a magnification of decay time differences.A different strategy may be adopted where rocks are positioned to minimise differences in decay times, but this will raise the pitch of the lowest available note significantly.Again there is trade-off, this time between pitch range and decay time inequalities.
As is often the case during the design process, a balance must be struck between competing factors, so that best possible solution may be implemented.Firstly, limits will need to be set on the minimum thickness for each type of rock.Upper and lower limits on the pitch range and decay rate can then be calculated for each rock type, taking account of constraints on bar length.The pitch range of the iRock instrument, final locations of rock types, and physical dimensions of each stone can then be finalised within the design.

Bar supports at nodal positions
The iRock instrument will be a public installation, making it imperative that rocks are fixed safely and securely.The supports used to achieve this must still allow for natural vibrations to ring within the bar.Traditionally, xylophone bars are suspended on a cord threaded through holes drilled horizontally through the bar at each of the two f 0 nodal planes.The cord allows the keys to be secure, while allowing enough freedom of movement for vibration to take place (Rossing, 2001).Rock is inherently more brittle than wood, and our bars will be of uniform cross-section (no undercuts between nodal positions) so it is likely that drilling in this manner will weaken the bars, leaving them prone to breakage.A number of potential support mechanisms were tested, and it was chosen to use a steel bracket support at each nodal position, with a recessed pin retained with loctite 638 supporting the bar.Some recordings were made using rocks supported in this manner, and the decay rates compared with those obtained before drilling and fitting of brackets.Some additional attenuation of the fundamental frequency was present, although the fundamental decay rate was still judged to be acceptable.The most significant effect was the damping of higher overtones (in particular the first overtone) due to the pin's position not corresponding to overtone nodal positions.One of the bracket's main advantages is that it will allow a stone to be removed and replaced if damage occurs.A slot is machined into the base, which will allow some adjustment for variations in nodal position and bar length.

AUGMENTATION
To provide additional musical expressivities, each bar of the instrument will be fitted with a piezoelectric crystal to capture the vibrations from the rocks.The signal from each piezoelectric crystal will be routed to an audio interface, which will translate analogue voltages into a real-time digital audio signal that can be manipulated by software.It is possible that some additional buffering/preamplification may be required before connection to the audio interface, both to boost signal strength and to match the high impedance of the piezoelectric crystal to the input impedance of the audio interface.The resulting digital audio signal is transmitted to a computer where it is used within the graphical programming environment provided by Max/MSP/Jitter.Audio and video outputs from the software present interactive multimedia feedback as the iRock instrument is played.The overall system architecture is summarised in Figure 13.

Digitally enhanced natural rock sounds
One of the project's major challenges is to produce a musically pleasing sound from a collection of rocks whose vibrations may ring for a few seconds, as in the case of some limestone samples, or give a short percussive attack for a fraction of a second, as found with some slate samples.It is therefore desirable to be able to augment the rocks' natural acoustic qualities to lengthen the sustain time.A number of possible methods are being considered to attaining a longer sustain is a form of additive synthesis in which sinusoidal components corresponding to the modal frequencies are added to the sound to extend its duration.The necessary frequency components may be known in advance for each bar, and the exponential decay envelope of each mode controlled by dynamically adjusting the level of the software generated sinusoidal components.Using this technique, the attack and transient behaviour that gives the rock its characteristic sound is preserved when the bar is struck, but transverse modal decays are at a rate determined by the player.Another option is to perform an FFT to determine the amplitude and frequency components, and implement some frequency domain processing to control decays before transforming back to the time domain (Kim-Boyle, 2004;Puckette, 2005;Settel, 1998).

Sample triggering
The sound produced by the iRock instrument needs not be that of a lithophone, as the voltage response from the piezoelectric crystal may be used to trigger sample playback or generate a synthesised sound (Puckette, 1998).The range of possible sounds is limited only by our imagination and computing power available to process the signals.Crazy sound effects and silly samples may be incorporated to add an element of fun to the project, giving the instrument instant appeal for children.Individual bars may be mapped to drum samples allowing multiple players to perform as a percussion ensemble.When used for triggering, the instrument becomes a flexible control surfacea physical interface suitable for use in a diverse array of musical applications.

Effects processing
Whether working with natural rock sounds, or using the instrument as a triggering control surface, the player will have the option to apply effects processing (e.g.reverberation, echo, delay etc.) to the outgoing signal.These effects may be software generated within Max/MSP or applied using external hardware.The addition of effects processing will expand further the musical potential of the iRock instrument.

Multimedia interactive feedback
The interactive multimedia element of the system will deliver audio and visual feedback to the player.Some elements that will contribute to the audio feedback have already been discussed.In addition, a visual display will be provided to give visualisations of the sound emanating from the iRock instrument, and information relating to the individual rock types to enhance the educational experience.A proposed screen layout is shown in Figure 14.
Real-time sound visualisations are located on the left, with the upper quadrant giving a waveform display of each bar vibrations (this may also be used to look at waveforms of the augmented sound to visualise the differences), and the lower quadrant giving a moving 3D plot of frequency, dB level and time (Charles, 2008).On the upper right hand side of the display, images taken using a scanning electron microscope will show the surface structure of each stone when struck with the mallet.Additional information on the stone's origin, geological type, and formation processes will be presented below.

CONCLUSION AND NEXT STEPS
In this paper we presented a brief overview of the Ruskin Rocks project with some background and design considerations of the interactive rock instrument.With reference to a standard model for the vibration of uniform bars, we have highlighted how differences in the rock types, such as the physical dimensions, density and Young's modulus, affect the rock's physical behaviour, and influence the overall instrument design.Main components of the iRock system's architecture have been summarised, and we have explored how these elements will combine to provide a musical interface that the public may interact with, and learn from.
The instrument and its exhibition space are scheduled to be launched with a demonstration and workshop with Dame Evelyn Glennie in August 2010.With the augmented instrument, we aim to provide a hands-on multimedia experience that is attractive to everyone and enable all users to explore a wide range of musical possibilities.

Figure 1 :
Figure 1: Musical stone instrumentsThe history of lithophones in England dates back to the 18 th century, with a chance discovery of musical stones by Peter Crosthwaite, at Skiddaw in 1785.Soon after, stonemason Joseph Richardson constructed a five and a half octave lithophone from Skiddaw stone, over a period of thirteen years.The Skiddaw rocks are of a geological type known as hornfels, a rock produced by contact metamorphism.John Ruskin commissioned a lithophone which is now housed at the Ruskin Museum at Coniston.

Figure 2 :
Figure 2: A picture of the Richardson Set (Rock, Bell and Steel Band)

Figure 4 :
Figure 4: Two inclined playing surface orientations

Figure 5 :
Figure 5: Transverse modes and nodal pointsThe constant c depends on the elastic (Young's) modulus, Y and density ρ of the material in the bar, and is given by:

Figure 9 :
Figure 9: Amplitude decay per cycle of vibration For an exponentially decaying vibration, the amplitude of successive maxima is related by δ γτ = =

Figure 10 :
Figure 10: A prototype rock key with bracket

Figure 12 :
Figure 12: Underside of rock key fitted with a piezo sensor

Figure 13 :
Figure 13: iRock system architecture and data flow

Figure 14 :
Figure 14: Interactive Graphical User Interface