Approaches to Visualising the Spatial Position of ‘Sound-objects’

In this paper we present the rationale and design for two systems (developed by the Integra Lab research group at Birmingham Conservatoire) implementing a common approach to interactive visualisation of the spatial position of ‘sound-objects’. The first system forms part of the AHRC-funded project ‘Transforming Transformation: 3D Models for Interactive Sound Design’, which entails the development of a new interaction model for audio processing whereby sound can be manipulated through grasp as if it were an invisible 3D object. The second system concerns the spatial manipulation of ‘beatboxer’ vocal sound using handheld mobile devices through already-learned physical movement. In both cases a means to visualise the spatial position of multiple sound sources within a 3D ‘stereo image’ is central to the system design, so a common model for this task was therefore developed. This paper describes the ways in which sound and spatial information are implemented to meet the practical demands of these systems, whilst relating this to the wider context of extant, and potential future methods for spatial audio visualisation.

Content

Author and article information

Contributors

Jamie Bullock

Balandino Di Donato

Conference

Publication date: July 2016

Publication date (Print): July 2016

Pages: 15-22

Affiliations

[1 ] Birmingham Conservatoire

Birmingham, UK

Article

DOI: 10.14236/ewic/EVA2016.4

SO-VID: 31f663e0-9518-4348-848e-5fc027faec2e

License:

This work is licensed under a Creative Commons Attribution 4.0 Unported License. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/

Conference name: Electronic Visualisation and the Arts

Conference acronym: EVA

Conference number:

Conference location: London, UK

Conference date: 12 - 14 July 2016

Conference sponsor: Electronic Workshops in Computing (eWiC)

Conference theme: Electronic Visualisation and the Arts

History

Product

1477-9358 BCS Learning & Development

Self URI (article page): https://www.scienceopen.com/hosted-document?doi=10.14236/ewic/EVA2016.4

Self URI (journal page): https://ewic.bcs.org/

REFERENCES

Z. AriefI. A. SulistijonoR. A. Ardiansyah 2015 Comparison of five time series EMG features extractions using Myo Armband 2015 International Electronics Symposium (IES), Surabaya, Indonesia 29-30 September 2015 11 14 IEEE
W. BaldonadoQ. MichelleA. WoodruffA. Kuchinsky 2000 Guidelines for using multiple views in information visualization Working Conference on Advanced Visual Interfaces, 110 119 ACM
T. BaudelM. Beaudouin-Lafon 1993 Charade: remote control of objects using free-hand gestures Communications of the ACM 36 7 28 35
R.J.V. BertinA. Berthoz 2004 Visuo-vestibular interaction in the reconstruction of travelled trajectories Experimental Brain Research 154 1 11 21
F. BioccaY. InoueA. LeeH. PolinskyA. Tang 2002 Visual cues and virtual touch: Role of visual stimuli and intersensory integration in cross-modal haptic illusions and the sense of presence Presence 2002 Porto, Portugal October 9-11 2002 410 428
P. BoulinguezS. SavazziC. A. Marzi 2009 Visual trajectory perception in humans: Is it lateralized? Clues from online rTMS of the middle-temporal complex (MT/V5) Behavioral brain research 197 2 481 486
A.S. Bregman 1994 Auditory Scene Analysis: The Perceptual Organization of Sound Bradford Books MIT Press
M. W. Calkins 1893 A Statistical Study of Pseudo-Chromesthesia and of Mental-Forms The American Journal of Psychology 5 4 439 464
O. Cassidy 2015 Hearing in Color; Seeing in Sound: Chromesthesia and Its Influences on Audio-Visual Work
D. W. ChandlerD. W. Grantham 1992 Minimum audible movement angle in the horizontal plane as a function of stimulus frequency and bandwidth, source azimuth, and velocity The Journal of the Acoustical Society of America 91 3 1624 1636
EA Sports ext-link-type="uri" xlink:href="http://nautil.us/issue/26/color/what-color-is-this-song">http://nautil.us/issue/26/color/what-color-is-this-song 15 March 2016
M. Eimer 2004 Multisensory integration: how visual experience shapes spatial perception Current biology 14 3 R115 R117
F. FrassinettiN. BologniniE. Làdavas 2002 Enhancement of visual perception by crossmodal visuo-auditory interaction Experimental Brain Research 147 3 332 343
K. GiannakisM. Smith 2000 Auditory-visual associations for music compositional processes: A survey International Computer Music Conference ICMC2000, Berlin, Germany
A.I. GollerL.J. OttenJ. Ward 2009 Seeing Sounds and Hearing Colors: An Event-related Potential Study of Auditory-Visual Synesthesia Journal of Cognitive Neuroscience 21 10 1869 1881
F. HaqueM. NancelD. Vogel 2015 Myopoint: Pointing and clicking using forearm mounted electromyography and inertial motion sensors 33rd Annual aCm Conference on Human Factors in Computing Systems, 3653 3656 ACM
J. JankowskiH. Martin 2013 A Survey of Interaction Techniques for Interactive 3D Environments Eurographics2013, Girona, Spain 65 93 Eurographics Association
M. Kawato 1999 Internal models for motor control and trajectory planning Current opinion in neurobiology 9 6 718 727
A. Lécuyer 2009 Simulating Haptic Feedback Using Vision: A Survey of Research and Applications of Pseudo-Haptic Feedback Presence 18 1 39 53
S. Lewis 2010 Seeing Sound: Hans Jenny and the Cymatic Atlas, University of Pittsburgh Pennsylvania, USA
L. E. Marks 1975 On Colored-Hearing Synesthesia: Cross-Modal Translations of Sensory Dimensions Psychological Bulletin 82 3 303 31
A. Mazumdar 2013 Principles and Techniques of Schlieren Imaging Systems Columbia University Computer Science Technical Reports.
MyoMapper ext-link-type="uri" xlink:href="https://github.com/balandinodidonato/MyoMapper">https://github.com/balandinodidonato/MyoMapper 22 March 2016
M. NancelO. ChapuisE. PietrigaX. D. YangP. P. IraniM. Beaudouin-Lafon 2013 High-precision pointing on large wall displays using small handheld devices Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Paris 27 April 2 May 2013 831 840 ACM
K. NymoenM. R. HaugenA. R. Jensenius 2015 MuMYO-Evaluating and Exploring the MYO Armband for Musical Interaction Proceedings of the International Conference on New Interfaces for Musical Expression, Baton Rouge, LA, USA 31 May 31-3 June 2015 NIME’15
M. OmataK. GoA. Imamiya 2000 A gesture-based interface for seamless communication between real and virtual worlds 6th ERCIM Workshop on User Interfaces for All.
E. PartesottiT. F. Tavares 2014 Color and emotion caused by auditory stimuli Ann Arbor, MI Michigan Publishing, University of Michigan Library
F. PlatzR. Kopiez 2012 When the eye listens: A meta-analysis of how audio-visual presentation enhances the appreciation of music performance Music Perception: An Interdisciplinary Journal 30 1 71 83
D. J. PolzellaD. W. Biers 1987 Chromesthetic responses to music: Replication and extension Perceptual and Motor Skills 65 2 439 443
Pure Data ext-link-type="uri" xlink:href="https://puredata.info/">https://puredata.info/ 22 March 2016
Processing ext-link-type="uri" xlink:href="https://processing.org/">https://processing.org/ 22 March 2016
K. SaberiD. R. Perrott 1990 Minimum audible movement angles as a function of sound source trajectory The Journal of the Acoustical Society of America 88 6 2639 2644
P. Schaeffer 1966 Traité des objets musicaux Editions du Seuil, Paris
Sega Amusements ext-link-type="uri" xlink:href="https://www.voutube.com/watch?v=WiORqfFk64s">https://www.voutube.com/watch?v=WiORqfFk64s 15 March 2016
G. S. Settles 2012 Schlieren and shadowgraph techniques: visualizing phenomena in transparent media. Springer Science & Business Media
B. Shneiderman 1997 Direct manipulation for comprehensible, predictable and controllable user interfaces Proceedings of the 2nd international conference on Intelligent user interfaces Orlando, FL, USA 06-09 January 1997 33 39 ACM
J. M. SpeigleJ. M. Loomis 1993 Auditory distance perception by translating observers Virtual Reality, IEEE Annual International Symposium, San Jose, CA, USA 16-23 March 1993 92 99 IEEE Computer Society
C. SpenceJ. Driver 2000 Attracting attention to the illusory location of a sound: reflexive cross modal orienting and ventriloquism Neuroreport 11 9 2057 2061
G. StevenD. Korsgaard 2015 An Exploratory Evaluation of User Interfaces for 3D Audio Mixing Audio Engineering Society Convention 138.
K. Stockhausen 1991 Studie II Multimediale 2
M.-A. Fracchia StoreyF. DH. A. Müller 1999 Cognitive design elements to support the construction of a mental model during software exploration Journal of Systems and Software 44 3 171 185
R. L. Storms 1998 Auditory-Visual Cross-Modal Perception Phenomena Naval Postgraduate School Monterey CA, USA
H. G. TaylorJ. M. Waldram 1933 Improvements in the schlieren method Journal of Scientific Instruments 10 12 378
Thalmic ext-link-type="uri" xlink:href="https://www.myo.com/">https://www.myo.com/ 22 March 2016
D. TranN. N. HaiderS. E. ValavanO. YarovyiL. P. LigthartI. E. LagerA. SzilagyiA. 2012 Architecture and Design Procedure of a Generic SWB Antenna with Superb Performances for Tactical Commands and Ubiquitous Communications Ultra Wideband - Current Status and Future Trends.
M. Vorländer 2007 Auralization: fundamentals of acoustics, modelling, simulation, algorithms and acoustic virtual reality. Springer Science & Business Media
WB Kids ext-link-type="uri" xlink:href="https://youtu.be/XqJqZKL8uEU">https://youtu.be/XqJqZKL8uEU 15 March 2016
3Dception ext-link-type="uri" xlink:href="http://www.twobigears.com/3dception.php">http://www.twobigears.com/3dception.php 12 February 2016

Comments

Comment on this article

[1] Z. AriefI. A. SulistijonoR. A. Ardiansyah 2015 Comparison of five time series EMG features extractions using Myo Armband 2015 International Electronics Symposium (IES), Surabaya, Indonesia 29-30 September 2015 11 14 IEEE

[2] W. BaldonadoQ. MichelleA. WoodruffA. Kuchinsky 2000 Guidelines for using multiple views in information visualization Working Conference on Advanced Visual Interfaces, 110 119 ACM

[3] T. BaudelM. Beaudouin-Lafon 1993 Charade: remote control of objects using free-hand gestures Communications of the ACM 36 7 28 35

[4] R.J.V. BertinA. Berthoz 2004 Visuo-vestibular interaction in the reconstruction of travelled trajectories Experimental Brain Research 154 1 11 21

[5] F. BioccaY. InoueA. LeeH. PolinskyA. Tang 2002 Visual cues and virtual touch: Role of visual stimuli and intersensory integration in cross-modal haptic illusions and the sense of presence Presence 2002 Porto, Portugal October 9-11 2002 410 428

[6] P. BoulinguezS. SavazziC. A. Marzi 2009 Visual trajectory perception in humans: Is it lateralized? Clues from online rTMS of the middle-temporal complex (MT/V5) Behavioral brain research 197 2 481 486

[7] A.S. Bregman 1994 Auditory Scene Analysis: The Perceptual Organization of Sound Bradford Books MIT Press

[8] M. W. Calkins 1893 A Statistical Study of Pseudo-Chromesthesia and of Mental-Forms The American Journal of Psychology 5 4 439 464

[9] O. Cassidy 2015 Hearing in Color; Seeing in Sound: Chromesthesia and Its Influences on Audio-Visual Work

[10] D. W. ChandlerD. W. Grantham 1992 Minimum audible movement angle in the horizontal plane as a function of stimulus frequency and bandwidth, source azimuth, and velocity The Journal of the Acoustical Society of America 91 3 1624 1636

[11] EA Sports ext-link-type="uri" xlink:href="http://nautil.us/issue/26/color/what-color-is-this-song">http://nautil.us/issue/26/color/what-color-is-this-song 15 March 2016

[12] M. Eimer 2004 Multisensory integration: how visual experience shapes spatial perception Current biology 14 3 R115 R117

[13] F. FrassinettiN. BologniniE. Làdavas 2002 Enhancement of visual perception by crossmodal visuo-auditory interaction Experimental Brain Research 147 3 332 343

[14] K. GiannakisM. Smith 2000 Auditory-visual associations for music compositional processes: A survey International Computer Music Conference ICMC2000, Berlin, Germany

[15] A.I. GollerL.J. OttenJ. Ward 2009 Seeing Sounds and Hearing Colors: An Event-related Potential Study of Auditory-Visual Synesthesia Journal of Cognitive Neuroscience 21 10 1869 1881

[16] F. HaqueM. NancelD. Vogel 2015 Myopoint: Pointing and clicking using forearm mounted electromyography and inertial motion sensors 33rd Annual aCm Conference on Human Factors in Computing Systems, 3653 3656 ACM

[17] J. JankowskiH. Martin 2013 A Survey of Interaction Techniques for Interactive 3D Environments Eurographics2013, Girona, Spain 65 93 Eurographics Association

[18] M. Kawato 1999 Internal models for motor control and trajectory planning Current opinion in neurobiology 9 6 718 727

[19] A. Lécuyer 2009 Simulating Haptic Feedback Using Vision: A Survey of Research and Applications of Pseudo-Haptic Feedback Presence 18 1 39 53

[20] S. Lewis 2010 Seeing Sound: Hans Jenny and the Cymatic Atlas, University of Pittsburgh Pennsylvania, USA

[21] L. E. Marks 1975 On Colored-Hearing Synesthesia: Cross-Modal Translations of Sensory Dimensions Psychological Bulletin 82 3 303 31

[22] A. Mazumdar 2013 Principles and Techniques of Schlieren Imaging Systems Columbia University Computer Science Technical Reports.

[23] MyoMapper ext-link-type="uri" xlink:href="https://github.com/balandinodidonato/MyoMapper">https://github.com/balandinodidonato/MyoMapper 22 March 2016

[24] M. NancelO. ChapuisE. PietrigaX. D. YangP. P. IraniM. Beaudouin-Lafon 2013 High-precision pointing on large wall displays using small handheld devices Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Paris 27 April 2 May 2013 831 840 ACM

[25] K. NymoenM. R. HaugenA. R. Jensenius 2015 MuMYO-Evaluating and Exploring the MYO Armband for Musical Interaction Proceedings of the International Conference on New Interfaces for Musical Expression, Baton Rouge, LA, USA 31 May 31-3 June 2015 NIME’15

[26] M. OmataK. GoA. Imamiya 2000 A gesture-based interface for seamless communication between real and virtual worlds 6th ERCIM Workshop on User Interfaces for All.

[27] E. PartesottiT. F. Tavares 2014 Color and emotion caused by auditory stimuli Ann Arbor, MI Michigan Publishing, University of Michigan Library

[28] F. PlatzR. Kopiez 2012 When the eye listens: A meta-analysis of how audio-visual presentation enhances the appreciation of music performance Music Perception: An Interdisciplinary Journal 30 1 71 83

[29] D. J. PolzellaD. W. Biers 1987 Chromesthetic responses to music: Replication and extension Perceptual and Motor Skills 65 2 439 443

[30] Pure Data ext-link-type="uri" xlink:href="https://puredata.info/">https://puredata.info/ 22 March 2016

[31] Processing ext-link-type="uri" xlink:href="https://processing.org/">https://processing.org/ 22 March 2016

[32] K. SaberiD. R. Perrott 1990 Minimum audible movement angles as a function of sound source trajectory The Journal of the Acoustical Society of America 88 6 2639 2644

[33] P. Schaeffer 1966 Traité des objets musicaux Editions du Seuil, Paris

[34] Sega Amusements ext-link-type="uri" xlink:href="https://www.voutube.com/watch?v=WiORqfFk64s">https://www.voutube.com/watch?v=WiORqfFk64s 15 March 2016

[35] G. S. Settles 2012 Schlieren and shadowgraph techniques: visualizing phenomena in transparent media. Springer Science & Business Media

[36] B. Shneiderman 1997 Direct manipulation for comprehensible, predictable and controllable user interfaces Proceedings of the 2nd international conference on Intelligent user interfaces Orlando, FL, USA 06-09 January 1997 33 39 ACM

[37] J. M. SpeigleJ. M. Loomis 1993 Auditory distance perception by translating observers Virtual Reality, IEEE Annual International Symposium, San Jose, CA, USA 16-23 March 1993 92 99 IEEE Computer Society

[38] C. SpenceJ. Driver 2000 Attracting attention to the illusory location of a sound: reflexive cross modal orienting and ventriloquism Neuroreport 11 9 2057 2061

[39] G. StevenD. Korsgaard 2015 An Exploratory Evaluation of User Interfaces for 3D Audio Mixing Audio Engineering Society Convention 138.

[40] K. Stockhausen 1991 Studie II Multimediale 2

[41] M.-A. Fracchia StoreyF. DH. A. Müller 1999 Cognitive design elements to support the construction of a mental model during software exploration Journal of Systems and Software 44 3 171 185

[42] R. L. Storms 1998 Auditory-Visual Cross-Modal Perception Phenomena Naval Postgraduate School Monterey CA, USA

[43] H. G. TaylorJ. M. Waldram 1933 Improvements in the schlieren method Journal of Scientific Instruments 10 12 378

[44] Thalmic ext-link-type="uri" xlink:href="https://www.myo.com/">https://www.myo.com/ 22 March 2016

[45] D. TranN. N. HaiderS. E. ValavanO. YarovyiL. P. LigthartI. E. LagerA. SzilagyiA. 2012 Architecture and Design Procedure of a Generic SWB Antenna with Superb Performances for Tactical Commands and Ubiquitous Communications Ultra Wideband - Current Status and Future Trends.

[46] M. Vorländer 2007 Auralization: fundamentals of acoustics, modelling, simulation, algorithms and acoustic virtual reality. Springer Science & Business Media

[47] WB Kids ext-link-type="uri" xlink:href="https://youtu.be/XqJqZKL8uEU">https://youtu.be/XqJqZKL8uEU 15 March 2016

[48] 3Dception ext-link-type="uri" xlink:href="http://www.twobigears.com/3dception.php">http://www.twobigears.com/3dception.php 12 February 2016

Celebrating 65 years of The Computer Journal - free-to-read perspectives - bcs.org/tcj65