The Parallels between the Study of Crossmodal Correspondence and the Design of Cross-Sensory Mappings

The aim of this paper is to examine how recent research findings and methodologies from the field of cognitive science could be utilised to inform the design and evaluation of cross-modal mappings for multimodal user interaction. In this paper we argue that by using empirical methods to enact embodied knowledge about cross-modal correspondences, we can form an adequate empirical framework for the design of multimodal mappings that successfully align with prior perceptual knowledge. This alignment can significantly improve the human computer dialogue and the analytical, creative and pedagogical value of user interfaces.

Content

Author and article information

Contributors

Augoustinos Tsiros

Conference

Publication date: July 2017

Publication date (Print): July 2017

Pages: 175-182

Affiliations

[0001]Centre for Interaction Design

Edinburgh Napier University

10 Colinton Road, EH10 5DT

Scotland

Article

DOI: 10.14236/ewic/EVA2017.39

SO-VID: de1090a2-8aca-45e1-99e8-1a44fcc8603d

License:

This work is licensed under a Creative Commons Attribution 4.0 Unported License. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/

Conference name: Electronic Visualisation and the Arts (EVA 2017)

Conference acronym: EVA

Conference number:

Conference location: London, UK

Conference date: 11 – 13 July 2017

Conference sponsor: Electronic Workshops in Computing (eWiC)

Conference theme: Electronic Visualisation and the Arts

History

Product

1477-9358 BCS Learning & Development

Self URI (article page): https://www.scienceopen.com/hosted-document?doi=10.14236/ewic/EVA2017.39

Self URI (journal page): https://ewic.bcs.org/

5. REFERENCES

A. B. BrandweinJ. J. FoxeN. N. RussoT. S. AltschulerH. GomesS. Molholm 2011 The development of audiovisual multisensory integration across childhood and early adolescence: a high-density electrical mapping study Cerebral Cortex (New York, N.Y.: 1991), 21(5) 1042 1055
B. CaramiauxF. BevilacquaN. Schnell 2010a Study on Gesture-Sound Similarity 3rd Music and Gesture Conference 1 2
B. CaramiauxF. BevilacquaN. Schnell 2010b Towards a gesture-sound cross-modal analysis I. KoppS. Wachsmuth Gesture in Embodied Communication and Human-Computer Interaction 158 170 Springer Heidelberg
B. CaramiauxJ. FrancoiseF. BevilacquaN. Schnell 2014 Mapping Through Listening Computer Music Journal 38 1 30
S. DolscheidS. HunniusD. Casasantoa. Majid 2014 Prelinguistic Infants Are Sensitive to Space-Pitch Associations Found Across Cultures Psychological Science April 1 6
Z. Eitan 2013 How pitch and loudness shape musical space and motion: new findings and persisting questions S.-L. TanA. J. CohenS. D. LipscombR. A. Kendall The Psychology of Music in Multimedia 161 187 Oxford University Press
Z. EitanA. SchupakA. GotlerL. E. Marks 2011 Lower pitch is larger, yet falling pitches shrink: Interaction of pitch change and size change in speeded discrimination Proceedings of Fechner Day.
B. L. GiordanoP. SusiniR. Bresin 2013 Perceptual Evaluation of Sound-Producing Objects. S. S.K. Franinovic Sonic Interaction Design 151 198 London, England MIT Press
R. GodøyE. HagaA. Jensenius 2006 Exploring music-related gestures by sound-tracing: A preliminary study 2nd ConGAS International Symposiumon Gesture Interfaces for Multimedia Systems
R. I. Godøy 2006 Gestural-Sonorous Objects: embodied extensions of Schaeffer’s conceptual apparatus Organised Sound 11 2 149
R. I. GodøyE. HagaA. R. Jensenius 2006 Playing “ Air Instruments”: Mimicry of Sound-producing Gestures by Novices and Experts. Gesture in HumanComputer Interaction and Simulation 3881 6801 256 267
R. Goldstone 1994 An efficient method for obtaining similarity data Behavior Research Methods, Instruments, & Computers 26 4 381 386
R. L. Goldstone 1994 The role of similarity in categorization: providing a groundwork Cognition 52 2 125 57
R. L. Goldstone 1998 Perceptual learning Annual Review of Psychology 49 585 612
R. Y. GranotZ. Eitan 2011 Musical tension and the interaction of dynamic auditory parameters Music Perception 28 219 245
S. Handel 2006 Perceptual Coherence: Hearing and Seeing. Oxford University Press
D. M. Hardison 2003 Acquisition of second-language speech: Effects of visual cues, context, and talker variability Applied Psycholinguistics 24 4 495 522
D. M. Hardison 2004 Generalization of computer-assisted prosody training: Quantitative and qualitative findings Language Learning & Technology 8 1 34 52
T. HermannA. HuntJ. G. Neuhoff 2011 The Bonification Handbook. The Bonification Handbook.
S. JeschonekS. PauenL. Babocsai 2012 Cross-modal mapping of visual and acoustic displays in infants: The effect of dynamic and static components Journal of Developmental Psychology. 10 337 358
D. Kahneman 2011 Thinking fast and slow. New York Farrar, Straus and Giroux
N. KriegeskorteM. Mur 2012 Inverse MDS: Inferring Dissimilarity Structure from Multiple Item Arrangements Frontiers in Psychology 3 July 245
M. B. Küssner 2014 Shape, drawing and gesture: cross-modal mappings of sound and music. King’s College London
M. B. KussnerD. Leech-Wilkinson 2013 Investigating the influence of musical training on crossmodal correspondences and sensorimotor skills in a realtime drawing paradigm Psychology of Music 42 3 448 469
M. B. KüssnerD. TidharH. M. PriorD. Leech-Wilkinson 2014 Musicians are more consistent: Gestural cross-modal mappings of pitch, loudness and tempo in real-time Frontiers in Psychology 5 July 789
M. S. LandyM. S. BanksD. C. Knill 2012 IdealObserver Models of Cue Integration. M. S. L. Julia TrommershäuserKonrad Kording Sensory Cue Integration 1 35 Oxford Scholarship Online
D. J. LewkowiczA. M. Hansen-Tift 2012 Infants deploy selective attention to the mouth of a talking face when learning speech Proceedings of the National Academy of Sciences of the United States of America 109 5 1431 6
W. LidwellK. HoldenJ. Butler 2003 Universal Principles of Design. Rock Port
V. U. LudwigI. AdachiT. Matsuzawa 2011 Visuoauditory mappings between high luminance and high pitch are shared by chimpanzees (Pan troglodytes) and humans. Proceedings of the National Academy of Sciences of the United States of America 108 51 20661 5
L. E. Marks 1974 On the Associations of Light and Sound. The Mediation of Brightness, Pitch, And Loudness American Journal Of Psychology 87 1-2 173 188
L. E. Marks 1989 On Cross-Modal Similarity: The Perceptual Structure of Pitch, Loudness, and Brightness Journal of Experimental Psychology: Human Perception and Performance 15 3 586 602
L. E. Marks 2004 Cross-modal interactions in speeded classification Gemma CalvertC. SpenceB. E. Stein The Handbook of Multisensory Processes 85 106 MIT press
L. E. MarksR. J. HammealM. H. Bornstein 1987 Perceiving similarity and comprehending metaphor Monogram Society Research Children Development 52 1 102
G. MartinoL. E. Marks 1999 Perceptual and linguistic interactions in speeded classification: Tests of the semantic coding hypothesis Perception 28 903 923
C. J. MondlochD. Maurer 2004 Do small white balls squeak? Pitch-object correspondences in young children Cognitive, Affective, & Behavioral Neuroscience 4 2 133 136
D. Norman 1999 Affordances, Conventions and Design Interactions 6 3 38 42
D. A. Norman 1983 Some observations on mental models. D. G.A. L. Stevens Mental models 7 14 Hillsdale, NJ Lawrence Erlbaum Associates, Publishers Inc
C. PariseC. Spence 2013 Audiovisual crossmodal correspondences in the general population The Oxford Handbook of Synesthesia 790 815
C. Parise 2015 Crossmodal Correspondences: Standing Issues and Experimental Guidelines Multisensory Research 29 October 7 28
L. ShamsU. Beierholm 2012 Humans’ Multisensory Perception, from Integration to Segregation, Follows Bayesian Inference. M. S. L. Julia TrommershäuserKonrad Kording Sensory Cue Integration 1 16 Oxford Scholarship Online
L. ShamsA. R. Seitz 2008 Benefits of multisensory learning Trends in Cognitive Sciences 12 11 411 7
S. ShimojoL. Shams 2001 Sensory modalities are not separate modalities: plasticity and interactions Current Opinion in Neurobiology 505 509
C. Spence 2007 Audiovisual multisensory integration Acoustical Science and Technology 28 2 61 70 https://doi.org/10.1250/ast.28.61
C. Spence 2011 Crossmodal correspondences: a tutorial review Attention, Perception & Psychophysics 73 4 971 95
A. SueyoshiD. M. Hardison 2005 The role of gestures and facial cues in second language listening comprehension Language Learning 55 4 661 699
S. WagnerE. WinnerD. CicchettiH. Gardner 1981 Metaphorical” mapping in human infants Child Development 52 728 731
P. WalkerJ. G. BremnerU. MasonJ. SpringK. MattockA. SlaterS. P. Johnson 2010 Preverbal infants’ sensitivity to synaesthetic cross-modality correspondences Psychological Science 21 1 21 5
V. Walsh 2003 A theory of magnitude: Common cortical metrices of time, space and quality Trends in Cognitive Sciences 7 483 488
C. Ware 2000 Information Visualization: Perception for Design. Morgen Kaufman

Comments

Comment on this article

[1] A. B. BrandweinJ. J. FoxeN. N. RussoT. S. AltschulerH. GomesS. Molholm 2011 The development of audiovisual multisensory integration across childhood and early adolescence: a high-density electrical mapping study Cerebral Cortex (New York, N.Y.: 1991), 21(5) 1042 1055

[2] B. CaramiauxF. BevilacquaN. Schnell 2010a Study on Gesture-Sound Similarity 3rd Music and Gesture Conference 1 2

[3] B. CaramiauxF. BevilacquaN. Schnell 2010b Towards a gesture-sound cross-modal analysis I. KoppS. Wachsmuth Gesture in Embodied Communication and Human-Computer Interaction 158 170 Springer Heidelberg

[4] B. CaramiauxJ. FrancoiseF. BevilacquaN. Schnell 2014 Mapping Through Listening Computer Music Journal 38 1 30

[5] S. DolscheidS. HunniusD. Casasantoa. Majid 2014 Prelinguistic Infants Are Sensitive to Space-Pitch Associations Found Across Cultures Psychological Science April 1 6

[6] Z. Eitan 2013 How pitch and loudness shape musical space and motion: new findings and persisting questions S.-L. TanA. J. CohenS. D. LipscombR. A. Kendall The Psychology of Music in Multimedia 161 187 Oxford University Press

[7] Z. EitanA. SchupakA. GotlerL. E. Marks 2011 Lower pitch is larger, yet falling pitches shrink: Interaction of pitch change and size change in speeded discrimination Proceedings of Fechner Day.

[8] B. L. GiordanoP. SusiniR. Bresin 2013 Perceptual Evaluation of Sound-Producing Objects. S. S.K. Franinovic Sonic Interaction Design 151 198 London, England MIT Press

[9] R. GodøyE. HagaA. Jensenius 2006 Exploring music-related gestures by sound-tracing: A preliminary study 2nd ConGAS International Symposiumon Gesture Interfaces for Multimedia Systems

[10] R. I. Godøy 2006 Gestural-Sonorous Objects: embodied extensions of Schaeffer’s conceptual apparatus Organised Sound 11 2 149

[11] R. I. GodøyE. HagaA. R. Jensenius 2006 Playing “ Air Instruments”: Mimicry of Sound-producing Gestures by Novices and Experts. Gesture in HumanComputer Interaction and Simulation 3881 6801 256 267

[12] R. Goldstone 1994 An efficient method for obtaining similarity data Behavior Research Methods, Instruments, & Computers 26 4 381 386

[13] R. L. Goldstone 1994 The role of similarity in categorization: providing a groundwork Cognition 52 2 125 57

[14] R. L. Goldstone 1998 Perceptual learning Annual Review of Psychology 49 585 612

[15] R. Y. GranotZ. Eitan 2011 Musical tension and the interaction of dynamic auditory parameters Music Perception 28 219 245

[16] S. Handel 2006 Perceptual Coherence: Hearing and Seeing. Oxford University Press

[17] D. M. Hardison 2003 Acquisition of second-language speech: Effects of visual cues, context, and talker variability Applied Psycholinguistics 24 4 495 522

[18] D. M. Hardison 2004 Generalization of computer-assisted prosody training: Quantitative and qualitative findings Language Learning & Technology 8 1 34 52

[19] T. HermannA. HuntJ. G. Neuhoff 2011 The Bonification Handbook. The Bonification Handbook.

[20] S. JeschonekS. PauenL. Babocsai 2012 Cross-modal mapping of visual and acoustic displays in infants: The effect of dynamic and static components Journal of Developmental Psychology. 10 337 358

[21] D. Kahneman 2011 Thinking fast and slow. New York Farrar, Straus and Giroux

[22] N. KriegeskorteM. Mur 2012 Inverse MDS: Inferring Dissimilarity Structure from Multiple Item Arrangements Frontiers in Psychology 3 July 245

[23] M. B. Küssner 2014 Shape, drawing and gesture: cross-modal mappings of sound and music. King’s College London

[24] M. B. KussnerD. Leech-Wilkinson 2013 Investigating the influence of musical training on crossmodal correspondences and sensorimotor skills in a realtime drawing paradigm Psychology of Music 42 3 448 469

[25] M. B. KüssnerD. TidharH. M. PriorD. Leech-Wilkinson 2014 Musicians are more consistent: Gestural cross-modal mappings of pitch, loudness and tempo in real-time Frontiers in Psychology 5 July 789

[26] M. S. LandyM. S. BanksD. C. Knill 2012 IdealObserver Models of Cue Integration. M. S. L. Julia TrommershäuserKonrad Kording Sensory Cue Integration 1 35 Oxford Scholarship Online

[27] D. J. LewkowiczA. M. Hansen-Tift 2012 Infants deploy selective attention to the mouth of a talking face when learning speech Proceedings of the National Academy of Sciences of the United States of America 109 5 1431 6

[28] W. LidwellK. HoldenJ. Butler 2003 Universal Principles of Design. Rock Port

[29] V. U. LudwigI. AdachiT. Matsuzawa 2011 Visuoauditory mappings between high luminance and high pitch are shared by chimpanzees (Pan troglodytes) and humans. Proceedings of the National Academy of Sciences of the United States of America 108 51 20661 5

[30] L. E. Marks 1974 On the Associations of Light and Sound. The Mediation of Brightness, Pitch, And Loudness American Journal Of Psychology 87 1-2 173 188

[31] L. E. Marks 1989 On Cross-Modal Similarity: The Perceptual Structure of Pitch, Loudness, and Brightness Journal of Experimental Psychology: Human Perception and Performance 15 3 586 602

[32] L. E. Marks 2004 Cross-modal interactions in speeded classification Gemma CalvertC. SpenceB. E. Stein The Handbook of Multisensory Processes 85 106 MIT press

[33] L. E. MarksR. J. HammealM. H. Bornstein 1987 Perceiving similarity and comprehending metaphor Monogram Society Research Children Development 52 1 102

[34] G. MartinoL. E. Marks 1999 Perceptual and linguistic interactions in speeded classification: Tests of the semantic coding hypothesis Perception 28 903 923

[35] C. J. MondlochD. Maurer 2004 Do small white balls squeak? Pitch-object correspondences in young children Cognitive, Affective, & Behavioral Neuroscience 4 2 133 136

[36] D. Norman 1999 Affordances, Conventions and Design Interactions 6 3 38 42

[37] D. A. Norman 1983 Some observations on mental models. D. G.A. L. Stevens Mental models 7 14 Hillsdale, NJ Lawrence Erlbaum Associates, Publishers Inc

[38] C. PariseC. Spence 2013 Audiovisual crossmodal correspondences in the general population The Oxford Handbook of Synesthesia 790 815

[39] C. Parise 2015 Crossmodal Correspondences: Standing Issues and Experimental Guidelines Multisensory Research 29 October 7 28

[40] L. ShamsU. Beierholm 2012 Humans’ Multisensory Perception, from Integration to Segregation, Follows Bayesian Inference. M. S. L. Julia TrommershäuserKonrad Kording Sensory Cue Integration 1 16 Oxford Scholarship Online

[41] L. ShamsA. R. Seitz 2008 Benefits of multisensory learning Trends in Cognitive Sciences 12 11 411 7

[42] S. ShimojoL. Shams 2001 Sensory modalities are not separate modalities: plasticity and interactions Current Opinion in Neurobiology 505 509

[43] C. Spence 2007 Audiovisual multisensory integration Acoustical Science and Technology 28 2 61 70 https://doi.org/10.1250/ast.28.61

[44] C. Spence 2011 Crossmodal correspondences: a tutorial review Attention, Perception & Psychophysics 73 4 971 95

[45] A. SueyoshiD. M. Hardison 2005 The role of gestures and facial cues in second language listening comprehension Language Learning 55 4 661 699

[46] S. WagnerE. WinnerD. CicchettiH. Gardner 1981 Metaphorical” mapping in human infants Child Development 52 728 731

[47] P. WalkerJ. G. BremnerU. MasonJ. SpringK. MattockA. SlaterS. P. Johnson 2010 Preverbal infants’ sensitivity to synaesthetic cross-modality correspondences Psychological Science 21 1 21 5

[48] V. Walsh 2003 A theory of magnitude: Common cortical metrices of time, space and quality Trends in Cognitive Sciences 7 483 488

[49] C. Ware 2000 Information Visualization: Perception for Design. Morgen Kaufman

Celebrating 65 years of The Computer Journal - free-to-read perspectives - bcs.org/tcj65