Blog
About
Search
Advanced search

168
views
52
references
 Top references
0
cited by
 Most cited
    
0 reviews
 Review
0
comments
 Comment
0
recommends
+1 Recommend
1 collections
    9
    shares
      191
      similar
       All similar
      • Record: found
      • Abstract: found
      • Article: found
      Is Open Access

      The Parallels between the Study of Crossmodal Correspondence and the Design of Cross-Sensory Mappings

      Authors:

      Publication date (Print):

      Conference name: Electronic Visualisation and the Arts (EVA 2017) (EVA)

      Conference theme: Electronic Visualisation and the Arts

      Conference date: 11 – 13 July 2017

      Keywords: Display and interaction design, Mappings, Perceptual learning, Crossmodal correspondence

      Read this article at

      ScienceOpen
      Bookmark
          There is no author summary for this article yet. Authors can add summaries to their articles on ScienceOpen to make them more accessible to a non-specialist audience.

          Abstract

          The aim of this paper is to examine how recent research findings and methodologies from the field of cognitive science could be utilised to inform the design and evaluation of cross-modal mappings for multimodal user interaction. In this paper we argue that by using empirical methods to enact embodied knowledge about cross-modal correspondences, we can form an adequate empirical framework for the design of multimodal mappings that successfully align with prior perceptual knowledge. This alignment can significantly improve the human computer dialogue and the analytical, creative and pedagogical value of user interfaces.

          Related collections

          Most cited references 52

          • Record: found
          • Abstract: found
          • Article: not found

          Crossmodal correspondences: a tutorial review.

           Charles Spence (2011)
          In many everyday situations, our senses are bombarded by many different unisensory signals at any given time. To gain the most veridical, and least variable, estimate of environmental stimuli/properties, we need to combine the individual noisy unisensory perceptual estimates that refer to the same object, while keeping those estimates belonging to different objects or events separate. How, though, does the brain "know" which stimuli to combine? Traditionally, researchers interested in the crossmodal binding problem have focused on the roles that spatial and temporal factors play in modulating multisensory integration. However, crossmodal correspondences between various unisensory features (such as between auditory pitch and visual size) may provide yet another important means of constraining the crossmodal binding problem. A large body of research now shows that people exhibit consistent crossmodal correspondences between many stimulus features in different sensory modalities. For example, people consistently match high-pitched sounds with small, bright objects that are located high up in space. The literature reviewed here supports the view that crossmodal correspondences need to be considered alongside semantic and spatiotemporal congruency, among the key constraints that help our brains solve the crossmodal binding problem.
          Article 0 comments Cited 188 times – based on 0 reviews     Review now
            Bookmark
            • Record: found
            • Abstract: found
            • Article: not found

            Benefits of multisensory learning.

             Ladan Shams,  Aaron R. Seitz (2008)
            Studies of learning, and in particular perceptual learning, have focused on learning of stimuli consisting of a single sensory modality. However, our experience in the world involves constant multisensory stimulation. For instance, visual and auditory information are integrated in performing many tasks that involve localizing and tracking moving objects. Therefore, it is likely that the human brain has evolved to develop, learn and operate optimally in multisensory environments. We suggest that training protocols that employ unisensory stimulus regimes do not engage multisensory learning mechanisms and, therefore, might not be optimal for learning. However, multisensory-training protocols can better approximate natural settings and are more effective for learning.
            Article 0 comments Cited 99 times – based on 0 reviews     Review now
              Bookmark
              • Record: found
              • Abstract: found
              • Article: not found

              Infants deploy selective attention to the mouth of a talking face when learning speech.

               D J Lewkowicz,  Immo Hansen (2012)
              The mechanisms underlying the acquisition of speech-production ability in human infancy are not well understood. We tracked 4-12-mo-old English-learning infants' and adults' eye gaze while they watched and listened to a female reciting a monologue either in their native (English) or nonnative (Spanish) language. We found that infants shifted their attention from the eyes to the mouth between 4 and 8 mo of age regardless of language and then began a shift back to the eyes at 12 mo in response to native but not nonnative speech. We posit that the first shift enables infants to gain access to redundant audiovisual speech cues that enable them to learn their native speech forms and that the second shift reflects growing native-language expertise that frees them to shift attention to the eyes to gain access to social cues. On this account, 12-mo-old infants do not shift attention to the eyes when exposed to nonnative speech because increasing native-language expertise and perceptual narrowing make it more difficult to process nonnative speech and require them to continue to access redundant audiovisual cues. Overall, the current findings demonstrate that the development of speech production capacity relies on changes in selective audiovisual attention and that this depends critically on early experience.
              Article 0 comments Cited 74 times – based on 0 reviews     Review now
                Bookmark
                All references

                Author and article information

                Contributors
                Augoustinos Tsiros
                Conference
                Publication date: July 2017
                Publication date (Print): July 2017
                Pages: 175-182
                Affiliations
                Centre for Interaction Design

                Edinburgh Napier University

                10 Colinton Road, EH10 5DT

                Scotland
                Article
                DOI: 10.14236/ewic/EVA2017.39
                Copyright statement: © Tsiros. Published by BCS Learning and Development Ltd. Proceedings of EVA London 2017, UK

                License:

                This work is licensed under a Creative Commons Attribution 4.0 Unported License. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/

                Conference name: Electronic Visualisation and the Arts (EVA 2017)
                Conference acronym: EVA
                Conference number:
                Conference location: London, UK
                Conference date: 11 – 13 July 2017
                Conference sponsor: Electronic Workshops in Computing (eWiC)
                Conference theme: Electronic Visualisation and the Arts
                Product
                Product Information: 1477-9358 BCS Learning & Development
                Self URI (journal page): https://ewic.bcs.org/
                Categories
                Series title: Electronic Workshops in Computing

                Data availability:

                ScienceOpen disciplines: Applied computer science, Computer science, Security & Cryptology, Graphics & Multimedia design, General computer science, Human-computer-interaction

                Keywords: Crossmodal correspondence, Display and interaction design, Mappings, Perceptual learning

                Comments

                Comment on this article

                Similar content 191

                See all similar

                Most referenced authors 146

                See all reference authors