1,114
views
0
recommends
+1 Recommend
1 collections
    0
    shares

      Celebrating 65 years of The Computer Journal - free-to-read perspectives - bcs.org/tcj65

      scite_
       
      • Record: found
      • Abstract: found
      • Conference Proceedings: found
      Is Open Access

      Can you hear the Colour? Towards a Synaesthetic and Multimodal Design Approach in Virtual Worlds

      proceedings-article
      ,
      34th British HCI Conference (HCI2021)
      Post-pandemic HCI – Living Digitally
      20th - 21st July 2021
      Multisensory, Virtual Environment, Synaesthetic-oriented approach, Change blindness, Time perception
      Bookmark

            Abstract

            Synaesthesia is a phenomenon where senses naturally combine resulting in, for example, ‘seeing’ music or ‘hearing’ colours. It is of interest in the field of Human-Computer Interaction as a way of creating new or enhanced experiences and interactions with Mixed Reality technologies. In Virtual Reality, research has mainly focused on evaluating advanced graphics and capturing immersion levels and User Experience within ‘typical’ and ‘expected’ interactions. This paper investigates how multimodal design characteristics can lay the foundations to a more ‘synaesthetic’ design approach in Mixed Reality to identify how ‘atypical’ interactions can also affect User Experience. 20 participants completed a maze activity, emotion and immersion surveys and interviews. Results suggest a significant increase in surprise, pride and inspiration and a decrease in interest and enthusiasm. The visual and audio aspects were well received by participants and the sensory elements had a positive effect on User Experience. Time perception was measured and 90 per cent of participants’ time estimations were longer than the actual time. Change blindness was investigated with most participants not noticing the visual or audio changes. Finally, we discuss how this study can inform future projects which aim to implement a synaesthetic-oriented and multimodal approach in Mixed Reality design.

            Content

            Author and article information

            Contributors
            Conference
            July 2021
            July 2021
            : 48-59
            Affiliations
            [0001]School of Informatics

            University of Leicester, University Road

            Leicester, LE1 7RH, UK
            Article
            10.14236/ewic/HCI2021.4
            1419ecaa-cc5b-40f5-92e0-f859880db055
            © Wright et al. Published by BCS Learning & Development Ltd. Proceedings of the BCS 34th British HCI Conference 2021, UK

            This work is licensed under a Creative Commons Attribution 4.0 Unported License. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/

            34th British HCI Conference
            HCI2021
            34
            London, UK
            20th - 21st July 2021
            Electronic Workshops in Computing (eWiC)
            Post-pandemic HCI – Living Digitally
            History
            Product

            1477-9358 BCS Learning & Development

            Self URI (article page): https://www.scienceopen.com/hosted-document?doi=10.14236/ewic/HCI2021.4
            Self URI (journal page): https://ewic.bcs.org/
            Categories
            Electronic Workshops in Computing

            Applied computer science,Computer science,Security & Cryptology,Graphics & Multimedia design,General computer science,Human-computer-interaction
            Time perception,Synaesthetic-oriented approach,Multisensory,Change blindness,Virtual Environment

            7. REFERENCES

            1. AWE - Augmented World Expo (2016) Curtis Hickman THE VOID: Creating The Illusion of Reality. Available from: https://www.youtube.com/watch?v=Ebwtq1HZJ2A (6 May 2021).

            2. , , and (2009) Colored-speech synaesthesia is triggered by multisensory, not unisensory, perception. Psychological Science, 20(5), pp.529-533.

            3. , and (1987) Hearing words and seeing colours: an experimental investigation of a case of synaesthesia. Perception, 16(6), pp.761-767.

            4. and (2018) Presence and Immersion in Virtual Reality. Encyclopedia of Computer Graphics and Games, pp. 1-10.

            5. Blender (2020) blender.org - Home of the Blender project – Free and Open 3D Creation Software. Available from: https://www.blender.org/ (6 May 2021).

            6. and (1997) Prospective and retrospective duration judgments: A meta-analytic review. Psychonomic bulletin & review, 4(2), pp.184-197.

            7. and (2006) Using thematic analysis in psychology. Qualitative research in psychology, 3(2), pp.77-101.

            8. (2021) Interactive Multisensory VibroAcoustic Therapeutic Intervention (iMVATi). Recent Advances in Technologies for Inclusive Well-Being, 196, p.325.

            9. (2017) Synesthesia, transformation and synthesis: toward a multi-sensory pedagogy of the image. The Senses and Society, 12(1), pp.1-17.

            10. , and (2019) Use of multisensory stimulation interventions in the treatment of major neurocognitive disorders. Psychiatry and Clinical Psychopharmacology, 29(4), pp.916-921.

            11. , , , , and (2010) Months in space: Synaesthesia modulates attention and action. Cognitive neuropsychology, 27(8), pp.665-679.

            12. and (2011, May) Sensor synaesthesia: touch in motion, and motion in touch. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 801-810).

            13. Hyper Reality Experience (2017) Hyper Reality Experience | Tick Tock Unlock. Available from: http://hyperrealityexperience.com/ (6 May 2021).

            14. and (2005, October) Multimodal human computer interaction: A survey. In: International workshop on human-computer interaction (pp. 1-15). Springer, Berlin, Heidelberg.

            15. Jisc Online surveys (2020) Online surveys. Available from: https://www.onlinesurveys.ac.uk/ (Accessed 6 May 2021).

            16. , , , and (2019, May) Synaesthetic-Translation Tool: Synaesthesia as an Interactive Material for Ideation. In: Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems (pp. 1-6).

            17. , and (2013) Mirror-touch synaesthesia changes representations of self-identity. Neuropsychologia, 51(5), pp.802-808.

            18. , , , , , , , and (2014) Effect of Virtual Reality PTSD Treatment on Mood and Neurocognitive Outcomes. CyberPsychology, Behavior & Social Networking, 17(7), pp. 439-446.

            19. (2017) Synesthetic approach in the design process for enhanced creativity and multisensory experiences. The Design Journal, 20(sup1), pp.S4519-S4528.

            20. Niantic (2016) Pokémon GO. Available from: https://pokemongolive.com/en/ (6 May 2021).

            21. (2016) The State of Escape: Escape Room Design and Facilities. Meaningful Play Available from: http://scottnicholson.com/pubs/stateofescape.pdf (6 May 2021)

            22. Oculus (2019) Oculus Rift S: VR Headset for VR-ready PCs | Oculus. Available from: https://www.oculus.com/rift-s/ (6 May 2021).

            23. , and , 1998. Recruiting research participants: a comparison of the costs and effectiveness of five recruitment strategies. The Gerontologist, 38(3), pp.295-302.

            24. , , , , , and (2020) Multisensory, Nature-Inspired Recharge Rooms Yield Short-Term Reductions in Perceived Stress Among Frontline Healthcare Workers. Frontiers in Psychology, 11, p.3213.

            25. , , and , 2011, January. A portable autonomous multisensory intervention device (pamid) for early detection of anxiety and agitation in patients with cognitive impairments. In: 2011 Annual International Conference of the IEEE Engineering in Medicine and Biology Society (pp. 4733-4736). IEEE.

            26. , , , , , , , and (2007) Affective interactions using virtual reality: the link between presence and emotions. CyberPsychology & Behavior, 10(1), pp.45-56.

            27. , , and (2013) Multisensory stimulation for people with dementia: a review of the literature. American Journal of Alzheimer's Disease & Other Dementias®, 28(1), pp.7-14.

            28. and (2010) Time perception, immersion and music in videogames. Proceedings of HCI 2010 24, pp.160-167.

            29. (2011, October) Multi sensory design. In: Proceedings of the Second Conference on Creativity and Innovation in Design (pp. 361-362).

            30. , , , , and (2019, April) The Heat is On: Exploring User Behaviour in a Multisensory Virtual Environment for Fire Evacuation. In: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (pp. 1-13). ACM.

            31. and (2009) Rapid prototyping a virtual fire drill environment using computer game technology. Fire safety journal, 44(4), pp.559-569.

            32. and (2013) Synesthesia: A new approach to understanding the development of perception. Psychology of Consciousness: Theory, Research, and Practice, 1(S), pp. 108-129.

            33. and (2019) Synaesthesia: The multisensory dining experience. International Journal of Gastronomy and Food Science, 18, p.100179.

            34. , and (2012) ‘Visual’ acuity of the congenitally blind using visual-to-auditory sensory substitution. PloS one, 7(3), p.e33136.

            35. , , , , and (2011, March). Leveraging change blindness for redirection in virtual environments. In: 2011 IEEE Virtual Reality Conference (pp. 159-166). IEEE.

            36. The VOID (2019) The VOID | A Virtual Reality Experience. Available from: http://web.archive.org/web/20200304070516if_/https://www.thevoid.com/ (8 May 2021).

            37. , and (2017) Escaping the crowd: An experimental study on the impact of a Virtual Reality experience in a shopping mall. Computers in Human Behavior, 77(1), pp. 437-450.

            38. (2003) Change deafness: The inability to detect changes between two voices. Journal of Experimental Psychology: Human Perception and Performance, 29(2), p.333.

            39. and (2007, April) Music organisation using colour synaesthesia. In CHI'07 extended abstracts on Human Factors in Computing Systems (pp. 1869-1874).

            40. , and (2019). A meta-analysis of memory ability in synaesthesia. Memory, 27(9), pp. 1299-1312.

            41. , , and (2018, September) User Perception of Heat Source Location for a Multisensory Fire Training Simulation. In: 2018 10th Computer Science and Electronic Engineering (CEEC) pp. 214-218. IEEE.

            42. , and (1988) Development and validation of brief measures of positive and negative affect: the PANAS scales. Journal of personality and social psychology, 54(6), p.1063.

            43. and (1998) Measuring presence in virtual environments: A presence questionnaire. Presence, 7(3), pp.225-240.

            Comments

            Comment on this article