Can you hear the Colour? Towards a Synaesthetic and Multimodal Design Approach in Virtual Worlds

Synaesthesia is a phenomenon where senses naturally combine resulting in, for example, ‘seeing’ music or ‘hearing’ colours. It is of interest in the field of Human-Computer Interaction as a way of creating new or enhanced experiences and interactions with Mixed Reality technologies. In Virtual Reality, research has mainly focused on evaluating advanced graphics and capturing immersion levels and User Experience within ‘typical’ and ‘expected’ interactions. This paper investigates how multimodal design characteristics can lay the foundations to a more ‘synaesthetic’ design approach in Mixed Reality to identify how ‘atypical’ interactions can also affect User Experience. 20 participants completed a maze activity, emotion and immersion surveys and interviews. Results suggest a significant increase in surprise, pride and inspiration and a decrease in interest and enthusiasm. The visual and audio aspects were well received by participants and the sensory elements had a positive effect on User Experience. Time perception was measured and 90 per cent of participants’ time estimations were longer than the actual time. Change blindness was investigated with most participants not noticing the visual or audio changes. Finally, we discuss how this study can inform future projects which aim to implement a synaesthetic-oriented and multimodal approach in Mixed Reality design.

Content

Author and article information

Contributors

Victoria Wright

Genovefa Kefalidou

Conference

Publication date: July 2021

Publication date (Print): July 2021

Pages: 48-59

Affiliations

[0001]School of Informatics

University of Leicester, University Road

Leicester, LE1 7RH, UK

Article

DOI: 10.14236/ewic/HCI2021.4

SO-VID: 1419ecaa-cc5b-40f5-92e0-f859880db055

License:

This work is licensed under a Creative Commons Attribution 4.0 Unported License. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/

Conference name: 34th British HCI Conference

Conference acronym: HCI2021

Conference number: 34

Conference location: London, UK

Conference date: 20th - 21st July 2021

Conference sponsor: Electronic Workshops in Computing (eWiC)

Conference theme: Post-pandemic HCI – Living Digitally

History

Product

1477-9358 BCS Learning & Development

Self URI (article page): https://www.scienceopen.com/hosted-document?doi=10.14236/ewic/HCI2021.4

Self URI (journal page): https://ewic.bcs.org/

7. REFERENCES

AWE - Augmented World Expo (2016) Curtis Hickman THE VOID: Creating The Illusion of Reality. Available from: https://www.youtube.com/watch?v=Ebwtq1HZJ2A (6 May 2021).
G. Bargary, K.J. Barnett, K.J. Mitchell and F.N. Newell (2009) Colored-speech synaesthesia is triggered by multisensory, not unisensory, perception. Psychological Science, 20(5), pp.529-533.
S. Baron-Cohen, M.A. Wyke and C. Binnie (1987) Hearing words and seeing colours: an experimental investigation of a case of synaesthesia. Perception, 16(6), pp.761-767.
M.I. Berkman and E. Akan (2018) Presence and Immersion in Virtual Reality. Encyclopedia of Computer Graphics and Games, pp. 1-10.
Blender (2020) blender.org - Home of the Blender project – Free and Open 3D Creation Software. Available from: https://www.blender.org/ (6 May 2021).
R.A. Block and D. Zakay (1997) Prospective and retrospective duration judgments: A meta-analytic review. Psychonomic bulletin & review, 4(2), pp.184-197.
V. Braun and V. Clarke (2006) Using thematic analysis in psychology. Qualitative research in psychology, 3(2), pp.77-101.
A.L. Brooks (2021) Interactive Multisensory VibroAcoustic Therapeutic Intervention (iMVATi). Recent Advances in Technologies for Inclusive Well-Being, 196, p.325.
S. Casini (2017) Synesthesia, transformation and synthesis: toward a multi-sensory pedagogy of the image. The Senses and Society, 12(1), pp.1-17.
C. Cheng, G.B. Baker and S.M. Dursun (2019) Use of multisensory stimulation interventions in the treatment of major neurocognitive disorders. Psychiatry and Clinical Psychopharmacology, 29(4), pp.916-921.
L. Diesendruck, L. Gertner, L. Botzer, L. Goldfarb, A. Karniel and A. Henik (2010) Months in space: Synaesthesia modulates attention and action. Cognitive neuropsychology, 27(8), pp.665-679.
K. Hinckley and H. Song (2011, May) Sensor synaesthesia: touch in motion, and motion in touch. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 801-810).
Hyper Reality Experience (2017) Hyper Reality Experience | Tick Tock Unlock. Available from: http://hyperrealityexperience.com/ (6 May 2021).
A. Jaimes and N. Sebe (2005, October) Multimodal human computer interaction: A survey. In: International workshop on human-computer interaction (pp. 1-15). Springer, Berlin, Heidelberg.
Jisc Online surveys (2020) Online surveys. Available from: https://www.onlinesurveys.ac.uk/ (Accessed 6 May 2021).
C.H. Lee, D. Lockton, J. Stevens, S.J. Wang and S. Ahn (2019, May) Synaesthetic-Translation Tool: Synaesthesia as an Interactive Material for Ideation. In: Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems (pp. 1-6).
L. Maister, M.J. Banissy and M. Tsakiris (2013) Mirror-touch synaesthesia changes representations of self-identity. Neuropsychologia, 51(5), pp.802-808.
R. McLay, V. Ram, J. Murphy, J. Spira, D.P. Wood, M.D. Wiederhold, B.K. Wiederhold, S. Johnston and D. Reeves (2014) Effect of Virtual Reality PTSD Treatment on Mood and Neurocognitive Outcomes. CyberPsychology, Behavior & Social Networking, 17(7), pp. 439-446.
S. Merter (2017) Synesthetic approach in the design process for enhanced creativity and multisensory experiences. The Design Journal, 20(sup1), pp.S4519-S4528.
Niantic (2016) Pokémon GO. Available from: https://pokemongolive.com/en/ (6 May 2021).
S. Nicholson (2016) The State of Escape: Escape Room Design and Facilities. Meaningful Play Available from: http://scottnicholson.com/pubs/stateofescape.pdf (6 May 2021)
Oculus (2019) Oculus Rift S: VR Headset for VR-ready PCs | Oculus. Available from: https://www.oculus.com/rift-s/ (6 May 2021).
J.H. Patrick, R.A. Pruchno and M.S. Rose, 1998. Recruiting research participants: a comparison of the costs and effectiveness of five recruitment strategies. The Gerontologist, 38(3), pp.295-302.
D. Putrino, J. Ripp, J.E. Herrera, M. Cortes, C. Kellner, D. Rizk and K. Dams-O’Connor (2020) Multisensory, Nature-Inspired Recharge Rooms Yield Short-Term Reductions in Perceived Stress Among Frontline Healthcare Workers. Frontiers in Psychology, 11, p.3213.
S. Rajasekaran, C. Luteran, H. Qu and C. Riley-Doucet, 2011, January. A portable autonomous multisensory intervention device (pamid) for early detection of anxiety and agitation in patients with cognitive impairments. In: 2011 Annual International Conference of the IEEE Engineering in Medicine and Biology Society (pp. 4733-4736). IEEE.
G. Riva, F. Mantovani, C.S. Capideville, A. Preziosa, F. Morganti, D. Villani, A. Gaggioli, C. Botella and M. Alcañiz (2007) Affective interactions using virtual reality: the link between presence and emotions. CyberPsychology & Behavior, 10(1), pp.45-56.
A. Sánchez, J.C. Millán-Calenti, L. Lorenzo-López and A. Maseda (2013) Multisensory stimulation for people with dementia: a review of the literature. American Journal of Alzheimer's Disease & Other Dementias®, 28(1), pp.7-14.
T. Sanders and P. Cairns (2010) Time perception, immersion and music in videogames. Proceedings of HCI 2010 24, pp.160-167.
H.N. Schifferstein (2011, October) Multi sensory design. In: Proceedings of the Second Conference on Creativity and Innovation in Design (pp. 361-362).
E. Shaw, T. Roper, T. Nilsson, G. Lawson, S.V. Cobb and D. Miller (2019, April) The Heat is On: Exploring User Behaviour in a Multisensory Virtual Environment for Fire Evacuation. In: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (pp. 1-13). ACM.
S.P. Smith and D. Trenholme (2009) Rapid prototyping a virtual fire drill environment using computer game technology. Fire safety journal, 44(4), pp.559-569.
F. Spector and D. Maurer (2013) Synesthesia: A new approach to understanding the development of perception. Psychology of Consciousness: Theory, Research, and Practice, 1(S), pp. 108-129.
C. Spence and J. Youssef (2019) Synaesthesia: The multisensory dining experience. International Journal of Gastronomy and Food Science, 18, p.100179.
E. Striem-Amit, M. Guendelman and A. Amedi (2012) ‘Visual’ acuity of the congenitally blind using visual-to-auditory sensory substitution. PloS one, 7(3), p.e33136.
E.A. Suma, S. Clark, D. Krum, S. Finkelstein, M. Bolas and Z. Warte (2011, March). Leveraging change blindness for redirection in virtual environments. In: 2011 IEEE Virtual Reality Conference (pp. 159-166). IEEE.
The VOID (2019) The VOID | A Virtual Reality Experience. Available from: http://web.archive.org/web/20200304070516if_/https://www.thevoid.com/ (8 May 2021).
H. Van Kerrebroeck, M. Brengman and K. Willems (2017) Escaping the crowd: An experimental study on the impact of a Virtual Reality experience in a shopping mall. Computers in Human Behavior, 77(1), pp. 437-450.
M.S. Vitevitch (2003) Change deafness: The inability to detect changes between two voices. Journal of Experimental Psychology: Human Perception and Performance, 29(2), p.333.
M. Voong and R. Beale (2007, April) Music organisation using colour synaesthesia. In CHI'07 extended abstracts on Human Factors in Computing Systems (pp. 1869-1874).
J. Ward, A. P. Field and T. Chin (2019). A meta-analysis of memory ability in synaesthesia. Memory, 27(9), pp. 1299-1312.
J. Wareing, G. Lawson, C. Abdullah and T. Roper (2018, September) User Perception of Heat Source Location for a Multisensory Fire Training Simulation. In: 2018 10th Computer Science and Electronic Engineering (CEEC) pp. 214-218. IEEE.
D. Watson, L.A. Clark and A. Tellegen (1988) Development and validation of brief measures of positive and negative affect: the PANAS scales. Journal of personality and social psychology, 54(6), p.1063.
B.G. Witmer and M.J. Singer (1998) Measuring presence in virtual environments: A presence questionnaire. Presence, 7(3), pp.225-240.

Comments

Comment on this article

[1] AWE - Augmented World Expo (2016) Curtis Hickman THE VOID: Creating The Illusion of Reality. Available from: https://www.youtube.com/watch?v=Ebwtq1HZJ2A (6 May 2021).

[2] G. Bargary, K.J. Barnett, K.J. Mitchell and F.N. Newell (2009) Colored-speech synaesthesia is triggered by multisensory, not unisensory, perception. Psychological Science, 20(5), pp.529-533.

[3] S. Baron-Cohen, M.A. Wyke and C. Binnie (1987) Hearing words and seeing colours: an experimental investigation of a case of synaesthesia. Perception, 16(6), pp.761-767.

[4] M.I. Berkman and E. Akan (2018) Presence and Immersion in Virtual Reality. Encyclopedia of Computer Graphics and Games, pp. 1-10.

[5] Blender (2020) blender.org - Home of the Blender project – Free and Open 3D Creation Software. Available from: https://www.blender.org/ (6 May 2021).

[6] R.A. Block and D. Zakay (1997) Prospective and retrospective duration judgments: A meta-analytic review. Psychonomic bulletin & review, 4(2), pp.184-197.

[7] V. Braun and V. Clarke (2006) Using thematic analysis in psychology. Qualitative research in psychology, 3(2), pp.77-101.

[8] A.L. Brooks (2021) Interactive Multisensory VibroAcoustic Therapeutic Intervention (iMVATi). Recent Advances in Technologies for Inclusive Well-Being, 196, p.325.

[9] S. Casini (2017) Synesthesia, transformation and synthesis: toward a multi-sensory pedagogy of the image. The Senses and Society, 12(1), pp.1-17.

[10] C. Cheng, G.B. Baker and S.M. Dursun (2019) Use of multisensory stimulation interventions in the treatment of major neurocognitive disorders. Psychiatry and Clinical Psychopharmacology, 29(4), pp.916-921.

[11] L. Diesendruck, L. Gertner, L. Botzer, L. Goldfarb, A. Karniel and A. Henik (2010) Months in space: Synaesthesia modulates attention and action. Cognitive neuropsychology, 27(8), pp.665-679.

[12] K. Hinckley and H. Song (2011, May) Sensor synaesthesia: touch in motion, and motion in touch. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 801-810).

[13] Hyper Reality Experience (2017) Hyper Reality Experience | Tick Tock Unlock. Available from: http://hyperrealityexperience.com/ (6 May 2021).

[14] A. Jaimes and N. Sebe (2005, October) Multimodal human computer interaction: A survey. In: International workshop on human-computer interaction (pp. 1-15). Springer, Berlin, Heidelberg.

[15] Jisc Online surveys (2020) Online surveys. Available from: https://www.onlinesurveys.ac.uk/ (Accessed 6 May 2021).

[16] C.H. Lee, D. Lockton, J. Stevens, S.J. Wang and S. Ahn (2019, May) Synaesthetic-Translation Tool: Synaesthesia as an Interactive Material for Ideation. In: Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems (pp. 1-6).

[17] L. Maister, M.J. Banissy and M. Tsakiris (2013) Mirror-touch synaesthesia changes representations of self-identity. Neuropsychologia, 51(5), pp.802-808.

[18] R. McLay, V. Ram, J. Murphy, J. Spira, D.P. Wood, M.D. Wiederhold, B.K. Wiederhold, S. Johnston and D. Reeves (2014) Effect of Virtual Reality PTSD Treatment on Mood and Neurocognitive Outcomes. CyberPsychology, Behavior & Social Networking, 17(7), pp. 439-446.

[19] S. Merter (2017) Synesthetic approach in the design process for enhanced creativity and multisensory experiences. The Design Journal, 20(sup1), pp.S4519-S4528.

[20] Niantic (2016) Pokémon GO. Available from: https://pokemongolive.com/en/ (6 May 2021).

[21] S. Nicholson (2016) The State of Escape: Escape Room Design and Facilities. Meaningful Play Available from: http://scottnicholson.com/pubs/stateofescape.pdf (6 May 2021)

[22] Oculus (2019) Oculus Rift S: VR Headset for VR-ready PCs | Oculus. Available from: https://www.oculus.com/rift-s/ (6 May 2021).

[23] J.H. Patrick, R.A. Pruchno and M.S. Rose, 1998. Recruiting research participants: a comparison of the costs and effectiveness of five recruitment strategies. The Gerontologist, 38(3), pp.295-302.

[24] D. Putrino, J. Ripp, J.E. Herrera, M. Cortes, C. Kellner, D. Rizk and K. Dams-O’Connor (2020) Multisensory, Nature-Inspired Recharge Rooms Yield Short-Term Reductions in Perceived Stress Among Frontline Healthcare Workers. Frontiers in Psychology, 11, p.3213.

[25] S. Rajasekaran, C. Luteran, H. Qu and C. Riley-Doucet, 2011, January. A portable autonomous multisensory intervention device (pamid) for early detection of anxiety and agitation in patients with cognitive impairments. In: 2011 Annual International Conference of the IEEE Engineering in Medicine and Biology Society (pp. 4733-4736). IEEE.

[26] G. Riva, F. Mantovani, C.S. Capideville, A. Preziosa, F. Morganti, D. Villani, A. Gaggioli, C. Botella and M. Alcañiz (2007) Affective interactions using virtual reality: the link between presence and emotions. CyberPsychology & Behavior, 10(1), pp.45-56.

[27] A. Sánchez, J.C. Millán-Calenti, L. Lorenzo-López and A. Maseda (2013) Multisensory stimulation for people with dementia: a review of the literature. American Journal of Alzheimer's Disease & Other Dementias®, 28(1), pp.7-14.

[28] T. Sanders and P. Cairns (2010) Time perception, immersion and music in videogames. Proceedings of HCI 2010 24, pp.160-167.

[29] H.N. Schifferstein (2011, October) Multi sensory design. In: Proceedings of the Second Conference on Creativity and Innovation in Design (pp. 361-362).

[30] E. Shaw, T. Roper, T. Nilsson, G. Lawson, S.V. Cobb and D. Miller (2019, April) The Heat is On: Exploring User Behaviour in a Multisensory Virtual Environment for Fire Evacuation. In: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (pp. 1-13). ACM.

[31] S.P. Smith and D. Trenholme (2009) Rapid prototyping a virtual fire drill environment using computer game technology. Fire safety journal, 44(4), pp.559-569.

[32] F. Spector and D. Maurer (2013) Synesthesia: A new approach to understanding the development of perception. Psychology of Consciousness: Theory, Research, and Practice, 1(S), pp. 108-129.

[33] C. Spence and J. Youssef (2019) Synaesthesia: The multisensory dining experience. International Journal of Gastronomy and Food Science, 18, p.100179.

[34] E. Striem-Amit, M. Guendelman and A. Amedi (2012) ‘Visual’ acuity of the congenitally blind using visual-to-auditory sensory substitution. PloS one, 7(3), p.e33136.

[35] E.A. Suma, S. Clark, D. Krum, S. Finkelstein, M. Bolas and Z. Warte (2011, March). Leveraging change blindness for redirection in virtual environments. In: 2011 IEEE Virtual Reality Conference (pp. 159-166). IEEE.

[36] The VOID (2019) The VOID | A Virtual Reality Experience. Available from: http://web.archive.org/web/20200304070516if_/https://www.thevoid.com/ (8 May 2021).

[37] H. Van Kerrebroeck, M. Brengman and K. Willems (2017) Escaping the crowd: An experimental study on the impact of a Virtual Reality experience in a shopping mall. Computers in Human Behavior, 77(1), pp. 437-450.

[38] M.S. Vitevitch (2003) Change deafness: The inability to detect changes between two voices. Journal of Experimental Psychology: Human Perception and Performance, 29(2), p.333.

[39] M. Voong and R. Beale (2007, April) Music organisation using colour synaesthesia. In CHI'07 extended abstracts on Human Factors in Computing Systems (pp. 1869-1874).

[40] J. Ward, A. P. Field and T. Chin (2019). A meta-analysis of memory ability in synaesthesia. Memory, 27(9), pp. 1299-1312.

[41] J. Wareing, G. Lawson, C. Abdullah and T. Roper (2018, September) User Perception of Heat Source Location for a Multisensory Fire Training Simulation. In: 2018 10th Computer Science and Electronic Engineering (CEEC) pp. 214-218. IEEE.

[42] D. Watson, L.A. Clark and A. Tellegen (1988) Development and validation of brief measures of positive and negative affect: the PANAS scales. Journal of personality and social psychology, 54(6), p.1063.

[43] B.G. Witmer and M.J. Singer (1998) Measuring presence in virtual environments: A presence questionnaire. Presence, 7(3), pp.225-240.

Studying business & IT? Drive your professional career forwards with BCS books - for a 20% discount click here: shop.bcs.org