Synaesthetic-oriented design, inspired by the phenomenon Synaesthesia, focuses on introducing unexpected or atypical sensory combinations into designs. Whilst this design approach has been used to design software, there is limited research into how it can affect User Experience (UX), especially in Mixed Reality technologies. This paper provides suggestions on how to create synaesthetic-oriented Virtual Environments (VEs) based on an exploratory study. The study had 30 participants split into three groups with different sensory combinations: synaesthetic-oriented (atypical sensory combinations), multimodal (typical sensory combinations) and visual-only (control group). Participants completed four mazes with puzzles which, for the non-control groups, had visual, auditory, tactile, olfactory and gustatory elements. The results suggest that positive aspects of synaesthetic-oriented design include surprise, participants creating their own links to the VE based on their past experiences and potentially presence. Challenges for synaesthetic-oriented design include balancing the simultaneous senses, not matching user expectations (negative surprise) and designing for participants with higher goal focus and/or need to rationalise. As the study was run remotely due to COVID-19, suggestions are also provided for incorporating smell and taste into a VE remotely.
Ablart, D., Velasco, C., Vi, C.T., Gatti, E. and Obrist, M. (2017) The how and why behind a multisensory art display. Interactions, 24(6). 38-43. https://doi.org/10.1145/3137091
Antle, A.N. and Wang, S. (2013) Comparing motor-cognitive strategies for spatial problem solving with tangible and multi-touch interfaces. In: Proceedings of the 7th International Conference on tangible, embedded and embodied interaction. Barcelona, Spain, 10-13 February 2013. New York, USA: ACM. 65-72. https://doi.org/10.1145/2460625.2460635
AR.js (n.d.) AR.js Documentation. Available from: https://ar-js-org.github.io/AR.js-Docs (28 June 2023)
Arnold, P., Khot, R.A. and Mueller, F. (2018) "You Better Eat to Survive" Exploring Cooperative Eating in Virtual Reality Games. In: Proceedings of the Twelfth International Conference on Tangible, Embedded, and Embodied Interaction. Stockholm, Sweden, 18-21 March 2018. New York, USA: ACM. 398-408. https://doi.org/10.1145/3173225.3173238
Atelier des Lumières (2022) Kandinsky | Atelier des Lumières - Site officiel. Available from: https://www.atelier-lumieres.com/en/kandinsky (28 June 2023)
Bach, J.S. (1720) European Archive. J.S. Bach - Sonata No.1, G minor - IV. Presto. Available from: https://musopen.org/music/3935-violin-sonata-in-g-minor-bwv-1001/ (28 June 2023)
Bach, J.S. (1721) European Archive. Brandenburg Concerto no. 1 in F major, BWV. 1046 - IV. Menuetto. Available from: https://musopen.org/music/3498-brandenburg-concerto-no-1-in-f-major-bwv-1046/ (28 June 2023)
Bach, J.S. (1723a) European Archive. J.S Bach - Suite No.4 in E flat - IV. Serabande Bourree No.1. Available from: https://musopen.org/music/4003-cello-suite-no-4-in-e-flat-major-bwv-1010/ (28 June 2023)
Bach, J.S. (1723b) European Archive. Orchestral suite no. 3 in D major, BWV 1068 - 1. Ouverture. Available from: https://musopen.org/music/3775-orchestral-suite-no-3-in-d-major-bwv-1068/ (28 June 2023)
Bach, J.S. (1735) The German Bach Soloists. Triple Concerto in D Minor for flute, oboe, violin & strings BWV 1064 - Complete Performance. Available from: https://musopen.org/music/3511-concerto-for-3-violins-in-d-major-bwv-1064r/ (28 June 2023)
Belloli, S., Porro, S., Virk, V.S., Etzi, R., Gallace, A., Bordegoni, M. and Carulli, M. (2020) The Kandinsky experience: A multisensory augmented reality application for cultural heritage. Computer-Aided Design, 18(4). 799-814. https://doi.org/10.14733/cadaps.2021.799-814
Blender (2023) blender.org – Home of the Blender project – Free and Open 3D Creation Software. Available from: https://www.blender.org/ (28 June 2023)
Casini, S. (2017) Synesthesia, transformation and synthesis: toward a multi-sensory pedagogy of the image. The Senses and Society, 12(1). 1-17. https://doi.org/10.1080/17458927.2017.1268811
Chisik, Y., Pons, P. and Jaen, J. (2018) Gastronomy meets ludology: Towards a definition of what it means to play with your (digital) food. In: Proceedings of the 2018 Annual Symposium on Computer-Human Interaction in Play Companion Extended Abstracts. Melbourne, VIC, Australia 28-31 October 2018. New York, USA: ACM. 155-168. https://doi.org/10.1145/3270316.3272056
Cing (2005) Another Code: Two Memories. Nintendo DS [Game]. Kyoto, Japan: Nintendo.
Cing (2009) Another Code: R – A Journey into Lost Memories. Nintendo Wii [Game]. Kyoto, Japan: Nintendo.
Conway, S. (2010) A circular wall? Reformulating the fourth wall for videogames. Journal of Gaming & Virtual Worlds, 2(2). 145- 155. https://doi.org/10.1386/jgvw.2.2.145_1
Creative Commons (n.d. a) Creative Commons – Public Domain Mark 1.0. Available from: https://creativecommons.org/publicdomain/mark/1.0/ (28 June 2023)
Creative Commons (n.d. b) Creative Commons – Attribution 3.0 Unported – CC BY 3.0. Available from: https://creativecommons.org/licenses/by/3.0/ (28 June 2023)
Creative Commons (n.d. c) Creative Commons – Attribution-NonCommercial 4.0 International – CC BY-NC 4.0. Available from: https://creativecommons.org/licenses/by-nc/4.0/ (28 June 2023)
Creed, C., Frutos-Pascual, M. and Williams, I. (2020) Multimodal Gaze Interaction for Creative Design. In: Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. Honolulu, HI, USA, 25-30 April 2020. New York, USA: ACM. 1-13. https://doi.org/10.1145/3313831.3376196
Debussy, C. (1894) Natalia Ensemble. Prelude to the Afternoon of a Faun (ensemble arr.). Available from: https://musopen.org/music/42994- prelude-to-the-afternoon-of-a-faun/ (28 June 2023)
Debussy, C. (1905) US Air Force Band. La Mer - 2 - Jeux de vagues. Available from: https://musopen.org/music/14381-la-mer/ (28 June 2023)
Debussy, C. (1915) Pitman, P. and Lee, B.E. Cello Sonata - I. Prologue - lent. Available from: https://musopen.org/music/4516-cello-sonata/ (28 June 2023)
Deroy, O. and Spence, C. (2013) Are we all born synaesthetic? Examining the neonatal synaesthesia hypothesis. Neuroscience & Biobehavioral Review, 37(7). 1240-1253. https://doi.org/10.1016/j.neubiorev.2013.04.001
Dix, A. (2007) Designing for appropriation. In: Proceedings of HCI 2007 The 21st British HCI Group Annual Conference. Lancaster, UK, 3-7 September 2007. Swindon, UK: British Computer Society Learning and Development Ltd. 1-4. https://doi.org/10.14236/ewic/HCI2007.53
Doucé, L. and Adams, C. (2020) Sensory overload in a shopping environment: Not every sensory modality leads to too much stimulation. Journal of Retailing and Consumer Services, 57. 102154. https://doi.org/10.1016/j.jretconser.2020.102154
Escape Hunt UK (2023) Escape Hunt UK | Play at Home Games | Download, Print & Play. Available from: https://playathome-uk.escapehunt.com/ (28 June 2023)
Feist, G.J. and Brady, T.R. (2004) Openness to experience, non-conformity, and the preference for abstract art. Empirical Studies of the Arts, 22(1), 77-89. https://doi.org/10.2190/Y7CA-TBY6-V7LR-76GK
Flavián, C., Ibáñez-Sánchez, S. and Orús, C. (2021) The influence of scent on virtual reality experiences: The role of aroma-content congruence. Journal of Business Research, 123. 289-301. https://doi.org/10.1016/j.jbusres.2020.09.036
Garcia-Ruiz, M.A., Kapralos, B. and Rebolledo-Mendez, G. (2021) An overview of olfactory displays in education and training. Multimodal Technologies and Interaction, 5(10). 64. https://doi.org/10.3390/mti5100064
GIFT TEN INDUSTRY K.K. (2018) Madorica Real Estate. PC, Nintendo Switch [Game]. Japan: GIFT TEN INDUSTRY K.K..
Google Arts & Culture (2021) Play a Kandinsky -- Google Arts & Culture. Available from: https://artsandculture.google.com/experiment/sgF5ivv105ukhA (28 June 2023)
GpnArts (2017) How to make a strong box from paper | DIY - Do it Yourself Origami. Available from: https://www.youtube.com/watch?v=caDwbe02l6g (28 June 2023)
Gross, J.J. and Levenson, R.W. (1995) Emotion elicitation using films. Cognition & Emotion, 9(1). 87-108. https://doi.org/10.1080/02699939508408966
Gross, A. and Silvennoinen, J. (2014) Surprise as a design strategy in goal-oriented mobile applications. In: Proceedings of the 5th International Conference on Applied Human Factors and Ergonomics. Krakow, Poland, 19-23 July 2014. The Applied Human Factors and Ergonomics. 4716-4726. http://urn.fi/URN:NBN:fi:jyu-201408072314
Hinckley, K. and Song, H. (2011) Sensor synaesthesia: touch in motion, and motion in touch. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. Vancouver, BC, Canada, 7-12 May 2011. New York, USA: ACM. 801-810. https://doi.org/10.1145/1978942.1979059
Huang, F., Huang, J. and Wan, X. (2019) Influence of virtual color on taste: Multisensory integration between virtual and real worlds. Computers in Human Behavior, 95. 168-174. https://doi.org/10.1016/j.chb.2019.01.027
Huber, B. and Gajos, K.Z. (2020) Conducting online virtual environment experiments with uncompensated, unsupervised samples. PLoS ONE, 15(1). e0227629. https://doi.org/10.1371/journal.pone.0227629
Hyper Reality Experience (2017) Hyper Reality Experience | Tick Tock Unlock. Available from: http://hyperrealityexperience.com/ (28 June 2023)
Immersive Galleries UK Ltd (2023) Home - Frameless. Available from: https://frameless.com/ (28 June 2023)
Just, D.K.C. (2017) Was Kandinsky a Synaesthete? Examining His Writings and Other Evidence. Multisensory Research, 30(3-5). 447-460. https://doi.org/10.1163/22134808-00002547
Kefalidou, G. and Sharples, S. (2016) Encouraging serendipity in research: Designing technologies to support connection-making. International Journal of Human-Computer Studies, 89. 1-23. https://doi.org/10.1016/j.ijhcs.2016.01.003
Kerruish, E. (2019) Arranging sensations: smell and taste in augmented and virtual reality. The Senses and Society, 14(1). 31-45. https://doi.org/10.1080/17458927.2018.1556952
Lee, C.H., Lockton, D., Stevens, J., Wang, S.J. and Ahn, S.H. (2019) Synaesthetic-Translation Tool: Synaesthesia as an Interactive Material for Ideation. In: Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems. Glasgow, Scotland, UK, 4-9 May 2019. New York, USA: ACM. 1-6. https://doi.org/10.1145/3290607.3312849
Liu, P., Stepanova, E.R., Kitson, A., Schiphorst, T. and Riecke and B.E. (2022). Virtual Transcendent Dream: Empowering People through Embodied Flying in Virtual Reality. In: CHI Conference on Human Factors in Computing Systems. New Orleans, LA, USA, 29 April-5 May 2022. New York, USA: ACM. 1-18. https://doi.org/10.1145/3491102.3517677
Lukava, T., Morgado Ramirez, D.Z. and Barbareschi, G. (2022) Two sides of the same coin: accessibility practices and neurodivergent users' experience of extended reality. Journal of Enabling Technologies, 16(2). 75-90. https://doi.org/10.1108/JET-03-2022-0025
Melo, M., Gonçalves, G., Monteiro, P., Coelho, H., Vasconcelos-Raposo, J. and Bessa, M (2020) Do multisensory stimuli benefit the virtual reality experience? A systematic review. IEEE Transactions on Visualization and Computer Graphics, 28(2). 1428-1442. https://doi.org/10.1109/TVCG.2020.3010088
Merter, S. (2017) Synesthetic approach in the design process for enhanced creativity and multisensory experiences. The Design Journal, 20(sup1). S4519-S4528. https://doi.org/10.1080/14606925.2017.1352948
mike_stranks (2016) Passing Traffic on country road.wav. Available from: https://freesound.org/people/mike_stranks/sounds/341603/ (28 June 2023)
Moser, C. and Tscheligi, M. (2013) Playful taste interaction. In: Proceedings of the 12th International Conference on Interaction Design and Children. New York, NY, USA, 24-27 June 2013. New York, USA: ACM. 340-343. https://doi.org/10.1145/2485760.2485828
Mozart, W.A. (1788) Musopen Symphony. Symphony No.40 in G minor, K.550 - III. Meneutto; Allegretto. Available from: https://musopen.org/music/1577-symphony-no-40-in-g-minor-k-550/ (28 June 2023)
Mozart, W.A. (1778) Skidmore College Orchestra. Flute Concerto in G Major, k. 313 - i. allegro maestoso. Available from: https://commons.wikimedia.org/wiki/File:Wolfgang_Amadeus_Mozart_-_flute_concerto_in_g_major,_k._313_- _i._allegro_maestoso.ogg (28 June 2023)
Muender, T., Bonfert, M., Reinschluessel, A.V., Malaka, R. and Döring, T. (2022) Haptic Fidelity Framework: Defining the Factors of Realistic Haptic Feedback for Virtual Reality. In: CHI Conference on Human Factors in Computing Systems. New Orleans, LA, USA, 29 April-5 May 2022. New York, USA: ACM. 1-17. https://doi.org/10.1145/3491102.3501953
Murer, M., Aslan, I. and Tscheligi, M. (2013) LOLLio: exploring taste as playful modality. In: Proceedings of the 7th International Conference on Tangible, Embedded and Embodied Interaction. Barcelona, Spain, 10-13 February 2013. New York, USA: ACM. 299-302. https://doi.org/10.1145/2460625.2460675
Nakamoto, T., Otaguro, S., Kinoshita, M., Nagahama, M., Ohinishi, K. and Ishida, T. (2008) Cooking up an interactive olfactory game display. IEEE Computer Graphics and Applications, 28(1). 75-78. https://doi.org/10.1109/MCG.2008.3
Nicholson, S. (2016) Ask Why: Creating a Better Player Experience through Environmental Storytelling and Consistency in Escape Room Design. In: Meaningful Play 2016. East Lansing, MI, USA, 20-22 October 2016. ETC Press. 1-7.
Nintendo EAD (2007) The Legend of Zelda: Phantom Hourglass. Nintendo DS [Game]. Kyoto, Japan: Nintendo.
Nintendo R&D3 and Locomotive Corporation (1990) Startropics. NES [Game]. Kyoto, Japan: Nintendo.
Obrist, M., Gatti, E., Maggioni, E., Vi, C.T. and Velasco, C. (2017) Multisensory experiences in HCI. IEEE MultiMedia, 24(2). 9-13. https://doi.org/10.1109/MMUL.2017.33
Paper Folds - Origami and Crafts ! (2015) Easy Origami – Paper Heart - A4 sheet !!. Available from: https://www.youtube.com/watch?v=jizmI8xUA6Q (28 June 2023)
Patrick, J.H., Pruchno, R.A. and Rose, M.S. (1998) Recruiting research participants: a comparison of the costs and effectiveness of five recruitment strategies. The Gerontologist, 38(3). 295-302. https://doi.org/10.1093/geront/38.3.295
Pavanatto, L., Lu, F., Davari, S., Harris, E., Folino, A., Imamov, S., Chekuri, S., Blustein, L., Lages, W.S. and Bowman, D.A. (2020) Get the job! An immersive simulation of sensory overload. In: 2020 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW). Atlanta, GA, USA, 22-26 March 2020. New York, USA: IEEE. 509-510. https://doi.org/10.1109/VRW50115.2020.00106
Pei, S. Chen, A., Lee, J. and Zhang, Y. (2022) Hand Interfaces: Using Hands to Imitate Objects in AR/VR for Expressive Interactions. In: CHI Conference on Human Factors in Computing Systems. New Orleans, LA, USA, 29 April-5 May 2022. New York, USA: ACM. 1-16. https://doi.org/10.1145/3491102.3501898
Perfetti Van Melle. Strawberry Cream | Chupa Chups (n.d.) https://www.chupachups.com.au/products/best/strawberry-cream. Accessed 16 January 2023
Purcell H (1694) Pini, C., Tunnell, J., Pini, A. and Lester, H. Distressed Innocence, Z.577. Available from: https://musopen.org/music/45438- distressed-innocence-z577/ (28 June 2023)
Raafat, O. and Becattini, N. (2022) Testing the Usability of Guidelines for the Design of Surprising Products. Proceedings of the Design Society, 2, 931-940. https://doi.org/10.1017/pds.2022.95
Ranasinghe, N., Jain, P., Tram, N.T.N., Koh, K.C.R., Tolley, D., Karwita, S., Lien-Ya. L., Liangkun, Y., Shamaiah, K., Tung, C.E.W., Yen, C.C. and Do, E.Y.L. (2018) Season traveller: Multisensory narration for enhancing the virtual reality experience. In: Proceedings of the 2018 CHI conference on human factors in computing systems. Montreal, QC, Canada, 21-26 April 2018. New York, USA: ACM. 1-13. https://doi.org/10.1145/3173574.3174151
Rao, H.V. and O'Modhrain, S. (2020) 2Across: A Comparison of Audio-Tactile and Screen-Reader Based Representations of a Crossword Puzzle. In: Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. Honolulu, HI, USA, 25-30 April 2020. New York, USA: ACM. 1-12. https://doi.org/10.1145/3313831.3376207
Riva, G., Mantovani, F., Capideville, C.S., Preziosa, A., Morganti, F., Villani, D., Gaggioli, A., Botella, C. and Alcañiz, M. (2007) Affective interactions using virtual reality: the link between presence and emotions. CyberPsychology & Behavior 10(1). 45-56. https://doi.org/10.1089/cpb.2006.9993
Rudenko, S. and Cabral, J.P. (2019) Synaesthesia: How can it be used to enhance the audio-visual perception of music and multisensory design in digitally enhanced environments?. In: Proceedings of SMM19, Workshop on Speech, Music and Mind 2019. Vienna, Austria, 14 September. 60-64. https://doi.org/10.21437/SMM.2019-13
Schatter, G., Züger, E. and Nitschke, C. (2005) A Synaesthetic Approach for a synthesizer Interface Based on Genetic Algorithms and fuzzy Sets. In: Proceedings of the International Computer Music Conference, Barcelona, Spain, 4-10 September. Michigan, USA: Michigan Publishing. 664-667.
Schnädelbach, H., Coughlan, T., Kefalidou, G., McAuley, D. and Meese, R. (2015) Creativity Bento Box: a physical resource pack to support interaction in virtual space. International Journal of Human-Computer Interaction, 31(11). 790-804. https://doi.org/10.1080/10447318.2015.1067478
Slater, M., Usoh, M. and Steed, A. (1994) Depth of presence in virtual environments. Presence: Teleoperators & Virtual Environments, 3(2). 130-144. https://doi.org/10.1162/pres.1994.3.2.130
Spence, C. (2015) Leading the consumer by the nose: on the commercialization of olfactory design for the food and beverage sector. Flavour, 4(1). 1-15. https://doi.org/10.1186/s13411-015-0041-1
Spence, C. and Youssef, J. (2019) Synaesthesia: The multisensory dining experience. International Journal of Gastronomy and Food Science, 18. 100179. https://doi.org/10.1016/j.ijgfs.2019.100179
stephan (n.d.) Sunny Day. Available from: https://soundbible.com/1661-Sunny-Day.html (28 June 2023)
Swain, F. (2014) Can synaesthesia be learnt?. Available from: https://www.bbc.com/future/article/20140611-can-synaesthesia-be-learnt (28 June 2023)
Tan, C.T., Foo, L.C., Yeo, A., Lee, J.S.A., Wan, E., Kok, X.F.K. and Rajendran, M. (2022) Understanding User Experiences Across VR Walking-in-place Locomotion Methods. In: CHI Conference on Human Factors in Computing Systems, New Orleans, LA, USA, 29 April-5 May 2022. New York, USA: ACM. 1-13. https://doi.org/10.1145/3491102.3501975
Telemann, G.P. (1735) Pitman, P. Trio III in G Major for Flute, Violin, Cello and Harpsichord - III. Largo. Available from: https://musopen.org/music/43840-trio-iii-in-g-major-for-flute-violin-cello-and-harpsichord/ (28 June 2023)
The VOID (2023) The Void. Available from: https://www.thevoid.com/ (28 June 2023)
Unity (2023) Unity Real-Time Development Platform | 3D, 2D VR & AR Engine. Available from: https://unity.com/ (28 June 2023)
Vi, C.T., Ablart, D., Gatti, E., Velasco, C. and Obrist, M. (2017) Not just seeing, but also feeling art: Mid-air haptic experiences integrated in a multisensory art exhibition. International Journal of Human-Computer Studies, 108. 1-14. https://doi.org/10.1016/j.ijhcs.2017.06.004
Voong, M. and Beale, R. (2007) Music organisation using colour synaesthesia. In: CHI'07 extended abstracts on Human Factors in Computing Systems, San Jose, CA, USA, 28 April-3 May 2007. New York, USA: ACM. 1869-1874. https://doi.org/10.1145/1240866.1240913
Ward, J. and Wright, T. (2014) Sensory substitution as an artificially acquired synaesthesia. Neuroscience & Biobehavioral Reviews, 41. 26-35. https://doi.org/10.1016/j.neubiorev.2012.07.007
Waterworth, J.A. (1997) Creativity and sensation: The case for synaesthetic media. Leonardo, 30(4). 327-330. https://muse.jhu.edu/article/607487
Waterzooi, T. (2022) Please, Touch The Artwork. PC, Android, Nintendo Switch [Game]. Thomas Waterzooi and Nakana.io.
Weinel, J (2021) Synaesthetic audio-visual sound toys in Virtual Reality. In: AM ’21: Audio Mostly 2021. Virtual/Trento, Italy, 1-3 September 2021. New York, USA: ACM. 135-138. https://doi.org/10.1145/3478384.3478400
Wilson, C (2014) People taught synaesthesia learn to read in color. Available from: https://www.newscientist.com/article/dn26578-people-taught-synaesthesia-learn-to-read-in-colour/ (28 June 2023)
Witmer, B.G. and Singer, M.J. (1998) Measuring presence in virtual environments: A presence questionnaire. Presence 7(3). 225-240. https://doi.org/10.1162/105474698565686