What You Hear is Where You Are is What I Hear:

Aiming at providing data points for a framework of adaptive audio implementations, this paper draws on previous results obtained from an ongoing series of experiments in ludomusicology. From discussing findings on enhanced user interaction in video games through the use of adaptive music, further insights are offered into optimising immersive experience by means of adjusted temporal-structural and expressive attributes according to interindividual differences in the cognitive-emotional processing of music. In addition to outlining those implications for the design of a user-aware audio engine, the presented study takes into account situational percepts and context effects as opposed to treating situation as a mere external stimulus. Not only the cognitive-emotional processing of music, but also situation awareness and tendency for immersive experience, has been shown to vary along different manifestations of personality traits. Accordingly, self-report measures of spatial presence and emotional response showcased correlations with trait empathising-systemising and trait absorption. Differential effects on those dimensions appeared depending on the use of situation-adaptive implementations of low and high arousing music stimuli in an action-adventure 3D video game. These findings suggest interindividual differences in the sub-attentional processing of music found in genre video games and are discussed with regards to the potential of psychologically informed sound design to affect a users’ mental model of situational context and experience of agency.

Content

Author and article information

Contributors

Hans-Peter Gasselseder

Conference

Publication date: July 2016

Publication date (Print): July 2016

Pages: 61-69

Affiliations

[1 ] Aalborg University

Communication and Psychology

Musikkens Plads 1, 9000 Aalborg

Denmark

Article

DOI: 10.14236/ewic/EVA2016.12

SO-VID: da1a787c-80ad-4dcd-a810-00fadcc13b48

License:

This work is licensed under a Creative Commons Attribution 4.0 Unported License. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/

Conference name: Electronic Visualisation and the Arts

Conference acronym: EVA

Conference number:

Conference location: London, UK

Conference date: 12 - 14 July 2016

Conference sponsor: Electronic Workshops in Computing (eWiC)

Conference theme: Electronic Visualisation and the Arts

History

Product

1477-9358 BCS Learning & Development

Self URI (article page): https://www.scienceopen.com/hosted-document?doi=10.14236/ewic/EVA2016.12

Self URI (journal page): https://ewic.bcs.org/

REFERENCES

S. Baron-CohenR. C. Knick MeyerM. K. Belmonte 2005 Sex Differences in the Brain. Implications for Explaining Autism Science 310 5749 819 823
M. G. BoltzB. EbendorfB. Field 2009 Audiovisual interactions. The impact of visual information on music perception and memory Music Perception 27 1 43 59
J. Bruner 1986 Actual Minds, Possible Worlds. Harvard University Press Cambridge, MA
J. BrunerL. Postman 1949 On the perception of incongruity. A paradigm Journal of Personality 18 206 223
A. J. Cohen 2001 Music as a source of emotion in film P. JuslinJ. Sloboda Music and Emotion Oxford University Press Oxford, UK
M. Csikszentmihalyi 1991 Flow The Psychology of Optimal Experience Harper and Row New York
D. DatcuS. G. LukoschH. Lukosch 2013 Comparing presence, workload and situational awareness in a collaborative real world and augmented reality scenario IEEE ISMAR 2013 Workshop on Collaboration in Merging Realities (CiMeR’13), Adelaide, SA, Australia 1 October ext-link-type="uri" xlink:href="http://cimer13.participatorysystems.org/opencms/export/sites/default/cimer13/papers/datcu_et_al.pdf">http://cimer13.participatorysystems.org/opencms/export/sites/default/cimer13/papers/datcu_et_al.pdf 2 April 2016
M. H. Davis 1980 A multidimensional approach to individual differences in empathy JSAS Catalog of Selected Documents in Psychology 10 85
J. DecetyJ. Grèzes 2006 The power of simulation Brain Research 1079 4 14
J.V. DraperD.B. KaberJ.M. Usher 1998 Telepresence Human Factors 4 3 354 375
L. ErmiF. Mäyrä 2005 Fundamental Components of the Gameplay Experience. Analysing Immersion S. de CastellJ. Jenson Changing Views - Worlds in Play. ext-link-type="uri" xlink:href="http://www.uta.fi/~tlilma/gameplay_experience.pdf">http://www.uta.fi/~tlilma/gameplay_experience.pdf 2 April 2016
S. GarridoE. Schubert 2011 Individual differences in the enjoyment of negative emotion Music Perception 28 279 296
H.-P. Gasselseder 2012 Re-Orchestrating Game Drama. Diploma Thesis at the Department of Psychology, Paris-Lodron University of Salzburg Austria
W. W. GaverG. Mandler 1987 Play it again, Sam. On liking music Cognition and Emotion 1 259 282
J. G. Greeno 1994 Gibson’s Affordances Psychological Review 101 2 336 342
G. D. Jacobucci 1998 Primary Appraisal as a Function of Attachment Pattern, Personality, and Situational Circumstances PhD thesis at the Department of Psychology, Simon Fraser University Canada
M. Jeannerod 2008 The sense of agency and its disturbances in schizophrenia. A reappraisal Experimental Brain Research 192 3 527 532
G. KreutzE. SchubertL. A. Mitchell 2008 Cognitive Styles of Music Listening Music Perception 26 1 57 73
S. LiljeströmP. N. JuslinD. Västfjäll 2012 Experimental evidence of the roles of music choice, social context, and listener personality in emotional reactions to music Psychology of Music 41 5 579 599
A. Maasø 2000 Synchronisieren ist unnorwegisch (Dubbing is atypical Norwegian) Montage AV 9 1 147 171
T. C. MurthaR. ConferP. L. Ackerman 1996 Toward an interactionist taxonomy of personality and situations Journal of Personality and Social Psychology 71 1 193 207
F. NagelR. KopiezO. GreweE. Altenmüller 2007 EMuJoy Behavior Research Methods 39 2 283 290
K. PetriniF. CrabbeC. SheridanF. E. Pollick 2011 The Music of Your Emotions PLoS ONE 6 4 ext-link-type="uri" xlink:href="http://dx.plos.org/10.1371/journal.pone.0019165.pdf">http://dx.plos.org/10.1371/journal.pone.0019165.pdf 2 April 2016
A. N. PopperR. R. Fay 1997 Evolution of the ear and hearing. Issues and questions Brain, Behavior and Evolution 50 4 213 220
D. RichardsE. FassbenderA. BilginW. F. Thompson 2008 An investigation of the role of background music in IVWs for learning. ALT-J Research in Learning Technology 16 3 231 244
J. M. RileyD. B. KaberJ. V. Draper 2003 Situation awareness and attention allocation measures for quantifying telepresence experiences in teleoperation Human Factors and Ergonomics in Manufacturing 14 1 51 67
G. RobillardS. BouchardT. FournierP. Renaud 2003 Anxiety and Presence during VR Immersion CyberPsychology & Behavior 6 5 467 476
Rocksteady (Developer) 2011 Batman. Arkham City. Warner Bros Interactive, Burbank, CA, USA
G. M. SandstromF. A. Russo 2011 Absorption in music Psychology of Music 11 1 13
E.-R. SausB. H. JohnsenJ. EidJ. F. Thayer 2012 Who benefits from simulator training Computers in Human Behavior 28 4 1262 1268
S. G. Shamay-TsooryJ. Aharon-PeretzD. Perry 2009 Two systems for empathy Brain 132 3 617 627
M. Slater 2003 A note on presence terminology Presence Connect 3 1 5
C. SpenceJ. Driver 1997 Audiovisual links in exogenous covert spatial attention Perception & Psychophysics 59 1 22
P. Vorderer et al 2004 Development of the MEC Spatial Presence Questionnaire, MEC-SPQ. Report for the European Comission, IST Programme ‘Presence Research Activites’
D. WeibelB. WissmathF. W. Mast 2010 Immersion in mediated environments. The role of personality traits Cyberpsychology, Behavior, and Social Networking 13 3 251 256
W. Wirth et al 2007 A Process Model of the Formation of Spatial Presence Experiences Media Psychology 9 493 525
J. YangA. McClellandA. Furnham 2015 The effect of background music on the cognitive performance of musicians Psychology of Music. ext-link-type="uri" xlink:href="http://pom.sagepub.com/content/early/2015/07/07Z0305735615592265.abstract">http://pom.sagepub.com/content/early/2015/07/07Z0305735615592265.abstract 2 April 2016
R. A. Zwaan 1999 Situation models. The mental leap into imagined worlds Current Directions in Psychological Science 8 15 18

Comments

Comment on this article

[1] S. Baron-CohenR. C. Knick MeyerM. K. Belmonte 2005 Sex Differences in the Brain. Implications for Explaining Autism Science 310 5749 819 823

[2] M. G. BoltzB. EbendorfB. Field 2009 Audiovisual interactions. The impact of visual information on music perception and memory Music Perception 27 1 43 59

[3] J. Bruner 1986 Actual Minds, Possible Worlds. Harvard University Press Cambridge, MA

[4] J. BrunerL. Postman 1949 On the perception of incongruity. A paradigm Journal of Personality 18 206 223

[5] A. J. Cohen 2001 Music as a source of emotion in film P. JuslinJ. Sloboda Music and Emotion Oxford University Press Oxford, UK

[6] M. Csikszentmihalyi 1991 Flow The Psychology of Optimal Experience Harper and Row New York

[7] D. DatcuS. G. LukoschH. Lukosch 2013 Comparing presence, workload and situational awareness in a collaborative real world and augmented reality scenario IEEE ISMAR 2013 Workshop on Collaboration in Merging Realities (CiMeR’13), Adelaide, SA, Australia 1 October ext-link-type="uri" xlink:href="http://cimer13.participatorysystems.org/opencms/export/sites/default/cimer13/papers/datcu_et_al.pdf">http://cimer13.participatorysystems.org/opencms/export/sites/default/cimer13/papers/datcu_et_al.pdf 2 April 2016

[8] M. H. Davis 1980 A multidimensional approach to individual differences in empathy JSAS Catalog of Selected Documents in Psychology 10 85

[9] J. DecetyJ. Grèzes 2006 The power of simulation Brain Research 1079 4 14

[10] J.V. DraperD.B. KaberJ.M. Usher 1998 Telepresence Human Factors 4 3 354 375

[11] L. ErmiF. Mäyrä 2005 Fundamental Components of the Gameplay Experience. Analysing Immersion S. de CastellJ. Jenson Changing Views - Worlds in Play. ext-link-type="uri" xlink:href="http://www.uta.fi/~tlilma/gameplay_experience.pdf">http://www.uta.fi/~tlilma/gameplay_experience.pdf 2 April 2016

[12] S. GarridoE. Schubert 2011 Individual differences in the enjoyment of negative emotion Music Perception 28 279 296

[13] H.-P. Gasselseder 2012 Re-Orchestrating Game Drama. Diploma Thesis at the Department of Psychology, Paris-Lodron University of Salzburg Austria

[14] W. W. GaverG. Mandler 1987 Play it again, Sam. On liking music Cognition and Emotion 1 259 282

[15] J. G. Greeno 1994 Gibson’s Affordances Psychological Review 101 2 336 342

[16] G. D. Jacobucci 1998 Primary Appraisal as a Function of Attachment Pattern, Personality, and Situational Circumstances PhD thesis at the Department of Psychology, Simon Fraser University Canada

[17] M. Jeannerod 2008 The sense of agency and its disturbances in schizophrenia. A reappraisal Experimental Brain Research 192 3 527 532

[18] G. KreutzE. SchubertL. A. Mitchell 2008 Cognitive Styles of Music Listening Music Perception 26 1 57 73

[19] S. LiljeströmP. N. JuslinD. Västfjäll 2012 Experimental evidence of the roles of music choice, social context, and listener personality in emotional reactions to music Psychology of Music 41 5 579 599

[20] A. Maasø 2000 Synchronisieren ist unnorwegisch (Dubbing is atypical Norwegian) Montage AV 9 1 147 171

[21] T. C. MurthaR. ConferP. L. Ackerman 1996 Toward an interactionist taxonomy of personality and situations Journal of Personality and Social Psychology 71 1 193 207

[22] F. NagelR. KopiezO. GreweE. Altenmüller 2007 EMuJoy Behavior Research Methods 39 2 283 290

[23] K. PetriniF. CrabbeC. SheridanF. E. Pollick 2011 The Music of Your Emotions PLoS ONE 6 4 ext-link-type="uri" xlink:href="http://dx.plos.org/10.1371/journal.pone.0019165.pdf">http://dx.plos.org/10.1371/journal.pone.0019165.pdf 2 April 2016

[24] A. N. PopperR. R. Fay 1997 Evolution of the ear and hearing. Issues and questions Brain, Behavior and Evolution 50 4 213 220

[25] D. RichardsE. FassbenderA. BilginW. F. Thompson 2008 An investigation of the role of background music in IVWs for learning. ALT-J Research in Learning Technology 16 3 231 244

[26] J. M. RileyD. B. KaberJ. V. Draper 2003 Situation awareness and attention allocation measures for quantifying telepresence experiences in teleoperation Human Factors and Ergonomics in Manufacturing 14 1 51 67

[27] G. RobillardS. BouchardT. FournierP. Renaud 2003 Anxiety and Presence during VR Immersion CyberPsychology & Behavior 6 5 467 476

[28] Rocksteady (Developer) 2011 Batman. Arkham City. Warner Bros Interactive, Burbank, CA, USA

[29] G. M. SandstromF. A. Russo 2011 Absorption in music Psychology of Music 11 1 13

[30] E.-R. SausB. H. JohnsenJ. EidJ. F. Thayer 2012 Who benefits from simulator training Computers in Human Behavior 28 4 1262 1268

[31] S. G. Shamay-TsooryJ. Aharon-PeretzD. Perry 2009 Two systems for empathy Brain 132 3 617 627

[32] M. Slater 2003 A note on presence terminology Presence Connect 3 1 5

[33] C. SpenceJ. Driver 1997 Audiovisual links in exogenous covert spatial attention Perception & Psychophysics 59 1 22

[34] P. Vorderer et al 2004 Development of the MEC Spatial Presence Questionnaire, MEC-SPQ. Report for the European Comission, IST Programme ‘Presence Research Activites’

[35] D. WeibelB. WissmathF. W. Mast 2010 Immersion in mediated environments. The role of personality traits Cyberpsychology, Behavior, and Social Networking 13 3 251 256

[36] W. Wirth et al 2007 A Process Model of the Formation of Spatial Presence Experiences Media Psychology 9 493 525

[37] J. YangA. McClellandA. Furnham 2015 The effect of background music on the cognitive performance of musicians Psychology of Music. ext-link-type="uri" xlink:href="http://pom.sagepub.com/content/early/2015/07/07Z0305735615592265.abstract">http://pom.sagepub.com/content/early/2015/07/07Z0305735615592265.abstract 2 April 2016

[38] R. A. Zwaan 1999 Situation models. The mental leap into imagined worlds Current Directions in Psychological Science 8 15 18

Celebrating 65 years of The Computer Journal - free-to-read perspectives - bcs.org/tcj65

What You Hear is Where You Are is What I Hear: : Optimising Immersive Experiences for Interindividual Differences