Measuring User Experience in Conversational Interfaces: A Comparison of Six Questionnaires

User experience (UX) has become an important aspect in the evaluation of interactive systems. In parallel, conversational interfaces have been increasingly used in many work and everyday settings. Although there have been various methods developed to evaluate conversational interfaces, there has been a lack of methods specifically focusing on evaluating user experience. This study reviews the six main questionnaires for evaluating conversational systems in order to assess the potential suitability of these questionnaires to measure various UX dimensions. We found that (i) four questionnaires included assessment items, in varying extents, to measure hedonic, aesthetic and pragmatic dimensions of UX; (ii) two questionnaires assessed affect, and one assessed frustration dimension; and, (iii) enchantment, playfulness and motivation dimensions have not been covered sufficiently by any questionnaires. We recommend using multiple questionnaires to obtain a more complete measurement of user experience or improve the assessment of a particular UX dimension.

Content

Author and article information

Contributors

A. Baki Kocabalil

Liliana Laranjo

Enrico Coiera

Conference

Publication date: July 2018

Publication date (Print): July 2018

Pages: 1-12

Affiliations

[1 ] Australian Institute of Health

Innovation, Macquarie University

Article

DOI: 10.14236/ewic/HCI2018.21

SO-VID: 79db6ce4-7488-4de5-8fae-201702d73fbb

Copyright statement: © Kocaballi et al. Published by BCS Learning and Development Ltd. Proceedings of British HCI 2018. Belfast, UK.

License:

This work is licensed under a Creative Commons Attribution 4.0 Unported License. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/

Conference name: Proceedings of the 32nd International BCS Human Computer Interaction Conference

Conference acronym: HCI

Conference number: 32

Conference location: Belfast, UK

Conference date: 4 - 6 July 2018

Conference sponsor: Electronic Workshops in Computing (eWiC)

Conference theme: Human Computer Interaction Conference

Product

Product Information: 1477-9358BCS Learning & Development

Self URI (article page): https://www.scienceopen.com/hosted-document?doi=10.14236/ewic/HCI2018.21

Self URI (journal page): https://ewic.bcs.org/

Comments

Comment on this article

[1] J.A. Bargas-AvilaK. Hornbæk 2011 Old wine in new bottles or novel challenges: a critical analysis of empirical studies of user experience. In Proc. Conference on Human Factors in Computing Systems ACM

[2] K. BattarbeeI. Koskinen 2005 Co-experience: user experience as interaction. CoDesign 1 1 5 18

[3] N.O. BernsenL. DybkjærU. Heid 1999 Current Practice in the Development and Evaluation of Spoken Language Dialogue Systems In Sixth European Conference on Speech Communication and Technology

[4] C. BijaniB.K. WhiteM. Vilrokx 2013 Giving voice to enterprise mobile applications In Proceedings of the 15th international conference on Human-computer interaction with mobile devices and services 428 433 ACM

[5] M. BlytheJ. ReidP. WrightE. Geelhoed 2006 Interdisciplinary criticism: analysing the experience of riot! a location-sensitive digital narrative Behaviour & Information Technology 25 2 127 139

[6] J. BosS. LarssonI. LewinC. MathesonD. Milward 1999 Survey of existing interactive systems. Trindi (Task Oriented Instructional Dialogue) report, (D1) 3

[7] M.M. BradleyP.J. Lang 1994 Measuring emotion: the self-assessment manikin and the semantic differential. Journal of behavior therapy and experimental psychiatry 25 1 49 59

[8] J. Brooke 1996 SUS-A quick and dirty usability scale. Usability evaluation in industry 189 194 4 7

[9] B. De CarolisI. MazzottaN. NovielliS. Pizzutilo 2010 Social robots and ECAs for accessing smart environments services In Proceedings of the International Conference on Advanced Visual Interfaces 275 278 ACM

[10] D. DeVaultR. ArtsteinG. BennT. DeyE. FastA. GainerK. GeorgilaJ. GratchA. HartholtM. LhommetG. Lucas 2014 SimSensei Kiosk: A virtual human interviewer for healthcare decision support In Proceedings of the 2014 international conference on Autonomous agents and multi-agent systems 1061 1068

[11] L. DybkjaerN.O. BernsenW. Minker 2004 Evaluation and usability of multimodal spoken language dialogue systems. Speech Communication 43 1-2 33 54

[12] M.A. GlynnJ. Webster 1992 The adult playfulness scale: An initial assessment. Psychological reports 71 1 83 103

[13] A. GoulatiD. Szostak 2011 August User experience in speech recognition of navigation devices: an assessment In Proceedings of the 13th International Conference on Human Computer Interaction with Mobile Devices and Services 517 520 ACM

[14] M. Hassenzahl 2001 The effect of perceived hedonic quality on product appealingness. International Journal of Human-Computer Interaction 13 4 481 499

[15] M. Hassenzahl 2003 The thing and I: understanding the relationship between user and product In Funology 31 42 Springer Netherlands

[16] M. HassenzahlM. BurmesterF. Koller 2003 AttrakDiff: A questionnaire to measure perceived hedonic and pragmatic quality In Mensch & Computer 187 196

[17] M. HassenzahlM. SchöbelT. Trautmann 2008 How motivational orientation influences the evaluation and choice of hedonic and pragmatic interactive products: The role of regulatory focus. Interacting with Computers 20 4-5 473 479

[18] M. HassenzahlA. Monk 2010 The inference of perceived usability from beauty. Human–Computer Interaction 25 3 235 260

[19] S. N. HaynesD. RichardE. S. Kubany 1995 Content validity in psychological assessment: A functional approach to concepts and methods Psychological assessment 7 3 238

[20] K.S. HoneR. Graham 2000 Towards a tool for the subjective assessment of speech system interfaces (SASSI). Natural Language Engineering 6 3-4 287 303

[21] K.S. HoneC. Baber 2001 Designing habitable dialogues for speech-based interaction with computers. International Journal of Human-Computer Studies 54 4 637 662

[22] K.S. HoneR. Graham 2001 Subjective assessment of speech-system interface usability In Seventh European Conference on Speech Communication and Technology

[23] M.E. HoqueM. CourgeonJ.C. MartinB. MutluR.W. Picard 2013 September Mach: My automated conversation coach In Proceedings of the 2013 ACM international joint conference on Pervasive and ubiquitous computing 697 706 ACM

[24] K. Hornbæk 2006 Current practice in measuring usability: Challenges to usability studies and research. International journal of human-computer studies 64 2 79 102

[25] K. HornbækE.L.C. Law 2007 Meta-analysis of correlations among usability measures In Proceedings of the SIGCHI conference on Human factors in computing systems 617 626 ACM

[26] S. JacobsonA. Pirinen 2007 Disabled persons as lead users in the domestic environment In Proceedings of the 2007 conference on Designing pleasurable products and interfaces 158 167 ACM

[27] S.A. JacksonH.W. Marsh 1996 Development and validation of a scale to measure optimal experience: The Flow State Scale. Journal of sport and exercise psychology 18 1 17 35

[28] D. John 1934 Art as experience New York Minton, Balch, and Company

[29] C. Kühnel 2012 Quantifying quality aspects of multimodal interactive systems Springer Science & Business Media

[30] L.B. Larsen 2003 Assessment of spoken dialogue system usability-what are we really measuring? In Eighth European Conference on Speech Communication and Technology

[31] T. LavieN. Tractinsky 2004 Assessing dimensions of perceived visual aesthetics of web sites. International journal of human-computer studies 60 3 269 298

[32] E.L. LawC., V. RotoM. HassenzahlA.P. VermeerenJ. Kort 2009 Understanding, scoping and defining user experience: a survey approach In Proc. Conference on Human Factors in Computing systems 719 728 ACM

[33] S. LeeJ. Choi 2017 Enhancing user experience with conversational agent for movie recommendation: Effects of self-disclosure and reciprocity. International Journal of Human-Computer Studies 103 95 105

[34] J.R. Lewis 2016 Standardized Questionnaires for Voice Interaction Design. Voice Interaction Design 1 1

[35] J. R. LewisM. L. Hardzinski 2015 Investigating the psychometric properties of the Speech User Interface Service Quality questionnaire International Journal of Speech Technology 18 3 479 487

[36] E.A. LockeG.P. Latham 2004 What should we do about motivation theory? Six recommendations for the twenty-first century. Academy of management review 29 3 388 403

[37] R. López-CózarZ. CallejasG. EspejoD. Griol 2011 Enhancement of Conversational Agents by Means of Multimodal Interaction. Conversational Agents and Natural Language Interaction: Techniques and Effective Practices: Techniques and Effective Practices 223 252

[38] E. LugerA. Sellen 2016 Like having a really bad PA: the gulf between user expectation and experience of conversational agents In Proc. Conference on Human Factors in Computing systems 5286 5297

[39] E. McAuleyT. DuncanV.V. Tammen 1989 Psychometric properties of the Intrinsic Motivation Inventory in a competitive sport setting: A confirmatory factor analysis. Research quarterly for exercise and sport 60 1 48 58

[40] J. McCarthyP. WrightJ. WallaceA. Dearden 2006 The experience of enchantment in human–computer interaction. Personal and ubiquitous computing 10 6 369 378

[41] M. McTearZ. CallejasD. Griol 2016 The conversational interface Springer

[42] A. MehrabianJ. A. Russell 1974 An approach to environmental psychology the MIT Press

[43] H.L. O’BrienE.G. Toms 2008 What is user engagement? A conceptual framework for defining user engagement with technology. Journal of the Association for Information Science and Technology 59 6 938 955

[44] M. D. Polkosky 2005 Toward a social-cognitive psychology of speech technology: Affective responses to speech-based e-service. Unpublished doctoral dissertation. University of South Florida

[45] M. D. Polkosky 2008 Machines as mediators: The challenge of technology for interpersonal communication theory and research E. Konjin (Ed.) Mediated interpersonal communication 34 57 New York Routledge

[46] M.D. PolkoskyJ.R. Lewis 2003 Expanding the MOS: Development and psychometric evaluation of the MOS-R and MOS-X. International Journal of Speech Technology 6 2 161 182

[47] M. PorcheronJ.E. FischerS. ReevesS. Sharples 2018 Voice Interfaces in Everyday Life In Proceedings of the CHI Conference on Human Factors in Computing Systems (CHI’18)

[48] W. Quesenbery 2014 The five dimensions of usability In Content and complexity 93 114 Routledge

[49] J. SauroJ.R. Lewis 2009 Correlations among prototypical usability metrics: evidence for the construct of usability In Proc. Conference on Human Factors in Computing systems 1609 1618 ACM

[50] A. Schmidt-Nielsen 1995 Intelligibility and acceptability testing for speech technology A. SyrdalR. BennettS. Greenspan (Eds.) Applied Speech Technology Boca Raton CRC Press

[51] H. SoronenS. PakarinenM. HansenM. TurunenJ. HakulinenJ. HellaJ.P. RajaniemiA. MeltoT. Laivo 2009 September User experience of speech-controlled media center for physically disabled users In Proceedings of the 13th International MindTrek Conference: Everyday Life in the Ubiquitous Era 2 5 ACM

[52] ISO 9241-210:2010 Ergonomics of human-system interaction — Part 210: Human-centred design for interactive systems

[53] P. TchankueD. VogtsJ. Wesson 2010 Design and evaluation of a multimodal interface for in-car communication systems In Proceedings of the 2010 Annual Research Conference of the South African Institute of Computer Scientists and Information Technologists 314 321 ACM

[54] M. TurunenJ. HakulinenA. MeltoT. HeimonenT. LaivoJ. Hella 2009 SUXES-user experience evaluation method for spoken and multimodal interaction In Tenth Annual Conference of the International Speech Communication Association

[55] P. Turner 2017 Aesthetics In A Psychology of User Experience 109 130 Springer

[56] J. WebsterL.K. TrevinoL. Ryan 1993 The dimensionality and correlates of flow in human-computer interactions. Computers in human behavior 9 4 411 426

[57] I. Wechsung 2014 An evaluation framework for multimodal interaction. Springer International doi, 10 978 3

[58] I. WechsungK.P. EngelbrechtC. KühnelS. MöllerB. Weiss 2012 Measuring the Quality of Service and Quality of Experience of multimodal human–machine interaction. Journal on Multimodal User Interfaces 6 1-2 73 85

[59] I. WechsungA.B. Naumann 2008 Evaluation methods for multimodal systems: A comparison of standardized usability questionnaires In International Tutorial and Research Workshop on Perception and Interactive Technologies for Speech-Based Systems 276 284 Springer Berlin, Heidelberg

[60] L. WulfM. GarschallJ. HimmelsbachM. Tscheligi 2014 Hands free-care free: elderly people taking advantage of speech-only interaction In Proceedings of the 8th Nordic Conference on Human-Computer Interaction: Fun, Fast, Foundational 203 206 ACM

[61] Q. XuL. LiG. Wang 2013 Designing engagement-aware agents for multiparty conversations In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems 2233 2242 ACM