+1 Recommend
1 collections
      • Record: found
      • Abstract: found
      • Conference Proceedings: found
      Is Open Access

      Measuring User Experience in Conversational Interfaces: A Comparison of Six Questionnaires


      1 , 1 ,   1

      Proceedings of the 32nd International BCS Human Computer Interaction Conference (HCI)

      Human Computer Interaction Conference

      4 - 6 July 2018

      User experience, conversational agents, dialogue systems, voice interfaces, evaluation, standardised questionnaires



            User experience (UX) has become an important aspect in the evaluation of interactive systems. In parallel, conversational interfaces have been increasingly used in many work and everyday settings. Although there have been various methods developed to evaluate conversational interfaces, there has been a lack of methods specifically focusing on evaluating user experience. This study reviews the six main questionnaires for evaluating conversational systems in order to assess the potential suitability of these questionnaires to measure various UX dimensions. We found that (i) four questionnaires included assessment items, in varying extents, to measure hedonic, aesthetic and pragmatic dimensions of UX; (ii) two questionnaires assessed affect, and one assessed frustration dimension; and, (iii) enchantment, playfulness and motivation dimensions have not been covered sufficiently by any questionnaires. We recommend using multiple questionnaires to obtain a more complete measurement of user experience or improve the assessment of a particular UX dimension.


            Author and article information

            July 2018
            July 2018
            : 1-12
            [1 ] Australian Institute of Health

            Innovation, Macquarie University
            © Kocaballi et al. Published by BCS Learning and Development Ltd. Proceedings of British HCI 2018. Belfast, UK.

            This work is licensed under a Creative Commons Attribution 4.0 Unported License. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/

            Proceedings of the 32nd International BCS Human Computer Interaction Conference
            Belfast, UK
            4 - 6 July 2018
            Electronic Workshops in Computing (eWiC)
            Human Computer Interaction Conference
            Product Information: 1477-9358BCS Learning & Development
            Self URI (journal page): https://ewic.bcs.org/
            Electronic Workshops in Computing
            1. 2011 Old wine in new bottles or novel challenges: a critical analysis of empirical studies of user experience. In Proc. Conference on Human Factors in Computing Systems ACM

            2. 2005 Co-experience: user experience as interaction. CoDesign 1 1 5 18

            3. 1999 Current Practice in the Development and Evaluation of Spoken Language Dialogue Systems In Sixth European Conference on Speech Communication and Technology

            4. 2013 Giving voice to enterprise mobile applications In Proceedings of the 15th international conference on Human-computer interaction with mobile devices and services 428 433 ACM

            5. 2006 Interdisciplinary criticism: analysing the experience of riot! a location-sensitive digital narrative Behaviour & Information Technology 25 2 127 139

            6. 1999 Survey of existing interactive systems. Trindi (Task Oriented Instructional Dialogue) report, (D1) 3

            7. 1994 Measuring emotion: the self-assessment manikin and the semantic differential. Journal of behavior therapy and experimental psychiatry 25 1 49 59

            8. 1996 SUS-A quick and dirty usability scale. Usability evaluation in industry 189 194 4 7

            9. 2010 Social robots and ECAs for accessing smart environments services In Proceedings of the International Conference on Advanced Visual Interfaces 275 278 ACM

            10. 2014 SimSensei Kiosk: A virtual human interviewer for healthcare decision support In Proceedings of the 2014 international conference on Autonomous agents and multi-agent systems 1061 1068

            11. 2004 Evaluation and usability of multimodal spoken language dialogue systems. Speech Communication 43 1-2 33 54

            12. 1992 The adult playfulness scale: An initial assessment. Psychological reports 71 1 83 103

            13. 2011 August User experience in speech recognition of navigation devices: an assessment In Proceedings of the 13th International Conference on Human Computer Interaction with Mobile Devices and Services 517 520 ACM

            14. 2001 The effect of perceived hedonic quality on product appealingness. International Journal of Human-Computer Interaction 13 4 481 499

            15. 2003 The thing and I: understanding the relationship between user and product In Funology 31 42 Springer Netherlands

            16. 2003 AttrakDiff: A questionnaire to measure perceived hedonic and pragmatic quality In Mensch & Computer 187 196

            17. 2008 How motivational orientation influences the evaluation and choice of hedonic and pragmatic interactive products: The role of regulatory focus. Interacting with Computers 20 4-5 473 479

            18. 2010 The inference of perceived usability from beauty. Human–Computer Interaction 25 3 235 260

            19. 1995 Content validity in psychological assessment: A functional approach to concepts and methods Psychological assessment 7 3 238

            20. 2000 Towards a tool for the subjective assessment of speech system interfaces (SASSI). Natural Language Engineering 6 3-4 287 303

            21. 2001 Designing habitable dialogues for speech-based interaction with computers. International Journal of Human-Computer Studies 54 4 637 662

            22. 2001 Subjective assessment of speech-system interface usability In Seventh European Conference on Speech Communication and Technology

            23. 2013 September Mach: My automated conversation coach In Proceedings of the 2013 ACM international joint conference on Pervasive and ubiquitous computing 697 706 ACM

            24. 2006 Current practice in measuring usability: Challenges to usability studies and research. International journal of human-computer studies 64 2 79 102

            25. 2007 Meta-analysis of correlations among usability measures In Proceedings of the SIGCHI conference on Human factors in computing systems 617 626 ACM

            26. 2007 Disabled persons as lead users in the domestic environment In Proceedings of the 2007 conference on Designing pleasurable products and interfaces 158 167 ACM

            27. 1996 Development and validation of a scale to measure optimal experience: The Flow State Scale. Journal of sport and exercise psychology 18 1 17 35

            28. 1934 Art as experience New York Minton, Balch, and Company

            29. 2012 Quantifying quality aspects of multimodal interactive systems Springer Science & Business Media

            30. 2003 Assessment of spoken dialogue system usability-what are we really measuring? In Eighth European Conference on Speech Communication and Technology

            31. 2004 Assessing dimensions of perceived visual aesthetics of web sites. International journal of human-computer studies 60 3 269 298

            32. C., 2009 Understanding, scoping and defining user experience: a survey approach In Proc. Conference on Human Factors in Computing systems 719 728 ACM

            33. 2017 Enhancing user experience with conversational agent for movie recommendation: Effects of self-disclosure and reciprocity. International Journal of Human-Computer Studies 103 95 105

            34. 2016 Standardized Questionnaires for Voice Interaction Design. Voice Interaction Design 1 1

            35. 2015 Investigating the psychometric properties of the Speech User Interface Service Quality questionnaire International Journal of Speech Technology 18 3 479 487

            36. 2004 What should we do about motivation theory? Six recommendations for the twenty-first century. Academy of management review 29 3 388 403

            37. 2011 Enhancement of Conversational Agents by Means of Multimodal Interaction. Conversational Agents and Natural Language Interaction: Techniques and Effective Practices: Techniques and Effective Practices 223 252

            38. 2016 Like having a really bad PA: the gulf between user expectation and experience of conversational agents In Proc. Conference on Human Factors in Computing systems 5286 5297

            39. 1989 Psychometric properties of the Intrinsic Motivation Inventory in a competitive sport setting: A confirmatory factor analysis. Research quarterly for exercise and sport 60 1 48 58

            40. 2006 The experience of enchantment in human–computer interaction. Personal and ubiquitous computing 10 6 369 378

            41. 2016 The conversational interface Springer

            42. 1974 An approach to environmental psychology the MIT Press

            43. 2008 What is user engagement? A conceptual framework for defining user engagement with technology. Journal of the Association for Information Science and Technology 59 6 938 955

            44. 2005 Toward a social-cognitive psychology of speech technology: Affective responses to speech-based e-service. Unpublished doctoral dissertation. University of South Florida

            45. 2008 Machines as mediators: The challenge of technology for interpersonal communication theory and research (Ed.) Mediated interpersonal communication 34 57 New York Routledge

            46. 2003 Expanding the MOS: Development and psychometric evaluation of the MOS-R and MOS-X. International Journal of Speech Technology 6 2 161 182

            47. 2018 Voice Interfaces in Everyday Life In Proceedings of the CHI Conference on Human Factors in Computing Systems (CHI’18)

            48. 2014 The five dimensions of usability In Content and complexity 93 114 Routledge

            49. 2009 Correlations among prototypical usability metrics: evidence for the construct of usability In Proc. Conference on Human Factors in Computing systems 1609 1618 ACM

            50. 1995 Intelligibility and acceptability testing for speech technology (Eds.) Applied Speech Technology Boca Raton CRC Press

            51. 2009 September User experience of speech-controlled media center for physically disabled users In Proceedings of the 13th International MindTrek Conference: Everyday Life in the Ubiquitous Era 2 5 ACM

            52. ISO 9241-210:2010 Ergonomics of human-system interaction — Part 210: Human-centred design for interactive systems

            53. 2010 Design and evaluation of a multimodal interface for in-car communication systems In Proceedings of the 2010 Annual Research Conference of the South African Institute of Computer Scientists and Information Technologists 314 321 ACM

            54. 2009 SUXES-user experience evaluation method for spoken and multimodal interaction In Tenth Annual Conference of the International Speech Communication Association

            55. 2017 Aesthetics In A Psychology of User Experience 109 130 Springer

            56. 1993 The dimensionality and correlates of flow in human-computer interactions. Computers in human behavior 9 4 411 426

            57. 2014 An evaluation framework for multimodal interaction. Springer International doi, 10 978 3

            58. 2012 Measuring the Quality of Service and Quality of Experience of multimodal human–machine interaction. Journal on Multimodal User Interfaces 6 1-2 73 85

            59. 2008 Evaluation methods for multimodal systems: A comparison of standardized usability questionnaires In International Tutorial and Research Workshop on Perception and Interactive Technologies for Speech-Based Systems 276 284 Springer Berlin, Heidelberg

            60. 2014 Hands free-care free: elderly people taking advantage of speech-only interaction In Proceedings of the 8th Nordic Conference on Human-Computer Interaction: Fun, Fast, Foundational 203 206 ACM

            61. 2013 Designing engagement-aware agents for multiparty conversations In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems 2233 2242 ACM


            Comment on this article