A Method Impact Assessment Framework for User Experience Evaluations with Children

Based upon a review of the literature, this paper presents a Method Impact Assessment Framework. Theoretically synthesized, the framework offers five dimensions: (1) the role of the child, (2) the user experience construct, (3) system, (4) epistemological perspective, (5) Practical and Ethical Concerns. Although other dimensions could have been construed, these were judged to be the most pertinent to understanding evaluation methods with children. The framework thus provides a critical lens in which evaluation methods can be assessed by the Children Computer Interaction (CCI) Community to inform method selection.

Content

Author and article information

Contributors

Gavin Sim

Bieke Zaman

Matthew Horton

Conference

Publication date: July 2017

Publication date (Print): July 2017

Pages: 1-9

Affiliations

[0001]University of Central Lancashire

Preston, UK.

[0002]KU Leuven

Leuven, Belgium.

Article

DOI: 10.14236/ewic/HCI2017.82

SO-VID: f2b240bb-65ee-4bc6-81d7-86c0688f3f55

License:

This work is licensed under a Creative Commons Attribution 4.0 Unported License. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/

Conference name: Proceedings of the 31st International BCS Human Computer Interaction Conference (HCI 2017)

Conference acronym: HCI

Conference number: 31

Conference location: Sunderland, UK

Conference date: 3 - 6 July 2017

Conference sponsor: Electronic Workshops in Computing (eWiC)

Conference theme: digital make-believe, with delegates considering our expansive

History

Product

1477-9358 BCS Learning & Development

Self URI (article page): https://www.scienceopen.com/hosted-document?doi=10.14236/ewic/HCI2017.82

Self URI (journal page): https://ewic.bcs.org/

reference

O. Akobo WodikeG. SimM. Horton 2014 Empowering Teenagers to Perform a Heuristic Evaluation of a Game 28th British HCI Conference. Southport, UK BCS
C. AngelíJ. AlexanderJ. A. Hunt 2014 ‘Draw, write and tell’: A literature review and methodological development on the ‘draw and write’research method Journal of Early Childhood Research 1476718X14538592
A. N. Antle 2008 Child-based personas: need, ability and experience Cognition, Technology & Work 10 155 166
A. N. AntleG. CornessA. Bevans 2011 Springboard: Designing image schema based embodied interaction for an abstract domain Whole Body Interaction Springer
J. Bardzell 2013 Critical and Cultural Approaches to HCI The SAGE Handbook of Digital Technology Research 130
J. BardzellS. Bardzell Interaction criticism: a proposal and framework for a new discipline of hci CHI'08 Extended Abstracts on Human Factors in Computing Systems 2008 ACM 2463 2472
W. BarendregtM. BekkerE. Baauw 2008 Development and evaluation of the problem identification picture cards method Cognition, Technology and Work 10 95 105
W. BarendregtM. BekkerD. G. BouwhuisE. Baauw 2006 Identifying usability and fun problems in a computer game during first use and some practice International Journal Human Computer Interaction 64 830 846
J. A. Bargas-AvilaK. Hornbaek 2011 Old wine in new bottles or novel challenges: a critical analysis of empirical studies of user experience Proceedings of the SIGCHI Conference on Human Factors in Computing Systems Vancouver, BC, Canada ACM
E. BauuwM. M. BekkerP. Markopoulos 2006 Assessing the applicability of the structured expert evaluation method (SEEM) for a wider age group Interaction Design and Children Tampere ACM
A. Bell 2007 Designing and testing questionnaires for children Journal of Research in Nursing 12 461 469
Bera 2011 Ethical Guidelines for Educational Research 22/07/15
C. Burke 2008 Play in focus
R. CordovilJ. Barreiros 2010 Adults' perception of children's height and reaching capability Acta psychologica 135 24 29
S. DavisK. NesbittE. Nalivaiko A Systematic Review of Cybersickness. Proceedings of the 2014 Conference on Interactive Entertainment 2014 ACM 1 9
P. DesmetP. Hekkert 2007 Framework of Product Experience International Journal of Design 1 57 66
A. Druin 1999 The design of children's technology Morgan Kaufmann Publishers San Francisco
J. Forlizzi 2008 The product ecology: Understanding social product use and supporting design culture International Journal of design 2 11 20
J. ForlizziK. Battarbee 2004 Understanding experience in interactive systems Proceedings of the 5th conference on Designing interactive systems: processes, practices, methods, and techniques. Cambridge, MA, USA ACM
L.-A. GallacherM. Gallacher 2008 Methodological Immaturity in Childhood Research? Thinking through 'participatory methods’ Childhood 15 499 516
D. Gauntlett 2007 Creative explorations: New approaches to identities and audiences Routledge
D. Gauntlett 2011 Creative Explorations: New Approaches to Identities and Audiences. London Routledge
J. J. Gibson 1979 The Ecological Approach to visual Perception Boston Houghton Miffflin
L. HannaK. RisdenK. J. Alexander 1997 Guidelines for usability testing with children Interactions 4 9 14
R. A. Hart 1992 Children's participation: From tokenism to citizenship UNICEF Innocenti Research Centre
H. R. HartsonT. S. AndreR. C. Williges 2003 Criteria for Evaluating Usability Evaluation Methods International Journal Human Computer Interaction 15 145 181
M. Hassenzahl 2005 The thing and I: understanding the relationship between user and product Funology. Springer
M. HassenzahlS. DiefenbachA. Goritz 2010 Needs, affects, and interactive products - facets of user experience Interacting with Computers 22 353 362
S. N. Hesse-BiberP. Leavy 2010 Handbook of Emergent Methods New York Guilford Press
M. Hill 2006 Children's Voices on Ways of Having a Voice Children's and young people's perspectives on methods used in reserach and consultation Childhood 13 69 89
K. Inkpen 1997 Three Important Research Agendas for Educational Multimedia: Learning, Children and Gender Educationi MultiMedia 97 Calgary
Iso 2010 Ergonomics of human system interaction - Prt 210: Human-centred design for interactive system Switserland International Standards Organisation
D. KahnemanE. DienerN. Schwarz 1999 Well-being: Foundations of hedonic psychology Russell Sage Foundation
M. KellettR. ForrestN. DentS. Ward 2004 "Just Teach us the Skills Please, We'll Do the Rest": Empowering Ten-Year-Olds as Researchers Children and Society 18 329 343
S. KujalaV. RotoK. Väänänen-Vainio-MattilaE. KarapanosA. Sinnelä 2011 UX Curve: A method for evaluating long-term user experience Interacting with Computers 23 473 483
G. Lansdown 2001 Promoting children's participation in democratic decision-making UNICEF Innocenti Research Centre
E. L.-C. LawV. RotoM. HassenzahlA. P. O. S. VermeerenJ. Kort 2009 Understanding, scoping and defining user experience: a survey approach Proceedings of the SIGCHI Conference on Human Factors in Computing Systems Boston, MA, USA ACM
E. L.-C. LawV. RotoA. P. O. S. VermeerenJ. KortM. Hassenzahl 2008 Towards a Shared Definition of User Experience CHI 2008. Florence ACM Press
T. Lyon 2002 Child witnesses and the oath In: H. L. WESTCOTTG. M. DAVIESR. H. C. BULL Children's testimony: A handbook of psychological research and forensic practice Chichester Wiley
T. W. Malone 1984 Heuristics for designing enjoyable user interfaces : Lessons from computer games Norwood, N .J Ablex
P. MarkopoulosM. Bekker 2003 On assessment of usability testing methods for children Interacting with Computers 15 227 243
L. McknightJ. C. Read 2011 PLU-E: a proposed framework for planning and conducting evaluation studies with children Proceedings of the 25th BCS Conference on Human-Computer Interaction Newcastle-upon-Tyne, United Kingdom British Computer Society
H. Mclaughlin 2006 Involving Young Service Usres as Co-Reserachers: Possibilities, Benefits and Costs British Journal of Social Work 36 1395 1410
A. G. MirnigA. MeschtscherjakovD. WurhoferT. MenewegerM. Tscheligi 2015 A Formal Analysis of the ISO 9241-210 Definition of User Experience Proceedings of the 33rd Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems Seoul, Republic of Korea ACM
V. MorrowM. Richards 1996 The ethics of social research with children: An overview Children & Society 10 90 105
H. Niesyto 2000 Youth research on video selfproductions reflections on a social- aesthetic approach Visual Studies 15 135 153
D. A. Norman 2004 Emotional design: Why we love (or hate) everyday things Basic books
M. NouwenM. V. MechelenB. Zaman 2015 A value sensitive design approach to parental software for young children Proceedings of the 14th International Conference on Interaction Design and Children. Boston, Massachusetts ACM
F. PucilloG. Cascini 2014 A framework for user experience, needs and affordances Design Studies 35 160 179
S. Punch 2002 RESEARCH WITH CHILDREN The same or different from research with adults? Childhood 9 321 341
J. C. Read 2012 Evaluating artefacts with children: age and technology effects in the reporting of expected and experienced fun 14th ACM International Conference on Multimodal Interaction Santa Monica ACM
J. C. ReadM. HortonG. SimP. GregoryD. FittonB. Cassidy 2013 CHECk: a tool to inform and encourage ethical practice in participatory design with children CHI '13 Extended Abstracts on Human Factors in Computing Systems Paris, France ACM
J. C. ReadS. Macfarlane 2006 Using the Fun Toolkit and Other Survey Methods to Gather Opinions in Child Computer Interaction Interaction Design and Children Tampere ACM Press
J. C. ReadS. J. MacfarlaneC. Endurability Casey Engagement and Expectations: Measuring Children's Fun Interaction Design and Children, 2002 Eindhoven, The Netherlands Shaker Publishing 189 198
J. A. Russell 2003 Core Affect and the Psychological Construction of Emotion Psychological Review 110 145 172
K. SchulzeH. Kromker 2010 A framework to measure user experience of interactive online products Proceedings of the 7th International Conference on Methods and Techniques in Behavioral Research Eindhoven, The Netherlands ACM
G. SimM. Horton 2012 Investigating children's opinions of games: Fun Toolkit vs This or That Interaction Design and Children Bremen, Germany ACM
G. SimS. MacfarlaneM. Horton 2005 Evaluating usability, fun and learning in educational software for children World Conference on Educational Multimedia, Hypermedia and Telecommunications Montreal AACE
Unicef 1989 UN Convention on the Rights of the child [Online] Available: http://www.unicef.org/crc/ 02/07/15 2015
J. VissersL. D. BotB. Zaman 2013 MemoLine: evaluating long-term UX with children Proceedings of the 12th International Conference on Interaction Design and Children New York, New York ACM
H. L. WestcottK. S. Littleton 2005 Exploring Meaning in Interviews with Children In: S. GREENED. HOGAN Reseraching Children's Experience London Sage
P. WrightJ. Mccarthy 2004 Technology as experience MIT Press
M. Wyness 2013 Children’s participation and intergenerational dialogue: Bringing adults back into the analysis Childhood 20 429 442
B. Zaman Introduction and validation of a pairwise comparison scale for UX evaluations and benchmarking with preschoolers Interact 2009 Uppsala Springer 634 637
B. ZamanV. V. Abeele 2010 Laddering with Young Children in User Experence Evaluations: Theoretical Groundings and a Practical Case IDC Barcelona ACM
B. ZamanV. V. AbeeleD. De Grooff 2013 Measuring product liking in preschool children: An evaluation of the Smileyometer and This or That methods International Journal of ChildComputer interaction 1 61 70

Comments

Comment on this article

[1] O. Akobo WodikeG. SimM. Horton 2014 Empowering Teenagers to Perform a Heuristic Evaluation of a Game 28th British HCI Conference. Southport, UK BCS

[2] C. AngelíJ. AlexanderJ. A. Hunt 2014 ‘Draw, write and tell’: A literature review and methodological development on the ‘draw and write’research method Journal of Early Childhood Research 1476718X14538592

[3] A. N. Antle 2008 Child-based personas: need, ability and experience Cognition, Technology & Work 10 155 166

[4] A. N. AntleG. CornessA. Bevans 2011 Springboard: Designing image schema based embodied interaction for an abstract domain Whole Body Interaction Springer

[5] J. Bardzell 2013 Critical and Cultural Approaches to HCI The SAGE Handbook of Digital Technology Research 130

[6] J. BardzellS. Bardzell Interaction criticism: a proposal and framework for a new discipline of hci CHI'08 Extended Abstracts on Human Factors in Computing Systems 2008 ACM 2463 2472

[7] W. BarendregtM. BekkerE. Baauw 2008 Development and evaluation of the problem identification picture cards method Cognition, Technology and Work 10 95 105

[8] W. BarendregtM. BekkerD. G. BouwhuisE. Baauw 2006 Identifying usability and fun problems in a computer game during first use and some practice International Journal Human Computer Interaction 64 830 846

[9] J. A. Bargas-AvilaK. Hornbaek 2011 Old wine in new bottles or novel challenges: a critical analysis of empirical studies of user experience Proceedings of the SIGCHI Conference on Human Factors in Computing Systems Vancouver, BC, Canada ACM

[10] E. BauuwM. M. BekkerP. Markopoulos 2006 Assessing the applicability of the structured expert evaluation method (SEEM) for a wider age group Interaction Design and Children Tampere ACM

[11] A. Bell 2007 Designing and testing questionnaires for children Journal of Research in Nursing 12 461 469

[12] Bera 2011 Ethical Guidelines for Educational Research 22/07/15

[13] C. Burke 2008 Play in focus

[14] R. CordovilJ. Barreiros 2010 Adults' perception of children's height and reaching capability Acta psychologica 135 24 29

[15] S. DavisK. NesbittE. Nalivaiko A Systematic Review of Cybersickness. Proceedings of the 2014 Conference on Interactive Entertainment 2014 ACM 1 9

[16] P. DesmetP. Hekkert 2007 Framework of Product Experience International Journal of Design 1 57 66

[17] A. Druin 1999 The design of children's technology Morgan Kaufmann Publishers San Francisco

[18] J. Forlizzi 2008 The product ecology: Understanding social product use and supporting design culture International Journal of design 2 11 20

[19] J. ForlizziK. Battarbee 2004 Understanding experience in interactive systems Proceedings of the 5th conference on Designing interactive systems: processes, practices, methods, and techniques. Cambridge, MA, USA ACM

[20] L.-A. GallacherM. Gallacher 2008 Methodological Immaturity in Childhood Research? Thinking through 'participatory methods’ Childhood 15 499 516

[21] D. Gauntlett 2007 Creative explorations: New approaches to identities and audiences Routledge

[22] D. Gauntlett 2011 Creative Explorations: New Approaches to Identities and Audiences. London Routledge

[23] J. J. Gibson 1979 The Ecological Approach to visual Perception Boston Houghton Miffflin

[24] L. HannaK. RisdenK. J. Alexander 1997 Guidelines for usability testing with children Interactions 4 9 14

[25] R. A. Hart 1992 Children's participation: From tokenism to citizenship UNICEF Innocenti Research Centre

[26] H. R. HartsonT. S. AndreR. C. Williges 2003 Criteria for Evaluating Usability Evaluation Methods International Journal Human Computer Interaction 15 145 181

[27] M. Hassenzahl 2005 The thing and I: understanding the relationship between user and product Funology. Springer

[28] M. HassenzahlS. DiefenbachA. Goritz 2010 Needs, affects, and interactive products - facets of user experience Interacting with Computers 22 353 362

[29] S. N. Hesse-BiberP. Leavy 2010 Handbook of Emergent Methods New York Guilford Press

[30] M. Hill 2006 Children's Voices on Ways of Having a Voice Children's and young people's perspectives on methods used in reserach and consultation Childhood 13 69 89

[31] K. Inkpen 1997 Three Important Research Agendas for Educational Multimedia: Learning, Children and Gender Educationi MultiMedia 97 Calgary

[32] Iso 2010 Ergonomics of human system interaction - Prt 210: Human-centred design for interactive system Switserland International Standards Organisation

[33] D. KahnemanE. DienerN. Schwarz 1999 Well-being: Foundations of hedonic psychology Russell Sage Foundation

[34] M. KellettR. ForrestN. DentS. Ward 2004 "Just Teach us the Skills Please, We'll Do the Rest": Empowering Ten-Year-Olds as Researchers Children and Society 18 329 343

[35] S. KujalaV. RotoK. Väänänen-Vainio-MattilaE. KarapanosA. Sinnelä 2011 UX Curve: A method for evaluating long-term user experience Interacting with Computers 23 473 483

[36] G. Lansdown 2001 Promoting children's participation in democratic decision-making UNICEF Innocenti Research Centre

[37] E. L.-C. LawV. RotoM. HassenzahlA. P. O. S. VermeerenJ. Kort 2009 Understanding, scoping and defining user experience: a survey approach Proceedings of the SIGCHI Conference on Human Factors in Computing Systems Boston, MA, USA ACM

[38] E. L.-C. LawV. RotoA. P. O. S. VermeerenJ. KortM. Hassenzahl 2008 Towards a Shared Definition of User Experience CHI 2008. Florence ACM Press

[39] T. Lyon 2002 Child witnesses and the oath In: H. L. WESTCOTTG. M. DAVIESR. H. C. BULL Children's testimony: A handbook of psychological research and forensic practice Chichester Wiley

[40] T. W. Malone 1984 Heuristics for designing enjoyable user interfaces : Lessons from computer games Norwood, N .J Ablex

[41] P. MarkopoulosM. Bekker 2003 On assessment of usability testing methods for children Interacting with Computers 15 227 243

[42] L. McknightJ. C. Read 2011 PLU-E: a proposed framework for planning and conducting evaluation studies with children Proceedings of the 25th BCS Conference on Human-Computer Interaction Newcastle-upon-Tyne, United Kingdom British Computer Society

[43] H. Mclaughlin 2006 Involving Young Service Usres as Co-Reserachers: Possibilities, Benefits and Costs British Journal of Social Work 36 1395 1410

[44] A. G. MirnigA. MeschtscherjakovD. WurhoferT. MenewegerM. Tscheligi 2015 A Formal Analysis of the ISO 9241-210 Definition of User Experience Proceedings of the 33rd Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems Seoul, Republic of Korea ACM

[45] V. MorrowM. Richards 1996 The ethics of social research with children: An overview Children & Society 10 90 105

[46] H. Niesyto 2000 Youth research on video selfproductions reflections on a social- aesthetic approach Visual Studies 15 135 153

[47] D. A. Norman 2004 Emotional design: Why we love (or hate) everyday things Basic books

[48] M. NouwenM. V. MechelenB. Zaman 2015 A value sensitive design approach to parental software for young children Proceedings of the 14th International Conference on Interaction Design and Children. Boston, Massachusetts ACM

[49] F. PucilloG. Cascini 2014 A framework for user experience, needs and affordances Design Studies 35 160 179

[50] S. Punch 2002 RESEARCH WITH CHILDREN The same or different from research with adults? Childhood 9 321 341

[51] J. C. Read 2012 Evaluating artefacts with children: age and technology effects in the reporting of expected and experienced fun 14th ACM International Conference on Multimodal Interaction Santa Monica ACM

[52] J. C. ReadM. HortonG. SimP. GregoryD. FittonB. Cassidy 2013 CHECk: a tool to inform and encourage ethical practice in participatory design with children CHI '13 Extended Abstracts on Human Factors in Computing Systems Paris, France ACM

[53] J. C. ReadS. Macfarlane 2006 Using the Fun Toolkit and Other Survey Methods to Gather Opinions in Child Computer Interaction Interaction Design and Children Tampere ACM Press

[54] J. C. ReadS. J. MacfarlaneC. Endurability Casey Engagement and Expectations: Measuring Children's Fun Interaction Design and Children, 2002 Eindhoven, The Netherlands Shaker Publishing 189 198

[55] J. A. Russell 2003 Core Affect and the Psychological Construction of Emotion Psychological Review 110 145 172

[56] K. SchulzeH. Kromker 2010 A framework to measure user experience of interactive online products Proceedings of the 7th International Conference on Methods and Techniques in Behavioral Research Eindhoven, The Netherlands ACM

[57] G. SimM. Horton 2012 Investigating children's opinions of games: Fun Toolkit vs This or That Interaction Design and Children Bremen, Germany ACM

[58] G. SimS. MacfarlaneM. Horton 2005 Evaluating usability, fun and learning in educational software for children World Conference on Educational Multimedia, Hypermedia and Telecommunications Montreal AACE

[59] Unicef 1989 UN Convention on the Rights of the child [Online] Available: http://www.unicef.org/crc/ 02/07/15 2015

[60] J. VissersL. D. BotB. Zaman 2013 MemoLine: evaluating long-term UX with children Proceedings of the 12th International Conference on Interaction Design and Children New York, New York ACM

[61] H. L. WestcottK. S. Littleton 2005 Exploring Meaning in Interviews with Children In: S. GREENED. HOGAN Reseraching Children's Experience London Sage

[62] P. WrightJ. Mccarthy 2004 Technology as experience MIT Press

[63] M. Wyness 2013 Children’s participation and intergenerational dialogue: Bringing adults back into the analysis Childhood 20 429 442

[64] B. Zaman Introduction and validation of a pairwise comparison scale for UX evaluations and benchmarking with preschoolers Interact 2009 Uppsala Springer 634 637

[65] B. ZamanV. V. Abeele 2010 Laddering with Young Children in User Experence Evaluations: Theoretical Groundings and a Practical Case IDC Barcelona ACM

[66] B. ZamanV. V. AbeeleD. De Grooff 2013 Measuring product liking in preschool children: An evaluation of the Smileyometer and This or That methods International Journal of ChildComputer interaction 1 61 70

Teaching business & IT courses? Request a digital desk copy of any BCS Publishing title via the BCS Bookshop Inspection Copy Request form.