User-based Evaluation of Gamification Elements in an Educational Application

Andrade, Pamela V; Law, Effie Lai-Chong

doi:10.14236/ewic/HCI2018.27

Abstract

Gamification has increasingly been utilised to motivate students to use technological tools and applications to learn. Although the existing research seems to recognise the benefits that gamification has in and outside classroom, little has been explored about individual gamification elements. To bridge this gap, we conducted a user-based evaluation of an educational application, Knowma+, with different gamification elements to understand which are more desirable to use from the teacher and student perspectives. Knowma+ is built upon relevant pedagogical concepts such as learning by questioning that fit well with the inclusion of gamification elements to enhance learning experience. Results of eye-tracking data suggest that more visually attractive gamification elements could capture the initial attention of both teacher and student participants. However, there were slight differences between the two groups in self-reported preferences with regard to the perceived usefulness of each gamification element as part of their teaching or learning approaches.

Content

Author and article information

Contributors

Pamela Andrade

Effie Lai-Chong Law

Conference

Publication date: July 2018

Publication date (Print): July 2018

Pages: 1-13

Affiliations

[1 ] University of Leicester

University Road, Leicester

LE1 7RH, United Kingdom

Article

DOI: 10.14236/ewic/HCI2018.27

SO-VID: c94ac84f-3c9b-412c-96cd-10359305beb7

Copyright statement: © Andrade et al. Published by BCS Learning and Development Ltd. Proceedings of British HCI 2018. Belfast, UK.

License:

This work is licensed under a Creative Commons Attribution 4.0 Unported License. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/

Conference name: Proceedings of the 32nd International BCS Human Computer Interaction Conference

Conference acronym: HCI

Conference number: 32

Conference location: Belfast, UK

Conference date: 4 - 6 July 2018

Conference sponsor: Electronic Workshops in Computing (eWiC)

Conference theme: Human Computer Interaction Conference

Product

Product Information: 1477-9358BCS Learning & Development

Self URI (article page): https://www.scienceopen.com/hosted-document?doi=10.14236/ewic/HCI2018.27

Self URI (journal page): https://ewic.bcs.org/

Categories

Series title: Electronic Workshops in Computing

A. AndersonD. HuttenlocherJ. KleinbergJ. Leskovec 2013 May Steering user behavior with badges In Proceedings of the 22nd international conference on World Wide Web 95 106 ACM
S. ArnabT. LimM. B. CarvalhoF. BellottiS. FreitasS. LouchartA. De Gloria 2015 Mapping learning and game mechanics for serious games analysis British Journal of Educational Technology 46 2 391 411
Y. AttaliM. Arieli-Attali 2015 Gamification in assessment: Do points affect test performance?. Computers & Education 83 57 63
J. BanfieldB. Wilkerson 2014 Increasing student intrinsic motivation and self-efficacy through gamification pedagogy Contemporary Issues in Education Research (Online) 7 4 291
K. BeckerS. Nicholson 2016 Gamification in the Classroom: Old Wine in New Badges Learning, Education and Games 61 67
Y. Beldarrain 2006 Distance Education Trends: Integrating new technologies to foster student interaction and collaboration Distance Education 27 2 139 153
H. R. BernardA. WutichG. W. Ryan 2016 Analyzing qualitative data: Systematic approaches SAGE publications
P. BlackD. Wiliam 2010 Inside the Black Box: Raising Standards through Classroom Assessment Phi Delta Kappan 92 1 81 90
J. Brooke 1996 SUS-A quick and dirty usability scale Usability evaluation in industry 189 194 4 7
A. BruunJ. Stage 2015 September An empirical study of the effects of three think-aloud protocols on identification of usability problems In Human-Computer Interaction 159 176 Springer Cham
D. Buehl 2013 Classroom strategies for interactive learning Newark, USA International Reading Association
I. Bunchball 2010 Gamification 101: An introduction to the use of game dynamics to influence behavior. White paper, 9
N. Burbules 2018 Watch IT: The risks and promises of information technologies for education. Routledge
C. CheongJ. FilippouF. Cheong 2014 Towards the gamification of learning: Investigating student perceptions of game elements Journal of Information Systems Education 25 3 233
A. Colburn 2006 What teacher educators need to know about inquiry-based instruction. Paper presented at the annual meeting of the Association for Education of Teachers in Science, Akron, OH
E. L. DeciR. M. Ryan 2010 Self‐determination John Wiley & Sons, Inc.
A. E. DeppingR. L. Mandryk 2017 May Why is This Happening to Me?: How Player Attribution can Broaden our Understanding of Player Experience In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems 1040 1052 ACM
S. Deterding 2012 Gamification: designing for motivation interactions 19 4 14 17
S. DeterdingD. DixonR. KhaledL. Nacke 2011 September From game design elements to gamefulness: defining gamification In Proceedings of the 15th international academic MindTrek conference: Envisioning future media environments 9 15 ACM
D. DichevaC. DichevG. AgreG. Angelova 2015 Gamification in education: a systematic mapping study Journal of Educational Technology & Society 18 3 75
K. EcclestoneJ. DaviesJ. DerrickJ. Gawn 2010 Transforming formative assessment in lifelong learning Berkshire, England Open University Press
C. EhmkeS. Wilson 2007 September Identifying web usability problems from eye-tracking data In Proceedings of the 21st British HCI Group Annual Conference on People and Computers: HCI... but not as we know it-Volume 1 119 128 British Computer Society
C. Erneling 2010 Towards discursive education New York Cambridge University Press
D. GoshevskiJ. VeljanoskaT. Hatziapostolou 2017 September A Review of Gamification Platforms for Higher Education In Proceedings of the 8th Balkan Conference in Informatics 28 ACM
J. HamariJ. KoivistoH. Sarsa 2014 January Does gamification work?--a literature review of empirical studies on gamification In System Sciences (HICSS), 2014 47th Hawaii International Conference on 3025 3034 IEEE
S. HeintzE. L. C. Law 2015 Game Elements-Attributes Model: a First Step towards a Structured Comparison of Educational Games. In DiGRA Conference
A. IosupD. Epema 2014 March An experience report on using gamification in technical higher education In Proceedings of the 45th ACM technical symposium on Computer science education 27 32 ACM
R. Koster 2005 A theory of fun for game design. Scottsdale, AZ: Paraglyph Press
R. N. LandersK. N. BauerR. C. Callan 2017 Gamification of task performance with leaderboards: A goal setting experiment Computers in Human Behavior 71 508 515
C. LehmannZ. Chase 2015 Building school 2.0 John Wiley & Sons
L. de-MarcosE. García-LópezA. García-CabotJ. A. Medina-MerodioA. DomínguezJ. J. Martínez-HerráizT. Diez-Folledo 2016 Social network analysis of a gamified e-learning course: Small-world phenomenon and network metrics as predictors of academic performance Computers in Human Behavior 60 312 321
E. MargolisL. Pauwels 2011 The Sage handbook of visual research methods. Sage. 433 435
E. D. MeklerF. BrühlmannA. N. TuchK. Opwis 2017 Towards understanding the effects of individual gamification elements on intrinsic motivation and performance Computers in Human Behavior 71 525 534
E. D. MeklerF. BrühlmannK. OpwisA. N. Tuch 2013 October Do points, levels and leaderboards harm intrinsic motivation?: an empirical analysis of common gamification elements In Proceedings of the First International Conference on gameful design, research, and applications 66 73 ACM
C. I. Muntean 2011 October Raising engagement in e-learning through gamification In Proc. 6th International Conference on Virtual Learning ICVL (Vol. 1). sn
L. E. NackeS. Deterding 2017 The maturing of gamification research Computers in Human Behavior 71 450 454
F. F. H. NahV. R. TelaproluS. RallapalliP. R. Venkata 2013 July Gamification of education using computer games In International Conference on Human Interface and the Management of Information 99 107 Springer Berlin, Heidelberg
S. Nicholson 2015 A recipe for meaningful gamification In Gamification in education and business 1 20 Springer Cham
J. Nielsen 1994 April Enhancing the explanatory power of usability heuristics In Proceedings of the SIGCHI conference on Human Factors in Computing Systems 152 158 ACM
H. O’BrienP. CairnsM. Hall 2018 A Practical Approach to Measuring User Engagement with the Refined User Engagement Scale (UES) and New UES Short Form. International Journal of Human-Computer Studies 112 28 39
J. L. OrquinN. J. AshbyA. D. Clarke 2016 Areas of interest as a signal detection problem in behavioral eye‐tracking research. Journal of Behavioral Decision Making 29 2-3 103 115
R. W. Picard 2000 Toward computers that recognize and respond to user emotion. IBM systems journal 39 3.4 705 719
A. PooleL. J. Ball 2006 Eye tracking in HCI and usability research. Encyclopedia of human computer interaction, 1 211 219
V. Rebolj 2009 E-education between Pedagogical and Didactic Theory and Practice. Organizacija 42 1
C. RichardsC. W. ThompsonN. Graham 2014 October Beyond designing for motivation: the importance of context in gamification. In Proceedings of the first ACM SIGCHI annual symposium on Computer-human interaction in play 217 226 ACM
K. RobsonK. PlanggerJ. KietzmannI. McCarthyL. Pitt 2015 Is it all a game? Understanding the principles of gamification Business Horizons 58 4 411 420
S. RothD. SchneckenbergC. W. Tsai 2015 The ludic drive as innovation driver: Introduction to the gamification of innovation Creativity and Innovation Management 24 2 300 306
M. SailerJ. U. HenseS. K. MayrH. Mandl 2017 How gamification motivates: An experimental study of the effects of specific game design elements on psychological need satisfaction Computers in Human Behavior 69 371 380
N. Seel 2012 Encyclopedia of the sciences of learning. Germany: Springer
D. W. Shaffer 2006 How computer games help children learn. Macmillan
N. Shin 2006 Online learner’s ‘flow’experience: an empirical study. British Journal of Educational Technology 37 5 705 720
J. SimõesR. D. RedondoA. F. Vilas 2013 A social gamification framework for a K-6 learning platform Computers in Human Behavior 29 2 345 353
S. de Sousa BorgesV. H. DurelliH. M. ReisS. Isotani 2014 March A systematic mapping on gamification applied to education In Proceedings of the 29th Annual ACM Symposium on Applied Computing 216 222 ACM
J. M. Spector 2000 System dynamics and interactive learning environments: Lessons learned and implications for the future. Simulation & Gaming 31 4 528 535
A. TashakkoriC. Teddlie (Eds.) 2010 Sage handbook of mixed methods in social & behavioral research (pp 305-306). Sage
G. F. TondelloH. PremsukhL. E. Nacke 2018 A Theory of Gamification Principles Through Goal-Setting Theory. In Proceedings of the 51st Hawaii International Conference on System Sciences (HICSS
P. Tynjälä 1999 Towards expert knowledge? A comparison between a constructivist and a traditional learning environment in the university. International journal of educational research 31 5 357 442
H. WangC. T. Sun 2011 September Game reward systems: Gaming experiences and social meanings In DiGRA Conference
K. WerbachD. Hunter 2012 For the win: How game thinking can revolutionize your business. Wharton Digital Press
D. West 2012 Digital schools. Washington D.C.: Brookings Institution Press
C. Williams 2011 Research methods. Journal of Business & Economics Research (JBER) 5 3
J. P. ZagalM. MateasC. Fernández-VaraB. HochhalterN. Lichti 2007 Towards an ontological language for game analysis. Worlds in Play: International Perspectives on Digital Games Research, 21, 21
G. ZichermannC. Cunningham 2011 Gamification by design: Implementing game mechanics in web and mobile apps. O’Reilly Media, Inc

Comments

[1] A. AndersonD. HuttenlocherJ. KleinbergJ. Leskovec 2013 May Steering user behavior with badges In Proceedings of the 22nd international conference on World Wide Web 95 106 ACM

[2] S. ArnabT. LimM. B. CarvalhoF. BellottiS. FreitasS. LouchartA. De Gloria 2015 Mapping learning and game mechanics for serious games analysis British Journal of Educational Technology 46 2 391 411

[3] Y. AttaliM. Arieli-Attali 2015 Gamification in assessment: Do points affect test performance?. Computers & Education 83 57 63

[4] J. BanfieldB. Wilkerson 2014 Increasing student intrinsic motivation and self-efficacy through gamification pedagogy Contemporary Issues in Education Research (Online) 7 4 291

[5] K. BeckerS. Nicholson 2016 Gamification in the Classroom: Old Wine in New Badges Learning, Education and Games 61 67

[6] Y. Beldarrain 2006 Distance Education Trends: Integrating new technologies to foster student interaction and collaboration Distance Education 27 2 139 153

[7] H. R. BernardA. WutichG. W. Ryan 2016 Analyzing qualitative data: Systematic approaches SAGE publications

[8] P. BlackD. Wiliam 2010 Inside the Black Box: Raising Standards through Classroom Assessment Phi Delta Kappan 92 1 81 90

[9] J. Brooke 1996 SUS-A quick and dirty usability scale Usability evaluation in industry 189 194 4 7

[10] A. BruunJ. Stage 2015 September An empirical study of the effects of three think-aloud protocols on identification of usability problems In Human-Computer Interaction 159 176 Springer Cham

[11] D. Buehl 2013 Classroom strategies for interactive learning Newark, USA International Reading Association

[12] I. Bunchball 2010 Gamification 101: An introduction to the use of game dynamics to influence behavior. White paper, 9

[13] N. Burbules 2018 Watch IT: The risks and promises of information technologies for education. Routledge

[14] C. CheongJ. FilippouF. Cheong 2014 Towards the gamification of learning: Investigating student perceptions of game elements Journal of Information Systems Education 25 3 233

[15] A. Colburn 2006 What teacher educators need to know about inquiry-based instruction. Paper presented at the annual meeting of the Association for Education of Teachers in Science, Akron, OH

[16] E. L. DeciR. M. Ryan 2010 Self‐determination John Wiley & Sons, Inc.

[17] A. E. DeppingR. L. Mandryk 2017 May Why is This Happening to Me?: How Player Attribution can Broaden our Understanding of Player Experience In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems 1040 1052 ACM

[18] S. Deterding 2012 Gamification: designing for motivation interactions 19 4 14 17

[19] S. DeterdingD. DixonR. KhaledL. Nacke 2011 September From game design elements to gamefulness: defining gamification In Proceedings of the 15th international academic MindTrek conference: Envisioning future media environments 9 15 ACM

[20] D. DichevaC. DichevG. AgreG. Angelova 2015 Gamification in education: a systematic mapping study Journal of Educational Technology & Society 18 3 75

[21] K. EcclestoneJ. DaviesJ. DerrickJ. Gawn 2010 Transforming formative assessment in lifelong learning Berkshire, England Open University Press

[22] C. EhmkeS. Wilson 2007 September Identifying web usability problems from eye-tracking data In Proceedings of the 21st British HCI Group Annual Conference on People and Computers: HCI... but not as we know it-Volume 1 119 128 British Computer Society

[23] C. Erneling 2010 Towards discursive education New York Cambridge University Press

[24] D. GoshevskiJ. VeljanoskaT. Hatziapostolou 2017 September A Review of Gamification Platforms for Higher Education In Proceedings of the 8th Balkan Conference in Informatics 28 ACM

[25] J. HamariJ. KoivistoH. Sarsa 2014 January Does gamification work?--a literature review of empirical studies on gamification In System Sciences (HICSS), 2014 47th Hawaii International Conference on 3025 3034 IEEE

[26] S. HeintzE. L. C. Law 2015 Game Elements-Attributes Model: a First Step towards a Structured Comparison of Educational Games. In DiGRA Conference

[27] A. IosupD. Epema 2014 March An experience report on using gamification in technical higher education In Proceedings of the 45th ACM technical symposium on Computer science education 27 32 ACM

[28] R. Koster 2005 A theory of fun for game design. Scottsdale, AZ: Paraglyph Press

[29] R. N. LandersK. N. BauerR. C. Callan 2017 Gamification of task performance with leaderboards: A goal setting experiment Computers in Human Behavior 71 508 515

[30] C. LehmannZ. Chase 2015 Building school 2.0 John Wiley & Sons

[31] L. de-MarcosE. García-LópezA. García-CabotJ. A. Medina-MerodioA. DomínguezJ. J. Martínez-HerráizT. Diez-Folledo 2016 Social network analysis of a gamified e-learning course: Small-world phenomenon and network metrics as predictors of academic performance Computers in Human Behavior 60 312 321

[32] E. MargolisL. Pauwels 2011 The Sage handbook of visual research methods. Sage. 433 435

[33] E. D. MeklerF. BrühlmannA. N. TuchK. Opwis 2017 Towards understanding the effects of individual gamification elements on intrinsic motivation and performance Computers in Human Behavior 71 525 534

[34] E. D. MeklerF. BrühlmannK. OpwisA. N. Tuch 2013 October Do points, levels and leaderboards harm intrinsic motivation?: an empirical analysis of common gamification elements In Proceedings of the First International Conference on gameful design, research, and applications 66 73 ACM

[35] C. I. Muntean 2011 October Raising engagement in e-learning through gamification In Proc. 6th International Conference on Virtual Learning ICVL (Vol. 1). sn

[36] L. E. NackeS. Deterding 2017 The maturing of gamification research Computers in Human Behavior 71 450 454

[37] F. F. H. NahV. R. TelaproluS. RallapalliP. R. Venkata 2013 July Gamification of education using computer games In International Conference on Human Interface and the Management of Information 99 107 Springer Berlin, Heidelberg

[38] S. Nicholson 2015 A recipe for meaningful gamification In Gamification in education and business 1 20 Springer Cham

[39] J. Nielsen 1994 April Enhancing the explanatory power of usability heuristics In Proceedings of the SIGCHI conference on Human Factors in Computing Systems 152 158 ACM

[40] H. O’BrienP. CairnsM. Hall 2018 A Practical Approach to Measuring User Engagement with the Refined User Engagement Scale (UES) and New UES Short Form. International Journal of Human-Computer Studies 112 28 39

[41] J. L. OrquinN. J. AshbyA. D. Clarke 2016 Areas of interest as a signal detection problem in behavioral eye‐tracking research. Journal of Behavioral Decision Making 29 2-3 103 115

[42] R. W. Picard 2000 Toward computers that recognize and respond to user emotion. IBM systems journal 39 3.4 705 719

[43] A. PooleL. J. Ball 2006 Eye tracking in HCI and usability research. Encyclopedia of human computer interaction, 1 211 219

[44] V. Rebolj 2009 E-education between Pedagogical and Didactic Theory and Practice. Organizacija 42 1

[45] C. RichardsC. W. ThompsonN. Graham 2014 October Beyond designing for motivation: the importance of context in gamification. In Proceedings of the first ACM SIGCHI annual symposium on Computer-human interaction in play 217 226 ACM

[46] K. RobsonK. PlanggerJ. KietzmannI. McCarthyL. Pitt 2015 Is it all a game? Understanding the principles of gamification Business Horizons 58 4 411 420

[47] S. RothD. SchneckenbergC. W. Tsai 2015 The ludic drive as innovation driver: Introduction to the gamification of innovation Creativity and Innovation Management 24 2 300 306

[48] M. SailerJ. U. HenseS. K. MayrH. Mandl 2017 How gamification motivates: An experimental study of the effects of specific game design elements on psychological need satisfaction Computers in Human Behavior 69 371 380

[49] N. Seel 2012 Encyclopedia of the sciences of learning. Germany: Springer

[50] D. W. Shaffer 2006 How computer games help children learn. Macmillan

[51] N. Shin 2006 Online learner’s ‘flow’experience: an empirical study. British Journal of Educational Technology 37 5 705 720

[52] J. SimõesR. D. RedondoA. F. Vilas 2013 A social gamification framework for a K-6 learning platform Computers in Human Behavior 29 2 345 353

[53] S. de Sousa BorgesV. H. DurelliH. M. ReisS. Isotani 2014 March A systematic mapping on gamification applied to education In Proceedings of the 29th Annual ACM Symposium on Applied Computing 216 222 ACM

[54] J. M. Spector 2000 System dynamics and interactive learning environments: Lessons learned and implications for the future. Simulation & Gaming 31 4 528 535

[55] A. TashakkoriC. Teddlie (Eds.) 2010 Sage handbook of mixed methods in social & behavioral research (pp 305-306). Sage

[56] G. F. TondelloH. PremsukhL. E. Nacke 2018 A Theory of Gamification Principles Through Goal-Setting Theory. In Proceedings of the 51st Hawaii International Conference on System Sciences (HICSS

[57] P. Tynjälä 1999 Towards expert knowledge? A comparison between a constructivist and a traditional learning environment in the university. International journal of educational research 31 5 357 442

[58] H. WangC. T. Sun 2011 September Game reward systems: Gaming experiences and social meanings In DiGRA Conference

[59] K. WerbachD. Hunter 2012 For the win: How game thinking can revolutionize your business. Wharton Digital Press

[60] D. West 2012 Digital schools. Washington D.C.: Brookings Institution Press

[61] C. Williams 2011 Research methods. Journal of Business & Economics Research (JBER) 5 3

[62] J. P. ZagalM. MateasC. Fernández-VaraB. HochhalterN. Lichti 2007 Towards an ontological language for game analysis. Worlds in Play: International Perspectives on Digital Games Research, 21, 21

[63] G. ZichermannC. Cunningham 2011 Gamification by design: Implementing game mechanics in web and mobile apps. O’Reilly Media, Inc