Heuristics for Course Workspace Design and Evaluation

Course workspace represents the interface between the teacher, the course content, and the student. Both student engagement and student learning are influenced by the course workspace in a similar way as the end-user experience is influenced by system interface. However, course workspaces are typically designed by teachers without a specific input from students, which contrasts the practice in end-user computing and instructional design where users are or should be involved in the early stages of interface design and evaluation. In this paper, we develop a heuristics framework for being applied in a participatory manner involving the student perspective in the evaluation of course workspaces. The framework has been validated on three Moodle course workspaces. The results showed that the framework and the participatory approach provided valuable insights into student experience with the course workspace, while keeping the evaluation effort manageable in terms of data analysis and interpretation. We believe that the heuristics framework and participatory approach could be valuable for teachers, academics, and practitioners that use e-learning platforms for designing course workspaces. The paper provides also examples of improvement areas that have been identified in the evaluation and highlights new research directions in this area.

Content

Author and article information

Contributors

Dorina Rajanen

Atte Tornberg

Mikko Rajanen

Conference

Publication date: July 2021

Publication date (Print): July 2021

Pages: 60-75

Affiliations

[0001]University of Oulu

Article

DOI: 10.14236/ewic/HCI2021.5

SO-VID: 6dc025f7-1acb-4d40-ba97-1066f14e1692

License:

This work is licensed under a Creative Commons Attribution 4.0 Unported License. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/

Conference name: 34th British HCI Conference

Conference acronym: HCI2021

Conference number: 34

Conference location: London, UK

Conference date: 20th - 21st July 2021

Conference sponsor: Electronic Workshops in Computing (eWiC)

Conference theme: Post-pandemic HCI – Living Digitally

History

Product

1477-9358 BCS Learning & Development

Self URI (article page): https://www.scienceopen.com/hosted-document?doi=10.14236/ewic/HCI2021.5

Self URI (journal page): https://ewic.bcs.org/

8. REFERENCES

S. A. Adepoju, & I. S. Shehu (2014). Usability evaluation of academic websites using automated tools. In Proc. of the 3rd International Conference on User Science and Engineering (i-USEr) (pp. 186-191). IEEE.
P. Albion (1999). Heuristic evaluation of educational multimedia: from theory to practice. In Proceedings ASCILITE 1999: 16th Annual Conf. of the Australasian Society for Computers in Learning in Tertiary Education: Responding to Diversity (pp. 9-15). Australasian Society for Computers in Learning in Tertiary Education (ASCILITE).
A. Alsumait, & A. Al-Osaimi (2009). Usability heuristics evaluation for child e-learning applications. In Proceedings of the 11th international conference on information integration and web-based applications & services (pp. 425-430).
C. Ardito, M. Costabile, M. Marsico, R. Lanzilotti, S. Levialdi, T. Roselli & V. Rossano (2006). An approach to usability evaluation of e-learning applications. Universal Access in the Information Society, 4(3), 270-283. doi:10.1007/s10209-005-0008-6
C. Ardito, M. De Marsico, R. Lanzilotti, S. Levialdi, T. Roselli, V. Rossano, & M. Tersigni (2004). Usability of e-learning tools. In Proceedings of the working conference on Advanced visual interfaces (pp. 80-84).
N. Bevan, J. Carter, S. Harker (2015). ISO 9241-11 revised: what have we learnt about usability since 1998? In: M. Kurosu (ed.) HCI 2015. LNCS, vol. 9169, pp. 143–151. Springer, Cham. doi:10.1007/978-3-319-20901-2_13
M. C. Boudreau, D. Gefen, & D. W. Straub (2001). Validation in information systems research: A state-of-the-art assessment. MIS Quarterly, 25, 1-16.
C. J. Carvalho, E. M. Borycki, & A. Kushniruk (2009). Ensuring the safety of health information systems: using heuristics for patient safety. Healthcare Quarterly 12: 49-54.
M. R. Davids, U. M. Chikte, & M. L. Halperin (2013). An efficient approach to improve the usability of e-learning resources: the role of heuristic evaluation. Advances in physiology education, 37(3), 242-248.
W. J. Doll & G. Torkzadeh (1988). The Measurement of End-User Computing Satisfaction. MIS Quarterly, 12(2), pp. 259-274. doi:10.2307/248851
L. P. Dringus, & M. S. Cohen (2005). An adaptable usability heuristic checklist for online courses. In Proceedings Frontiers in Education 35th Annual Conference (pp. T2H-6). IEEE.
T. Erenler, & H. Hale (2018). Heuristic evaluation of e-learning. International Journal of Organizational Leadership, 7, 195-210.
P. Georgiakakis, A. Papasalouros, S. Retalis, K. Siassiakos, & N. Papaspyrou (2005). Evaluating the usability of web-based learning management systems. THEMES in Education, 6(1), 45-59.
S. Hermawati, & G. Lawson (2016). Establishing usability heuristics for heuristics evaluation in a specific domain: Is there a consensus?. Applied ergonomics, 56, 34-51.
M. Hertzum (2010). Images of usability. Intl. Journal of Human–Computer Interaction, 26(6), 567-600.
M. Hertzum, & N. E. Jacobsen (2001). The evaluator effect: A chilling fact about usability evaluation methods. International Journal of Human-Computer Interaction, 13(4), 421-443.
O. Hyppönen, & S. Lindén (2009). Handbook for teachers: course structures, teaching methods and assessment. Publications of Teaching and Learning Development Unit 5/2009. Helsinki University of Technology, Espoo.
ISO International Standardization Organization (1991). ISO/IEC: 9126 Information technology – Software Product Evaluation – Quality characteristics and guidelines for their use.
ISO International Standardization Organization (1998) ISO 9241-11: Ergonomic Requirements for Office Work with Visual Display Terminals (VDTs) - Part 11: Guidance on Usability.
ISO International Standardization Organization (2010). ISO 9241-210. Ergonomics of human system interaction – Part 210: Human-centred design for interactive systems.
M. Y. Ivory, & M. A. Hearst (2001). The state of the art in automating usability evaluation of user interfaces. ACM Computing Surveys 33(4), 470-516.
C. Jiménez, C. Rusu, D. Gorgan, & R. Inostroza (2013). Grid applications to process, supervise and analyze earth science related phenomena: what about usability?. In Proc. of the 2013 Chilean Conf. on Human-Computer Interaction (pp. 94-97).
G. Kakasevski, M. Mihajlov, S. Arsenovski, & S. Chungurski (2008). Evaluating usability in learning management system Moodle. In Iti 2008-30th international conference on information technology interfaces (pp. 613-618). IEEE.
C. M. Karat, R. Campbell, & T. Fiegel (1992). Comparison of empirical testing and walkthrough methods in user interface evaluation. In Proceedings of the SIGCHI conference on Human factors in computing systems (pp. 397-404).
K. Kogi (2006). Participatory methods effective for ergonomic workplace improvement. Applied ergonomics, 37(4), 547-554.
A. Koohang, & J. du Plessis (2004). Architecting usability properties in the e-learning instructional design process. International Journal on E-learning, 3(3), 38-44.
S. Kujala, V. Roto, K. Väänänen-Vainio-Mattila, E. Karapanos, A. Sinnelä (2011). UX curve: a method for evaluating long-term user experience. Interact. Comput. 23(5), 473–483.
E. Lehtinen, A. Gegenfurtner, L. Helle, & R. Säljö (2020). Conceptual change in the development of visual expertise. International Journal of Educational Research, 100, 101545.
C. P. Lim, & C. S. Chai (2008). Teachers’ pedagogical beliefs and their planning and conduct of computer-mediated classroom lessons. British Journal of Educational Technology, 39(5), 807-828.
D. Marghescu (2009). Usability evaluation of information systems: a review of five international standards. In: W. Wojtkowskiet al. (eds.) Information Systems Development, pp. 131–142. Springer, Boston. doi:10.1007/978-0-387-68772-8_11
J. McCarthy, P. Wright (2004). Technology as experience. Interactions 11(5), 42–43.
B. Mehlenbacher, L. Bennett, T. Bird, M. Ivey, J. Lucas, J. Morton, & L. Whitman (2005). Usable e-learning: A conceptual model for evaluation and design. In Proceedings of HCI International (Vol. 2005, p. 11th).
G. Meiselwitz, & W. Sadera (2008). Investigating the connection between usability and learning outcomes in online learning environments. Journal of Online Learning and Teaching, 4(2), 234-242.
R. Molich, & J. Nielsen (1990). Improving a human-computer dialogue. Communications of the ACM, 33(3), 338-348.
M Monari (2005). Evaluation of Collaborative Tools in Web-Based E-Learning Systems. Unpublished Master’s Thesis, Royal Institute of Technology, Stockholm, Sweden.
Moodle (2017) Why is Moodle the world’s most widely used learning platform? https://moodle.com/news/moodle-worlds-widely-used-learning-platform/, last retrieved 9.9.2020
Moodle (2018) About Moodle - Pedagogy https://docs.moodle.org/39/en/Pedagogy, last retrieved 4.9.2020
J. S. Mtebe, & M. M. Kissaka (2015). Heuristics for evaluating usability of learning management systems in Africa. In 2015 IST-Africa Conference (pp. 1-13). IEEE.
W. T. Nakamura, E. H. T. de Oliveira, & T. Conte (2017). Usability and User Experience Evaluation of Learning Management Systems-A Systematic Mapping Study. In International Conference on Enterprise Information Systems (Vol. 2, pp. 97-108). SCITEPRESS.
S. S. Nash, & M. Moore (2014). Moodle Course Design Best Practices. Packt Publishing Ltd.
R. Naylor, M. Dollinger, M. Mahat, & M. Khawaja (2020). Students as customers versus as active agents: conceptualising the student role in governance and quality assurance. Higher Education Research & Development, 1-14.
J. Nielsen (1993) Usability Engineering. Academic Press, Boston
J. Nielsen (1994). Heuristic evaluation. In J. Nielsen, and R.L. Mack (Eds.), Usability inspection methods. New York: John Wiley & Sons.
J. Nielsen, & R. Molich (1990). Heuristic evaluation of user interfaces. In Proceedings of the SIGCHI conference on Human factors in computing systems (pp. 249-256).
P. Nokelainen (2006). An empirical assessment of pedagogical usability criteria for digital learning material with elementary school students. Journal of Educational Technology & Society, 9(2), 178-197.
S. Ozkan & R. Koseler (2009). Multi-dimensional students’ evaluation of e-learning systems in the higher education context: An empirical investigation. Computers & Education, 53(4), 1285-1296. doi:10.1016/j.compedu.2009.06.011
A. S. Palincsar (1998). Social constructivist perspectives on teaching and learning. Annual review of psychology, 49(1), 345-375.
F. Paz, J. A. Pow-Sang, & C. Collazos (2018). Formal protocol to conduct usability heuristic evaluations in the context of the software development process. Int. J. Eng. Technol, 7(2.28), 10-19.
G. J. Posner, K. A. Strike, P. W. Hewson, & W. A. Gertzog (1982). Accommodation of a scientific conception: Toward a theory of conceptual change. Science education, 66(2), 211-227.
L. Postareff, & S. Lindblom-Ylänne (2008). Variation in teachers' descriptions of teaching: Broadening the understanding of teaching in higher education. Learning and Instruction, 18(2), 109-120.
R. S. Prawat (1992). Teachers' beliefs about teaching and learning: A constructivist perspective. American Journal of Education, 100(3), 354-395.
J. Preece, H. Sharp, & Y. Rogers (2015). Interaction design: beyond human-computer interaction. John Wiley & Sons.
D. Rajanen, T. Clemmensen, N. Iivari, Y. Inal, K. Rizvanoglu, A. Sivaji, A. Roche (2017) UX professionals’ definitions of usability and UX – A comparison between Turkey, Finland, Denmark, France and Malaysia. In: R. Bernhauptet al. (eds) Human-Computer Interaction – INTERACT 2017. LNCS vol. 10516 (16th IFIP TC.13 International Conference on HCI - INTERACT 2017). [Cross Ref]
M. Rajanen & D. Rajanen (2020) Usability: A Cybernetics Perspective. In Proc. of the 6th International Workshop on Socio-Technical Perspective in IS development (STPIS'20).
T. C. Reeves, L. Benson, D. Elliott, M. Grant, D. Holschuh, B. Kim, . . . S. Loh (2002). Usability and instructional design heuristics for e-learning evaluation (pp. 1615-1621). Association for the Advancement of Computing in Education (AACE).
A. Rentróia-Bonito, A. Martins, T. Guerreiro, & J. Jorge (2008). Evaluating learning support systems usability: An empirical approach. Communication & Cognition, 41(1), 143.
K. Sagar, & A. Saha (2017). Qualitative usability feature selection with ranking: a novel approach for ranking the identified usability problematic attributes for academic websites using data-mining techniques. Human-centric Computing and Information Sciences, 7(1), 29.
J. Salas, A. Chang, L. Montalvo, A. Núñez, M. Vilcapoma, A. Moquillaza, . . . F. Paz (2019). Guidelines to evaluate the usability and user experience of learning support platforms: A systematic review. Communications in Computer and Information Science, 1114, 238-254. doi:10.1007/978-3-030-37386-3_18
V. F. M. Salvador, & L. de Assis Moura (2010). Heuristic evaluation for automatic radiology reporting transcription systems. In 10th International Conference on Information Science, Signal Processing and their Applications (ISSPA 2010) (pp. 292-295). IEEE.
L. Senol, H. Gecili, & P. O. Durdu (2014). Usability evaluation of a moodle based learning management system. In EdMedia+ Innovate Learning (pp. 850-858). Association for the Advancement of Computing in Education (AACE).
B. Shneiderman (1998). Designing the User Interface, Addison-Wesley Publishing Company, USA.
B. Shneiderman, C. Plaisant, M. S. Cohen, S. Jacobs, N. Elmqvist, (2018). Designing the User Interface: Strategies for Effective Human-Computer Interaction. 6th Ed. Pearson.
P. Siklander, M. Kangas, S. Ruhalahti, & S. Korva (2017). Exploring triggers for arousing interest in the online learning. In International Technology, Education and Development Conference (pp. 9081-9089).
A. Sivaji, A. Abdullah, & A. G. Downe (2011). Usability testing methodology: Effectiveness of heuristic evaluation in E-government website development. In Proc of the 5fth Asia Modelling Symposium (pp. 68-72). IEEE.
A. Sivaji, M. R. Abdullah, A. G. Downe, & W. F. W. Ahmad (2013). Hybrid usability methodology: integrating heuristic evaluation with laboratory testing across the software development lifecycle. In Proc. of the 10th International Conference on Information Technology: New Generations (pp. 375-383). IEEE.
A. Sivaji, S. F. Nielsen & T. Clemmensen (2017) A textual feedback tool for empowering participants in usability and UX evaluations. Int. J. Human–Computer Interact. 33(5), pp. 357–370, 2017.
D. Squires, & J. Preece (1996). Usability and learning: evaluating the potential of educational software. Computers & Education, 27(1), 15-22.
D. Squires, & J. Preece (1999). Predicting quality in educational software. Interacting with computers, 11(5), 467-483.
S. Ssemugabi, & R. De Villiers (2007). A comparative study of two usability evaluation methods using a web-based e-learning application. In Proceedings of the 2007 annual research conference of the South African institute of computer scientists and information technologists on IT research in developing countries (pp. 132-142).
D. Tolhurst (1992). A checklist for evaluating content-based hypertext computer software. Educational Technology, 32(3), 17-21.
K. Trigwell, M. Prosser, & P. Taylor (1994). Qualitative differences in approaches to teaching first year university science. Higher Education, 27(1), 75-84.
J. M. Toribio-Guzmán, A. García-Holgado, F. S. Pérez, F. J. García-Peñalvo, & M. A. F. Martín (2016). Study of the usability of the private social network SocialNet using heuristic evaluation. In Proceedings of the XVII International Conference on Human Computer Interaction (pp. 1-5).
M. van Welie, G. C. van der Veer, & A. P. W. Eliëns (1999). Breaking down Usability. In Human Computer Interaction-Proceedings of Interact 99, 30th August-3rd September 1999, Edinburgh, Scotland (pp. 613-620). IOS Press.
M. Windschitl (2002). Framing constructivism in practice as the negotiation of dilemmas: An analysis of the conceptual, pedagogical, cultural, and political challenges facing teachers. Review of Educational Research, 72(2), 131-175.
J. O. Wobbrock, & J. A. Kientz (2016). Research contributions in human-computer interaction. Interactions, 23(3), 38-44.
G. B. Wright (2011). Student-centered learning in higher education. International Journal of Teaching and Learning in Higher Education, 23(1), 92-97.
P. Zaharias & P. Koutsabasis (2012). Heuristic evaluation of e-learning courses: A comparative analysis of two e-learning heuristic sets. Campus-Wide Information Systems, 29(1), 45-60. doi:10.1108/10650741211192046
P. Zaharias & A. Poulymenakou (2006). Implementing learner-centred design: The interplay between usability and instructional design practices. Interactive Technology and Smart Education, 3(2), 87-100. doi:10.1108/17415650680000055
P. Zaharias & A. Poulymenakou (2009). Developing a Usability Evaluation Method for e-Learning Applications: Beyond Functional Usability. International Journal of Human-Computer Interaction, 25(1), 75-98. doi:10.1080/10447310802546716

Comments

Comment on this article

[1] S. A. Adepoju, & I. S. Shehu (2014). Usability evaluation of academic websites using automated tools. In Proc. of the 3rd International Conference on User Science and Engineering (i-USEr) (pp. 186-191). IEEE.

[2] P. Albion (1999). Heuristic evaluation of educational multimedia: from theory to practice. In Proceedings ASCILITE 1999: 16th Annual Conf. of the Australasian Society for Computers in Learning in Tertiary Education: Responding to Diversity (pp. 9-15). Australasian Society for Computers in Learning in Tertiary Education (ASCILITE).

[3] A. Alsumait, & A. Al-Osaimi (2009). Usability heuristics evaluation for child e-learning applications. In Proceedings of the 11th international conference on information integration and web-based applications & services (pp. 425-430).

[4] C. Ardito, M. Costabile, M. Marsico, R. Lanzilotti, S. Levialdi, T. Roselli & V. Rossano (2006). An approach to usability evaluation of e-learning applications. Universal Access in the Information Society, 4(3), 270-283. doi:10.1007/s10209-005-0008-6

[5] C. Ardito, M. De Marsico, R. Lanzilotti, S. Levialdi, T. Roselli, V. Rossano, & M. Tersigni (2004). Usability of e-learning tools. In Proceedings of the working conference on Advanced visual interfaces (pp. 80-84).

[6] N. Bevan, J. Carter, S. Harker (2015). ISO 9241-11 revised: what have we learnt about usability since 1998? In: M. Kurosu (ed.) HCI 2015. LNCS, vol. 9169, pp. 143–151. Springer, Cham. doi:10.1007/978-3-319-20901-2_13

[7] M. C. Boudreau, D. Gefen, & D. W. Straub (2001). Validation in information systems research: A state-of-the-art assessment. MIS Quarterly, 25, 1-16.

[8] C. J. Carvalho, E. M. Borycki, & A. Kushniruk (2009). Ensuring the safety of health information systems: using heuristics for patient safety. Healthcare Quarterly 12: 49-54.

[9] M. R. Davids, U. M. Chikte, & M. L. Halperin (2013). An efficient approach to improve the usability of e-learning resources: the role of heuristic evaluation. Advances in physiology education, 37(3), 242-248.

[10] W. J. Doll & G. Torkzadeh (1988). The Measurement of End-User Computing Satisfaction. MIS Quarterly, 12(2), pp. 259-274. doi:10.2307/248851

[11] L. P. Dringus, & M. S. Cohen (2005). An adaptable usability heuristic checklist for online courses. In Proceedings Frontiers in Education 35th Annual Conference (pp. T2H-6). IEEE.

[12] T. Erenler, & H. Hale (2018). Heuristic evaluation of e-learning. International Journal of Organizational Leadership, 7, 195-210.

[13] P. Georgiakakis, A. Papasalouros, S. Retalis, K. Siassiakos, & N. Papaspyrou (2005). Evaluating the usability of web-based learning management systems. THEMES in Education, 6(1), 45-59.

[14] S. Hermawati, & G. Lawson (2016). Establishing usability heuristics for heuristics evaluation in a specific domain: Is there a consensus?. Applied ergonomics, 56, 34-51.

[15] M. Hertzum (2010). Images of usability. Intl. Journal of Human–Computer Interaction, 26(6), 567-600.

[16] M. Hertzum, & N. E. Jacobsen (2001). The evaluator effect: A chilling fact about usability evaluation methods. International Journal of Human-Computer Interaction, 13(4), 421-443.

[17] O. Hyppönen, & S. Lindén (2009). Handbook for teachers: course structures, teaching methods and assessment. Publications of Teaching and Learning Development Unit 5/2009. Helsinki University of Technology, Espoo.

[18] ISO International Standardization Organization (1991). ISO/IEC: 9126 Information technology – Software Product Evaluation – Quality characteristics and guidelines for their use.

[19] ISO International Standardization Organization (1998) ISO 9241-11: Ergonomic Requirements for Office Work with Visual Display Terminals (VDTs) - Part 11: Guidance on Usability.

[20] ISO International Standardization Organization (2010). ISO 9241-210. Ergonomics of human system interaction – Part 210: Human-centred design for interactive systems.

[21] M. Y. Ivory, & M. A. Hearst (2001). The state of the art in automating usability evaluation of user interfaces. ACM Computing Surveys 33(4), 470-516.

[22] C. Jiménez, C. Rusu, D. Gorgan, & R. Inostroza (2013). Grid applications to process, supervise and analyze earth science related phenomena: what about usability?. In Proc. of the 2013 Chilean Conf. on Human-Computer Interaction (pp. 94-97).

[23] G. Kakasevski, M. Mihajlov, S. Arsenovski, & S. Chungurski (2008). Evaluating usability in learning management system Moodle. In Iti 2008-30th international conference on information technology interfaces (pp. 613-618). IEEE.

[24] C. M. Karat, R. Campbell, & T. Fiegel (1992). Comparison of empirical testing and walkthrough methods in user interface evaluation. In Proceedings of the SIGCHI conference on Human factors in computing systems (pp. 397-404).

[25] K. Kogi (2006). Participatory methods effective for ergonomic workplace improvement. Applied ergonomics, 37(4), 547-554.

[26] A. Koohang, & J. du Plessis (2004). Architecting usability properties in the e-learning instructional design process. International Journal on E-learning, 3(3), 38-44.

[27] S. Kujala, V. Roto, K. Väänänen-Vainio-Mattila, E. Karapanos, A. Sinnelä (2011). UX curve: a method for evaluating long-term user experience. Interact. Comput. 23(5), 473–483.

[28] E. Lehtinen, A. Gegenfurtner, L. Helle, & R. Säljö (2020). Conceptual change in the development of visual expertise. International Journal of Educational Research, 100, 101545.

[29] C. P. Lim, & C. S. Chai (2008). Teachers’ pedagogical beliefs and their planning and conduct of computer-mediated classroom lessons. British Journal of Educational Technology, 39(5), 807-828.

[30] D. Marghescu (2009). Usability evaluation of information systems: a review of five international standards. In: W. Wojtkowskiet al. (eds.) Information Systems Development, pp. 131–142. Springer, Boston. doi:10.1007/978-0-387-68772-8_11

[31] J. McCarthy, P. Wright (2004). Technology as experience. Interactions 11(5), 42–43.

[32] B. Mehlenbacher, L. Bennett, T. Bird, M. Ivey, J. Lucas, J. Morton, & L. Whitman (2005). Usable e-learning: A conceptual model for evaluation and design. In Proceedings of HCI International (Vol. 2005, p. 11th).

[33] G. Meiselwitz, & W. Sadera (2008). Investigating the connection between usability and learning outcomes in online learning environments. Journal of Online Learning and Teaching, 4(2), 234-242.

[34] R. Molich, & J. Nielsen (1990). Improving a human-computer dialogue. Communications of the ACM, 33(3), 338-348.

[35] M Monari (2005). Evaluation of Collaborative Tools in Web-Based E-Learning Systems. Unpublished Master’s Thesis, Royal Institute of Technology, Stockholm, Sweden.

[36] Moodle (2017) Why is Moodle the world’s most widely used learning platform? https://moodle.com/news/moodle-worlds-widely-used-learning-platform/, last retrieved 9.9.2020

[37] Moodle (2018) About Moodle - Pedagogy https://docs.moodle.org/39/en/Pedagogy, last retrieved 4.9.2020

[38] J. S. Mtebe, & M. M. Kissaka (2015). Heuristics for evaluating usability of learning management systems in Africa. In 2015 IST-Africa Conference (pp. 1-13). IEEE.

[39] W. T. Nakamura, E. H. T. de Oliveira, & T. Conte (2017). Usability and User Experience Evaluation of Learning Management Systems-A Systematic Mapping Study. In International Conference on Enterprise Information Systems (Vol. 2, pp. 97-108). SCITEPRESS.

[40] S. S. Nash, & M. Moore (2014). Moodle Course Design Best Practices. Packt Publishing Ltd.

[41] R. Naylor, M. Dollinger, M. Mahat, & M. Khawaja (2020). Students as customers versus as active agents: conceptualising the student role in governance and quality assurance. Higher Education Research & Development, 1-14.

[42] J. Nielsen (1993) Usability Engineering. Academic Press, Boston

[43] J. Nielsen (1994). Heuristic evaluation. In J. Nielsen, and R.L. Mack (Eds.), Usability inspection methods. New York: John Wiley & Sons.

[44] J. Nielsen, & R. Molich (1990). Heuristic evaluation of user interfaces. In Proceedings of the SIGCHI conference on Human factors in computing systems (pp. 249-256).

[45] P. Nokelainen (2006). An empirical assessment of pedagogical usability criteria for digital learning material with elementary school students. Journal of Educational Technology & Society, 9(2), 178-197.

[46] S. Ozkan & R. Koseler (2009). Multi-dimensional students’ evaluation of e-learning systems in the higher education context: An empirical investigation. Computers & Education, 53(4), 1285-1296. doi:10.1016/j.compedu.2009.06.011

[47] A. S. Palincsar (1998). Social constructivist perspectives on teaching and learning. Annual review of psychology, 49(1), 345-375.

[48] F. Paz, J. A. Pow-Sang, & C. Collazos (2018). Formal protocol to conduct usability heuristic evaluations in the context of the software development process. Int. J. Eng. Technol, 7(2.28), 10-19.

[49] G. J. Posner, K. A. Strike, P. W. Hewson, & W. A. Gertzog (1982). Accommodation of a scientific conception: Toward a theory of conceptual change. Science education, 66(2), 211-227.

[50] L. Postareff, & S. Lindblom-Ylänne (2008). Variation in teachers' descriptions of teaching: Broadening the understanding of teaching in higher education. Learning and Instruction, 18(2), 109-120.

[51] R. S. Prawat (1992). Teachers' beliefs about teaching and learning: A constructivist perspective. American Journal of Education, 100(3), 354-395.

[52] J. Preece, H. Sharp, & Y. Rogers (2015). Interaction design: beyond human-computer interaction. John Wiley & Sons.

[53] D. Rajanen, T. Clemmensen, N. Iivari, Y. Inal, K. Rizvanoglu, A. Sivaji, A. Roche (2017) UX professionals’ definitions of usability and UX – A comparison between Turkey, Finland, Denmark, France and Malaysia. In: R. Bernhauptet al. (eds) Human-Computer Interaction – INTERACT 2017. LNCS vol. 10516 (16th IFIP TC.13 International Conference on HCI - INTERACT 2017). [Cross Ref]

[54] M. Rajanen & D. Rajanen (2020) Usability: A Cybernetics Perspective. In Proc. of the 6th International Workshop on Socio-Technical Perspective in IS development (STPIS'20).

[55] T. C. Reeves, L. Benson, D. Elliott, M. Grant, D. Holschuh, B. Kim, . . . S. Loh (2002). Usability and instructional design heuristics for e-learning evaluation (pp. 1615-1621). Association for the Advancement of Computing in Education (AACE).

[56] A. Rentróia-Bonito, A. Martins, T. Guerreiro, & J. Jorge (2008). Evaluating learning support systems usability: An empirical approach. Communication & Cognition, 41(1), 143.

[57] K. Sagar, & A. Saha (2017). Qualitative usability feature selection with ranking: a novel approach for ranking the identified usability problematic attributes for academic websites using data-mining techniques. Human-centric Computing and Information Sciences, 7(1), 29.

[58] J. Salas, A. Chang, L. Montalvo, A. Núñez, M. Vilcapoma, A. Moquillaza, . . . F. Paz (2019). Guidelines to evaluate the usability and user experience of learning support platforms: A systematic review. Communications in Computer and Information Science, 1114, 238-254. doi:10.1007/978-3-030-37386-3_18

[59] V. F. M. Salvador, & L. de Assis Moura (2010). Heuristic evaluation for automatic radiology reporting transcription systems. In 10th International Conference on Information Science, Signal Processing and their Applications (ISSPA 2010) (pp. 292-295). IEEE.

[60] L. Senol, H. Gecili, & P. O. Durdu (2014). Usability evaluation of a moodle based learning management system. In EdMedia+ Innovate Learning (pp. 850-858). Association for the Advancement of Computing in Education (AACE).

[61] B. Shneiderman (1998). Designing the User Interface, Addison-Wesley Publishing Company, USA.

[62] B. Shneiderman, C. Plaisant, M. S. Cohen, S. Jacobs, N. Elmqvist, (2018). Designing the User Interface: Strategies for Effective Human-Computer Interaction. 6th Ed. Pearson.

[63] P. Siklander, M. Kangas, S. Ruhalahti, & S. Korva (2017). Exploring triggers for arousing interest in the online learning. In International Technology, Education and Development Conference (pp. 9081-9089).

[64] A. Sivaji, A. Abdullah, & A. G. Downe (2011). Usability testing methodology: Effectiveness of heuristic evaluation in E-government website development. In Proc of the 5fth Asia Modelling Symposium (pp. 68-72). IEEE.

[65] A. Sivaji, M. R. Abdullah, A. G. Downe, & W. F. W. Ahmad (2013). Hybrid usability methodology: integrating heuristic evaluation with laboratory testing across the software development lifecycle. In Proc. of the 10th International Conference on Information Technology: New Generations (pp. 375-383). IEEE.

[66] A. Sivaji, S. F. Nielsen & T. Clemmensen (2017) A textual feedback tool for empowering participants in usability and UX evaluations. Int. J. Human–Computer Interact. 33(5), pp. 357–370, 2017.

[67] D. Squires, & J. Preece (1996). Usability and learning: evaluating the potential of educational software. Computers & Education, 27(1), 15-22.

[68] D. Squires, & J. Preece (1999). Predicting quality in educational software. Interacting with computers, 11(5), 467-483.

[69] S. Ssemugabi, & R. De Villiers (2007). A comparative study of two usability evaluation methods using a web-based e-learning application. In Proceedings of the 2007 annual research conference of the South African institute of computer scientists and information technologists on IT research in developing countries (pp. 132-142).

[70] D. Tolhurst (1992). A checklist for evaluating content-based hypertext computer software. Educational Technology, 32(3), 17-21.

[71] K. Trigwell, M. Prosser, & P. Taylor (1994). Qualitative differences in approaches to teaching first year university science. Higher Education, 27(1), 75-84.

[72] J. M. Toribio-Guzmán, A. García-Holgado, F. S. Pérez, F. J. García-Peñalvo, & M. A. F. Martín (2016). Study of the usability of the private social network SocialNet using heuristic evaluation. In Proceedings of the XVII International Conference on Human Computer Interaction (pp. 1-5).

[73] M. van Welie, G. C. van der Veer, & A. P. W. Eliëns (1999). Breaking down Usability. In Human Computer Interaction-Proceedings of Interact 99, 30th August-3rd September 1999, Edinburgh, Scotland (pp. 613-620). IOS Press.

[74] M. Windschitl (2002). Framing constructivism in practice as the negotiation of dilemmas: An analysis of the conceptual, pedagogical, cultural, and political challenges facing teachers. Review of Educational Research, 72(2), 131-175.

[75] J. O. Wobbrock, & J. A. Kientz (2016). Research contributions in human-computer interaction. Interactions, 23(3), 38-44.

[76] G. B. Wright (2011). Student-centered learning in higher education. International Journal of Teaching and Learning in Higher Education, 23(1), 92-97.

[77] P. Zaharias & P. Koutsabasis (2012). Heuristic evaluation of e-learning courses: A comparative analysis of two e-learning heuristic sets. Campus-Wide Information Systems, 29(1), 45-60. doi:10.1108/10650741211192046

[78] P. Zaharias & A. Poulymenakou (2006). Implementing learner-centred design: The interplay between usability and instructional design practices. Interactive Technology and Smart Education, 3(2), 87-100. doi:10.1108/17415650680000055

[79] P. Zaharias & A. Poulymenakou (2009). Developing a Usability Evaluation Method for e-Learning Applications: Beyond Functional Usability. International Journal of Human-Computer Interaction, 25(1), 75-98. doi:10.1080/10447310802546716

Studying business & IT? Drive your professional career forwards with BCS books - for a 20% discount click here: shop.bcs.org