The Usability of E-learning Platforms in Higher Education: A Systematic Mapping Study

The use of e-learning in higher education has increased significantly in recent years, which has led to several studies being conducted to investigate the usability of the platforms that support it. A variety of different usability evaluation methods and attributes have been used, and it has therefore become important to start reviewing this work in a systematic way to determine how the field has developed in the last 15 years. This paper describes a systematic mapping study that performed searches on five electronic libraries to identify usability issues and methods that have been used to evaluate e-learning platforms. Sixty-one papers were selected and analysed, with the majority of studies using a simple research design reliant on questionnaires. The usability attributes measured were mostly related to effectiveness, satisfaction, efficiency, and perceived ease of use. Furthermore, several research gaps have been identified and recommendations have been made for further work in the area of the usability of online learning.


INTRODUCTION
Although the use of learning technology, commonly termed e-learning, has become the new norm in higher education, several studies have criticised the technologies developed to support it (Marković and Jovanović, 2012). The quality of learning received via technologies such as virtual learning environments (VLEs) often does not meet the expectations of students and teachers, with the usability of VLEs being one of the main factors affecting learning efficacy (Raspopovic et al., 2014) and one of the biggest factors faced by e-learning platform designers (Ardito et al., 2004). Usability plays an important role in the success of e-learning platforms, with end users spending a long time familiarising themselves with the platform's functionality, as it is not usable, instead of studying the content of the subject (Costabile et al., 2005). This means that the cost of poor usability is high, with learners left feeling frustrated (Plata andAlado, 2009), unsatisfied, andineffective (Abedour andSmith, 2006).
Although in the last few decades several studies relating to the usability of e-learning platforms have been published, few reviews have evaluated and synthesised this information. This paper therefore describes a systematic mapping study (SMS) that has been undertaken following the methodology outlined by Kitchenham, Budgen and Brereton (2015). An SMS is a secondary study method used to identify and summarise the knowledge about a particular area contained in papers and articles; it also helps to categorise and evaluate information (Kitchenham and Charters, 2007;Kitchenham, Budgen and Brereton, 2015). The main aim of this mapping study is to identify the nature and extent of e-learning and usability research that has been conducted, focusing on areas such as the usability attributes measured, usability methods employed, and level of study of the participants.
Section 2 describes related platforms and reviews and section 3 outlines the methodology for the mapping study, including the search method and inclusion/exclusion criteria. Section 4 presents the results and sections 5 and 6 discuss the limitations and conclusions of this work.

E-learning Platforms
The following section outlines some of the more common e-learning platforms (not including bespoke platforms made by universities to meet their particular needs (Jain, 2015)).

Blackboard
Matthew Pittinsky and Michael Chasen, introduced Blackboard in 1997. Blackboard is a Virtual Learning Environment (Logan and Neumann, 2010), where students are able to access course The Usability of E-learning Platforms in Higher Education: A Systematic Mapping Study • Khaled Abuhlfaia • Ed de Quincey 2 information (often in secured areas) and download and upload course materials and homework (Conrad, 2016). It can also be used to improve the efficiency of communication between learners and their institution. It supports a number of additional learning activities including conveying daily messages, tasks, course content, chat rooms, assignments, quizzes, exams, and grades (Eldridge, 2014).

Modular Object-Oriented Dynamic Learning
Environment (Moodle) The Moodle VLE platform is used widely all over the world by institutes, universities, companies and independent educators (Al-Ajlan and Zedan, 2008). It has been used by more than 94 million users in 233 countries (Moodle.org, 2018). Moodle was originally created by Martin Dougiamas in Australia in 1999, (Fuentes et al., 2012) and is an open source product (Kumar, Gankotiya and Dutta, 2011).

Sakai
Sakai is a platform developed by a group of academic institutions and commercial organizations, working together to develop a Collaborative Learning Environment (CLE) (Aggarwal et al., 2012). Sakai was released in 2005 and is currently on version 12.0, which was released in March 2018. Similar to Moodle, Sakai is an open source platform and has been used within e-learning by over 350 institutions and by 4 million learners around the world (Sakaiproject.org, 2018). However, to be able to modify the user-interface of Sakai you need to have relevant programming skills in this field (Chauhan et al., 2015).

Previous Studies
There have been two previous SMSs and several more general reviews in the area of the usability of e-learning platforms. However, the two previous studies have different aims to the study described in this paper, which are outlined in the following section.
2.2. 1 Nakamura, de Oliveira and Conte (2017) An SMS was performed by Nakamura , de Oliveira and Conte (2017) on the usability and user experience (UX) of learning management systems (LMS). This review aimed to analyse all studies that focused on LMS usability and UX evaluation techniques covering learning factors, type, availability performing method, restriction and origin, published between January 2004 and August 2016. The results of this SMS revealed that there is a need to study the lack of specific feedback given to address some of the issues and that more research is needed in the area of LMS.
The following are some of the main differences between our study and Nakamura et al.'s SMS.
Although the two studies have almost the same number of papers (62 and 61), due to a difference in the inclusion and exclusion criteria, we have included five papers that were excluded in their study and they included eight studies that we excluded. Nakamura et al.'s study included papers that are not related to higher education and included work related to mobile learning (which we have excluded). Moreover, our study has used Wiley Online Library and Google Scholar, whereas Nakamura et al. used (2010) investigated usability evaluation methods (UEMs) for e-learning platforms, covering papers published between 2000 and 2010. They analysed relevant papers to determine how pedagogical aspects and criteria were treated when usability evaluations were performed. They then presented a summary of all the UEMs from their included studies and factors related to e-learning. However, some common evaluation methods were not found in the 27 papers they included, e.g. focus groups, interviews, and log file analysis. This study summarised four important pedagogical usability" factors: designing the content for learning, assessments, user's motivation to learn, and authoring supportive tools. Furthermore, they found that some studies were not fully aware of the importance of pedagogical aspects in usability (as opposed to general usability). However, their focus was limited to evaluation methods and limited by the lack of access to the full text of some articles. Moreover, the pedagogical usability factors summarised were based on their knowledge and what they had learnt from the case studies found. They admit that there may be other studies and usability factors that they have not considered (Bernerus and Zhang, 2010).

Freire, Arezes and Campos (2012)
Freire, Arezes and Campos (2012) presented a literature review about the relationship between ergonomics and usability in e-learning covering the last 30 years. There are no details regarding whether the research was conducted in a systematic or narrative way. The analysis of this review's results enabled the authors to identify three differences among the UEMs used dependent on whether the system's performance, the user's performance or the dialogue between users and systems was being evaluated. The majority of methods that were used to test e-3 learning's usability were the same as those used for general systems, i.e. not specialist methods. This study contained a variety of points of view, as it included researchers from different scientific areas, such as Ergonomics, Computer Science, and Education. The researchers' conclusion was that the most important point is knowing how to combine the most relevant methods for each type of evaluation and type of stakeholder. Vukovac, Kirinic and Klicek (2010) The aim of this review was to identify a set of criteria for choosing appropriate methods to test the usability of online learning platforms. In addition to analysing current UEMs for e-learning platforms, they compared UEMs for distance learning. The conclusion of this review was that factors relating to effectiveness, time, ease of application, cost and efficiency can affect the decision regarding which usability testing method to use on e-learning platforms (Ssemugabi and de Villers, 2007b). The authors also found that instructions for the methods are lacking and that more information would be needed if the methods were to be adopted by other researchers. However, this review covered a period up to 2010, whereas we have covered a period up to 2016.

Ssemugabi and de Villiers (2007b)
The aim of this study was to compare the results of two evaluation methods on the Info3Net system: the UEMs and heuristic evaluation (HE). Ssemugabi and de Villiers (2007b) concluded that the results that were gathered using HE were similar to the result that was collected using a survey. However, more problems were identified by the four expert evaluators than the 61 students. Moreover, the HE conducted by the experts showed that it seemed to be an adequate and appropriate method to evaluate the e-learning systems.

Conclusion
Several reviews concerning the usability of elearning have been conducted. However, the mapping study described in this paper has a different focus and set of research questions as well as different inclusion and exclusion criteria and is based on a systematic, reproducible method.

RESEARCH METHOD
This SMS has been conducted in line with the guidelines provided by Kitchenham, Budgen and Brereton (2015), with the main stages shown in Figure 1. This section outlines the protocol of our SMS, including the research questions used to frame the study; the search strategy, e.g. which search strings, databases, and inclusion and exclusion criteria were used; and the rules for extracting data and classifying primary studies.

Research Questions
The following research questions were formulated to explore the areas of usability that contribute to the effectiveness and success of e-learning among students, staff and lecturers in universities: RQ1. What are the main attributes that have been used to assess the usability of e-learning platforms? RQ2. Which usability issues that have been identified in e-learning platforms? RQ3. What methods/techniques are used to evaluate the usability of e-learning platforms? RQ4. Which e-learning platform(s) has been evaluated? RQ5. What was the level of study of the sample used in each paper? RQ6. Which data analysis methods have been used? RQ7. What data analysis tools have been used?

Search strategy
The search strategy aims to identify the most relevant literature related to the study area, focusing on research within articles, papers and journals Kitchenham and Charters (2007). The construction of the search strings followed the steps described by Brereton et al. (2007) and (Kitchenham, Budgen and Brereton, 2015 This resulted in the following keywords used in this search: ("Usability" OR "Usable" OR "ease of use" OR "user experience") AND ("E-learning" OR "Distance learning" OR "Distance education" OR "Elearning" OR "electronic learning").
The digital libraries used were the Institute of Electrical and Electronics Engineers (IEEE) Xplore Digital Library, Association for Computing Machinery (ACM) Digital Library, Google Scholar (search engine), Wiley Online Library, and ScienceDirect.
The search was limited to the period January 2002 to December 2016. The main reason for this was that e-learning platforms have been consistently updated and therefore versions before 2002 would not be representative of the current state of the art in the area. Moreover, the evolution of online technologies has had a significant effect on the delivery methods for e-learning courses, which again makes studies before 2002 less relevant.
To evaluate the validity of the search strings, ten key papers were identified from the relevant literature. An initial pilot search was then undertaken using the proposed search strings to test whether these key papers would be returned in the results. The search strings were then adjusted dependent on the results and each database's particular search criteria.

Search process
The full search strings were then inputted into each of the chosen digital libraries. All papers were downloaded based on their titles at this stage. The details of all the returned papers were then imported into the Mendeley software, helping the researcher to remove all instances of duplicate papers from different digital libraries. Next, the inclusion and exclusion criteria outlined in the following section were applied. This is where the papers were successively screened based on their relevance for the current review.

Inclusion and exclusion criteria
The first author carried out an initial selection process by applying the criteria to the title of all downloaded papers. If it was not clear whether the paper complied with an inclusion or exclusion criteria, it was included for further screening, which involved reading the abstract for each study and if needed, the introduction and conclusions. In some cases, it was necessary to read the full text to decide whether the paper was to be included in this study. The full details are in Tables 1 and 2. Papers presenting usability attributes other usability problems. Publications not directly related to our topic. 4 Where the data analysis process is not presented. 5 Publications related to the evaluation of materials. 6 Non-reviewed literature. 7 Study sample, which concerned about disabled users.

Data extraction strategy
This section outlines the collection of data from each of the included papers. A spreadsheet was created to store the extracted information from each of the included studies with each row representing one article, enabling further comparison and analysis. Meta-data was collected from each paper, such as the title of the paper, the authors, publication year, place of publication, and abstract (see Table 3). To achieve the objectives of this SMS, more specific data, for instance, the usability attributes considered (based on those specified by Nielsen [1993] and the International Standards Organization (ISO) [1998]), the method used, and the sample size, was needed to answer the research questions. The full details are in Table  3 below. To validate the data extraction process, the protocol was piloted with the two authors and an expert in SMS authors, who compared data extraction results across ten papers until a suitable level of agreement was met.
The first author of this paper then extracted the data for the 61 publications included in the study.
To be fully satisfied that the information gathered was accurate, all included papers were read in full at this stage.

RESULTS AND ANALYSIS
A total of 7,767 hits resulted from searches using the specified search strings on Google Scholar and the chosen databases, ACM Digital Library, Wiley Online Library, IEEE Xplore, and ScienceDirect. The methodology described in section 3 was then followed with 199 papers downloaded from Google Scholar and the chosen libraries based on the titles of the publications found.
A secondary selection was then made based on reading the abstracts, introductions and conclusions and applying the inclusion and exclusion criteria. This resulted in 53 failing to meet the inclusion criteria and 146 remaining papers that needed to be read in full (and the inclusion and exclusion criteria applied). Following this process, a total of 61 articles were included in the study (32 conferences papers and 29 journal articles, see Figure 2). The following table shows the number of papers found and included from each of the chosen libraries. The majority of the papers were downloaded from Google Scholar and only three papers were from Wiley Online Library. Figure 2 shows the distribution of papers over years and the place of publication.  The following section reviews the primary results of the mapping study as well as the potential limitations. It provides the answer for each research question.
Results for RQ1 "What are the main attributes that have been used to assess the usability of elearning platforms?". Table 5 illustrates the main usability attributes that have been explored by the primary studies. Effectiveness was the main attribute that has been investigated with 23 studies, followed by satisfaction (19), then efficiency (17). However, there was less focus on learnability and memorability, with 14 and 8 papers respectively. Perceived ease of use was explored in 14 studies and perceived usefulness in 11 papers. However, 12 papers did not specify which usability problems they studied.
We found that there are more papers focusing on effectiveness and satisfaction when evaluating elearning systems. However, only a few papers considered error and memorability, highlighting the potential need for more work in this area. Results for RQ2. "Which usability issues that have been identified in e-learning platforms?". Although Nielsen and the ISO identified six main usability attributes (used in RQ1), there are other, more specific usability problems that affect the UX on elearning platforms, which are also useful to classify. Table 6 shows a number of common usability problems that have been identified. Information quality was the most common problem found with eight studies, followed by attitude to use the system (7) and navigation (6). Helpfulness was found in four studies. However, 22 of the 61 papers did not specify which usability problems they encountered. Results for RQ3 "What are the methods/techniques used to evaluate the usability of e-learning platforms?". Table 7 below shows the usability evaluation/testing methods that were used in the included papers. A questionnaire was the main method used in 50 studies, followed by interviews (12) and observation (9). Focus groups were used in five studies and think aloud in six studies. HE was used in three papers and eye tracking in two.
Although a questionnaire was the main overall method used, 16 studies combined two methods or more, e.g. interview and observation. Specific types of questionnaire included the System Usability Scale (SUS), the Software Usability Measurement Inventory (SUMI), and the web-based learning and usability questionnaire (WLUQ). However, a number of studies did not give details about the type of questionnaire or questions they used, something that would be useful for future studies. The table below shows a breakdown of the types of questionnaire that were used. SUS was used in seven studies, SUMI was used in four studies, and each of the other questionnaires was used in one study. Results for RQ4 "Which e-learning platform(s) has been evaluated?". Table 9 shows that bespoke elearning systems were evaluated in 20 papers and that the term LMS was used to describe platforms in 13 studies. Moodle was used in 12 studies; however, it is worth noting that some studies used both LMS and Moodle (studies 21, 50 and 51). The generic terms VLEs (7), web-based platforms and course management system (CMS) were used in three studies. However, 17 studies did not mention which platform they tested. Results for RQ5 "What was the level of study of the sample used in each paper?". The table below classifies the participants in the study based on their level of study. The users in 27 papers were undergraduate students, followed by master's students with 14, PhD students with six, expert users with three, and administrators with two.
Many studies emphasise that the participants' level of education may affect the usability of the elearning platforms, so it is concerning that 23 papers did not mention the level of study of respondents. Overall, undergraduate students were the main users to test the systems, with very few studies targeting expert users and administrative staff.

Users level of study
Results for RQ6 "Which data analysis methods have been used?". The table below shows software that has been used to analyse the results from the usability evaluations/tests, such as SPSS (9), LISREL (2), and two other tools, Tobii Studio and QSR Nvivo, each of which was used in one study. However, the majority of papers did not indicate which software they used to analyse their data.  Interestingly, there were no papers that used any form of automated usability testing, which can often be applied in the early stages of usability testing.

LIMITATIONS
The limitations of this study relate to misclassification, publication bias, selection bias, and inexactness in data extraction. SMSs suffer from the common issue of publication bias, i.e. negative results may not be published or cited, and positive results may be published faster than negative results (Kitchenham, Budgen and Brereton, 2015).
Selection bias refers to the misrepresentation of statistical analysis due to criteria that have been used in the selection of published papers (Fernandez et al., 2011). To try to mitigate this threat, detailed inclusion/exclusion criteria were determined and validated by the authors. A related issue is that during data extraction, it is possible that information is overlooked or misclassified by reviewers. To mitigate this threat, the extraction and classification of the studies were conducted by the first author, with cross-checking by the two authors and an expert in SMS on ten papers until the protocol and extraction process was agreed upon and standardised.

CONCLUSIONS AND FUTURE WORK
This SMS aimed to answer seven research questions concerning the usability of e-learning in higher education. We have presented the current state of the art according to 61 papers containing primary studies. Although there are two similar previous SMSs in this area, they each have a different focus and research questions.
The findings of this study suggest that adding other usability factors, such as navigation and attitude, to the standard usability attributes and using a combination of evaluation methods, such as focus groups and questionnaires, would be useful. There seems to be reliance on questionnaires, which miss tacit and semi-tacit knowledge but need to link to the literature in this area (Rugg and Petre, 2007). Moreover, some studies did not provide certain vital information, for example, the platform they tested, the participants' background, and the tools they used to analyse their data.
This mapping study gives a clear idea of the usability of e-learning to guide future researchers in this area. We observed that a few studies used the methods focus group, heuristic evaluation, and eye tracking, as well as some usability factors, for instance perceived usefulness, memorability, and error. In the previous paper by Harrati et al. (2016), they expressed the opinion that it is possible to investigate the usability factors of memorability and learnability, and their relationship with ease-of-use over time by analysing the participant's usage logs.
Our mapping study has demonstrated that such an approach has not been used to date in primary studies.
We hope that our findings will identify several research gaps for future research, such as which of the most appropriate usability techniques can be applied to evaluate e-learning platforms.

ACKNOWLEDGMENTS
We would like to acknowledge Azzaytuna University and the Libyan Ministry of Higher Education and Scientific Research for the PhD scholarship provided for Khaled Abuhlfaia. We would also like thank our colleague Professor Pearl Brereton for her useful comments and suggestions. Chauhan, J., Batbayar, K., Sharma, R., Sharma, D., Popli, D., Kumar, N., & Goel, A. (2015).  (2)