A Toolkit for the Usability Evaluation of Digital Health Technologies

Introduction: NHS digital applications are often assessed against the NHS Digital Technology Assessment Criteria (DTAC). However, Small to Medium Enterprise (SME) developers often find DTAC itself ‘heavy going’. One criterion of DTAC is usability and SME developers need guidance on how to meet this criterion. My PhD is a four-part study to develop such guidance. Methods: (i) a scoping review of the methods of usability testing, (ii) development of a novel way of employing a usability scale, dividing it into discrete chunks to prevent “questionnaire fatigue”, (iii) development of a toolkit, (iv) application of the toolkit with SMEs. Results: (i) The scoping review identified 133 papers. (ii) The simpler way of administering the usability scale gave comparable results. (iii) I developed a toolkit that gives guidance on which testing method to employ, the number of participants, and the types of output. (iv) I have used the toolkit to guide three SMEs (two more in progress) in testing digital health technologies. Conclusions: The toolkit will aid SMEs in meeting DTAC criterion for usability. Further research is needed to refine and validate the toolkit and develop methods for evaluating new types of digital health applications such as virtual reality and voice interfaces.


INTRODUCTION
I am a part-time PhD student and I am also currently working as a Research Associate in the Ehealth Productivity and Innovation in Cornwall and the Isles of Scilly (EPIC) project. The EPIC project aims to grow the digital health ecosystem in Cornwall and the Isles of Scilly by assisting Cornish Small and Medium Enterprises (SMEs) in developing digital health applications (EPIC, 2017). This is done by providing academic support, innovation funding, knowledge exchange and business support. Both my work and PhD research involve usability testing of digital health applications. As part of my work I am supporting SMEs by providing guidance on usability testing. To provide the best possible guidance, I conducted a scoping review on the methods of usability testing in the development of digital health applications. I have completed and published the scoping review (Maramba et al, 2019).
I am now using the results of the scoping review to inform the usability testing being done by SMEs receiving support from the EPIC project. One example is the usability testing I performed of the MyPreOp web-based preoperative assessment system . Based on this and other experiences guiding SMEs in conducting usability testing, I have started designing a toolkit to guide the process of usability testing of digital health applications. I am using the toolkit together with SMEs to plan and conduct their usability testing and refining the toolkit collaboratively. The aim is for the toolkit to be an easy to use guide for digital health application developers to meet the criterion for evidence of usability, as set by the NHS Digital Technology Assessment Criteria (DTAC).

BACKGROUND
Digital health technologies are being promoted by the National Health Service (NHS) in England as part of the new Long Term Plan. As stated in section 5 of the plan, the aim is for digitally-enabled care to go mainstream across the NHS. To support this, the plan proposes to "work with the wider NHS, the voluntary sector, developers, and individuals in creating a range of apps to support particular conditions" (NHS, 2019) . This aim acknowledges the phenomenal growth in the eHealth sector, with an estimated 325,000 mobile health apps available for download in 2017 (Pohl, 2017).
In line with this, the Long Term Plan states the need to work with the wider NHS, the voluntary sector, developers, and individuals in creating a range of digital health apps to support particular conditions.

DTAC usability requirements
Existing guidance for usability evaluation activities for digital health technologies include the NHS DTAC (DTAC, 2022)  The requirements of DTAC for usability are the following: • Does the developer engage users in the development of the product? • Are all key user journeys mapped to ensure that the whole user problem is solved, or is it clear to users how it fits into their pathway or journey? • Does the developer undertake user acceptance testing to validate usability of the system?

Problem Statement
SMEs developing digital health applications need to generate evidence that will satisfy the DTAC with regards to usability. However, in their present form, the criteria do not adequately describe how to generate this evidence. And whilst larger developers may have in-house expertise in usability evaluation, the smaller developers may not.

Research Purpose and Questions
The purpose of this research is to investigate how best to provide guidance in the practice of robust user experience evaluation amongst developers of digital health technologies.The following research questions were developed: 1. What are the current methods for usability evaluation of digital health applications as found in the published literature?
2. What are the enablers and barriers to embedding meaningful user experience evaluation activities amongst developers?
3. How can developers be guided in conducting meaningful user experience evaluation activities in the development of their applications?

Usability testing of digital health applications.
I used the results of the scoping review to guide usability testing of digital health technologies. I also developed a novel way of employing a usability scale, dividing it into discrete chunks to prevent "questionnaire fatigue". This method was employed in the usability testing of a computerised preoperative assessment system. The use of this method did not affect preoperative assessment completion times and gave satisfactory results (n=1400 responses).

Barriers and Enablers Identified
The following barriers and enablers were identified from the literature and from qualitative work done with the SMEs: Barriers: SMEs having little knowledge of user experience and usability testing methods. Little or no access to patient groups for co-design and user testing. Enablers: Support from academic institutions providing knowledge transfer about methodologies. Formation of networks of stakeholders in digital health through projects like EPIC.

Ongoing and future work
Based upon the findings of the literature reviews, existing evaluation frameworks for other domains, and usability studies personally conducted, I developed a framework for usability testing of digital health applications. It specifies the kind of tests that would be suitable for various stages of development of the application, as well as the objectives, the number of participants needed, and the approximate time involved. A summary version of the proposed framework is included as Appendix 1.

CONCLUSION
The toolkit intends to guide a developer in gathering evidence to satisfy the requirements of DTAC in the areas of usability. Further research needs to be done on: 1. Validating the toolkit in successfully meeting the DTAC criteria on usability.
2. Further updating the toolkit to include new evaluation methods and new interfaces, for example, virtual reality and voice interfaces.
3. Creating an interactive, digital version of the framework, either as a web or mobile app.

Introduction
The following is a framework for usability testing of digital health applications. It specifies the kind of tests that would be suitable for various stages of development of the application, as well as the objectives, the number of participants needed, and the approximate time involved. It is followed by an explanation of the test methods.