Towards measuring user engagement in internet interventions for common mental disorders

Citation for pulished version (APA): Kleine Stegemann, S., Ebenfeld, L., Thiart, H., Berking, M., & Funk, B. (2012). Towards measuring user engagement in internet interventions for common mental disorders: Workshop on People, Computers, and Psychotherapy. In Proceedings of HCI2012 People & Computers XXVI: The 26th BCS Conference on Human Computer Interaction (Electronic Workshops in Computing (eWiC)). BCS The Chartered Institute for IT.


INTRODUCTION
Internet-based interventions for treatment of people with common mental disorders have gained much attention over the past years (Barak, Hen, Boniel-Nissim and Shapira (2008); Proudfoot et al. (2011)). Research has shown effectiveness of such interventions for a number of disease patterns, including depression  ;Spek et al. (2007)), panic and anxiety (Meulenbeck et al. (2010); Cuijpers et al. (2009)) and stress (Lange, van de Ven, Schrieken and Smit (2004)). Furthermore, it is commonly noted that internet interventions are advantageous for users by providing high accessibility at any time as well as anonymity and protection of privacy (Proudfoot et al. (2011)).
In contrast to the "binary" concept of dropout, adherence is commonly understood as an objective measure for how intensely a user participates in the intervention (Christensen, Griffiths and Farrer (2009)). That said, there is no general agreement on a definition of adherence. Donkin et al. (2011) reported that adherence is frequently measured in terms of usage-data from the website, for example number of logins, number of modules completed and time spent online. It has also been reported that, in general, adherence is positively associated with the outcome measure (Ritterband et al. (2009);Christensen, Griffiths and Farrer (2009);Donkin et al. (2011);Hilvert et al. (2012)).
Overall, research with respect to adherence demands for a deeper understanding of that concept. Christensen, Griffiths and Farrer (2009) said that "we need to understand more about adherence in order to maximise the impact of Internet-based interventions on the disease burden associated with common mental disorders." This is not only true because adherence can be seen as one determinant of effectiveness (Hilvert et al. (2012)), but also with respect to the dissemination of internetbased interventions. This is related to the fact that interventions with low adherence will most likely not show the effectiveness that would be necessary for their general acceptance.
In the literature on internet interventions, the connection of user engagement and adherence is repeatedly mentioned. For example, Donkin et al. (2011) note that "little is known about the degree to which user's engagement matches the usage pattern for which the websites are designed". Thus, a deeper understanding of how technology engages users will contribute to our understanding of website adherence and, consequently, of behaviour change through the internet (Christensen, Griffiths and Farrer (2009);Fogg (2009)). Also, the need for engagement as a complementary measure for assessing adherence is mentioned by Donkin et al. (2011). However, despite the recognition of user engagement as an important factor for adherence, to our knowledge little work has been carried out to further explore the concept in the field of internet interventions. An example is the recent work of Doherty, Coyle and Sharry (2012) who presented a platform to provide more engaging interventions.
On the other hand, user engagement is an important concept in the discipline of human computer interaction (Peters, Castellano and de Freitags (2009)). This is related to the fact, that computers and computer programs are no longer recognised as merely functional products but rather as enjoyable artefacts that create a unified and holistic experience for the user (Kleine Stegemann and Fiore (2006)).
Adopting this notion for internet interventions is, in our opinion, crucial for their success outside the world of research. Instead of building rather functional and boring websites, we need to strive for the creation of enjoyable interventions that involve users emotionally. As O'Brien and Toms (2008) note, "successful technologies are not just usable; they engage users". This is even more true with respect to adherence and dropout.
In order to explore and understand user engagement in internet interventions, instruments to measure engagement are needed. To this end, our work strives to establish a questionnaire for user engagement in internet interventions for mental disorders. We furthermore intend to evaluate and use the questionnaire in a number of randomised controlled trials that are conducted in the European project "Kompetenztandem Gesundheits-Trainings-Online -eMen-talHealth", a cooperation of the Leuphana University Lueneburg, University of Marburg, VU University Amsterdam and Minddistrict, the Netherlands.

USER ENGAGEMENT
The literature on user engagement in human computer interaction does not give a common definition of that concept. For instance, Laurel (1993) defines it as "a desirable -even essential -human response to computer-mediated activities" and emphasises the playfulness and sensory integration in engagement. On the contrary, Quesenbury (2003) suggests that user engagement is a dimension of usability, i.e. "the degree to which the tone and style of the interface makes the product". Peters, Castellano and de Freitags (2009) reported that engagement is sometimes referred to as a process as well as as an experience or as a cognitive state of mind.
An in-depth discussion about the definition of user engagement is not part of this work and has been done elsewhere (e.g. O'Brien and Toms (2008); Peters, Castellano and de Freitags (2009)). For our research, we adopt a holistic view of user engagement that reflects the affective experience of a user, including but not limited to usability. Moreover, we follow O'Brien and Toms (2008) in their "operational definition" of user engagement as a process comprised of four stages: point of engagement, period of sustained engagement, disengagement and reengagement. (2011) reported an increasing use of quantitative methods in HCI research that goes along with a stronger focus on user experience research. In their review, they noted that many authors used self-developed questionnaires. That said, the items are often not stated in the text and questionnaires are frequently not validated.

Bargas-Avila and Hornbaek
While validated instruments for measuring user experience have been developed (see for example Hassenzahl (2004) and Lavie and Tractinsky (2004)), they commonly address user engagement only in part or not at all. Contrary, in their recent work, O'Brien and Toms (2010) have developed a questionnaire that is specifically focused on user engagement. The published version consists of 31 items that cover the following factors: focused attention, perceived usability, aesthetics, endurability, novelty and felt involvement. A first study was conducted to select the items from a larger set that has been derived from an extensive literature review. Item selection was performed through exploratory factor analysis. The selected items have then been validated in a second study by means of a structural equation model based on the author's hypotheses.
To our knowledge, the questionnaire of O'Brien and Toms (2010) is the only fully published scale aimed directly at user engagement. So instead of adding another self-developed questionnaire to the research landscape, we intend to build on their work. While the items are verbalised for e-commerce, they are in essence domain-independent which facilitates adaptation to the field of our research.
Adaptation and translation will be carried out in a multi-step process. First, items of the questionnaire will be adapted to internet interventions by a psychologist and a computer scientist from our group. Two independent versions will be created and discussed in order to come up with a agreedupon adaptation of the questionnaire. If necessary, external experts will be involved in the discussion.
Because we intend to use the questionnaire in German randomised controlled trials, the adapted questionnaire will be translated and customised to German culture according to the recommendations made by Beaton, Bombardier, Guillemin and Ferraz (2000) and Wild et al. (2005). The German questionnaire will be evaluated in several trials spanning different disease patterns including depression, panic and work-related stress. In order to get sufficient evidence for the evaluation and interpretation of the results, we calculate a minimal sample size of 200 participants.

RESEARCH GOALS
Our primary research goal is to provide a validated instrument for measuring user engagement in internet interventions. Furthermore, we intend to gain a deeper insight in how user engagement affects adherence and in turn therapy success. From the literature (e.g. Christensen, Griffiths and Farrer (2009);Donkin et al. (2011), our hypothesis is that user engagement is positively associated with adherence. To validate this assumption, we will record traditional measures for adherence as well and examine the relationship between these variables and user engagement as reported by the users. In addition, we will examine if and how user engagement affects outcome measures directly. Finally, the different factors covered by the questionnaire will be analysed with respect to adherence and outcome measures.
There are a number of limitations we are currently aware of. First, it is unclear if and how therapy success will affect the subjective sensation of user engagement of the participants, thereby introducing the possibility of a bias. Second, while literature provides evidence for the role of engagement with respect to adherence, it is unclear if these concepts are rather supplemental or overlapping. However, we hope to clarify this question with our research. Lastly, the experience generated by interventions used in our trials will likely be similar because they are all based on the same software solution. As a consequence, the collected data might not be sufficient to compare different levels of user engagement.

CONCLUSION
In this paper, we discussed the need for a deeper understanding of user engagement in internet interventions for mental disorders, especially with respect to the problem of low adherence. We outlined our project of adopting and validating the questionnaire from O'Brien and Toms (2010) in order to provide a measurement instrument for user engagement in internet interventions. The questionnaire will be used to capture data in several randomised controlled trials. From the results, we hope to get insights in how user engagement affects therapy success and to explore the relationship between adherence and user experience.