Evaluating the Usefulness of the Adapted Thematic Analysis in the Process of Design

A number of HCI studies reported thematic analysis method as a useful tool for evaluating existing technologies and informing design of new technologies. Whilst these studies reported on the resulting designs, they lacked detailed account about the use of the method itself or how the output of the method was linked to proposed designs. To explore this ‘knowledge gap’, thematic analysis method was applied to a data set resulting from semi-structured interviews conducted with five families. The adaptation of the thematic analysis approach led to a new ‘output’ in which findings were structured into a Design Map consisting of Categories, Themes and Codes. The usefulness of this Design Map in the process of design was evaluated in a study conducted with participants studying design at an undergraduate level. This paper describes the rationale for using Lean UX approach to refine the aims of the study and reports results from the Exit Questionnaires aiming to elicit participants’ opinions about the use of a Design Map in the process of design.


INTRODUCTION
Thematic analysis, drawn from different fields (Joffe and Yardley, 2004;Braun and Clarke, 2006;Boyatzis, 1998;Fereday, and Muir-Cochrane, 2006), has been reported as a useful data analysis method in recent HCI research. These reports lacked detailed explanations about the use of the method itself, or how it linked to resulting design solutions, so a decision was made to use new data, gathered through semi-structured interviews with families with primary school children (called 'Introductory Interviews'), and explore the method thematic further.
After different approaches to the method were examined, the 6-Phase approach outlined in Braun and Clarke (2006) was chosen. The first round of analysis resulted in a high-level thematic map accompanied by the analytic narrative, the outcome useful in HCI research where the aim is to present an overall summary of findings. The low-level design insights, useful for informing the design of new technologies or improving the functionality of existing technologies, were, however, completely lost in the analysis process. In the second round of analysis, the 6-Phase approach was adapted, resulting in the Introductory Interviews Design Map consisting of Categories, Themes and Codes, with the findings being structured in a way that provides a high-level overview, whilst also preserving lowlevel design detail. To explore the usefulness of the Introductory Interviews Design Map in the process of design, a study with undergraduate design students as participants was conducted.

Exploring the TA 'knowledge gap'
A number of HCI studies (Pykhtina et al., 2012;Money et al. 2008;Tanaka et al., 2012;Toth et al., 2012;Güldenpfennig, Reitberger and Fitzpatrick, 2012) reported thematic analysis method outlined in Joffe and Yardley (2004) and Braun and Clarke (2006), as being useful for evaluating existing technologies and informing design of new technologies. Whilst these studies reported on the resulting designs, they lacked detailed account about the use of the method or how the output of the method, a thematic map accompanied by data extracts embedded within the analytic narrative, besides making an argument in relation to the research questions, linked to the actual proposed designs. To explore this 'knowledge gap', the thematic analysis method outlined in Braun and Clarke (2006) was applied to a data set resulting from an exploratory study investigating family dynamics and the use of technology within families 2 with primary school children. The data was gathered through semi-structured interviews with five families (called Introductory Interviews) and analysed following the 6-Phase approach outlined in Braun and Clarke (2006). Through this process, it was discovered that grouping themes into higherlevel categories in order to construct a thematic map (in Phase 4), was useful for representing a general overview in social science studies, though not as useful in the research where the aim was to gain insights into participants' behavior and the use of technology, so these insights can be used to improve functionality of existing technologies or inform design of new technologies (Brown and Stockman, 2013). The resulting thematic map was deemed too abstract to guide design decisions, whilst the analytic narrative, though underpinned with the data extract relevant to the research question, was lacking low-level detail necessary to drive design.

Adapting TA and producing a Design Map
Instead of producing a thematic map, at the end of Phase 4 of the analysis, a detailed account of the family dynamics and the use of technology within families with primary school children was initially structured into a 'thematic frame', containing a list of Themes captured across all five families. These Themes were further grouped into higher-level Categories forming a Design Map ( fig.1). Each Category name in the Design Map was capitalized and highlighted in green font to enable it to stand out. The number of Themes each Category contained was listed in the bracket next to the Category name, with every Theme contained within that Category being listed underneath.

Figure 1: Introductory Interviews Design Map
The number of Codes contained within each of the Themes and a number of families talking about the Themes were added in a bracket next to the name of the Theme they were related to. Each Category was placed in a text box with a thin black border to enable visual separation between the categories and easier navigation through the Design Map. A list of Codes contained within the Themes was placed in a separate document, using the italic capitalised green font for the names of the Categories, bold black font for the names of the Themes and text indent before each of the listed Codes to enable easier visual navigation through the document.

Recruiting participants
29 participants (12 male and 17 female), from the Bachelor of Fine Arts (Hons) programme at the University of Malta, were recruited for the study. A lecturer from the programme, a native Maltese speaker, assisted with the study by providing insights into the tools and techniques the participants were taught (which led to some of these being incorporated into the study) and by offering bilingual clarification during the study.

Using Lean UX approach to declare user assumptions and identify priorities for testing
The study was driven by two hypotheses resulting from the adaptation of the thematic analysis approach: Solving Usability Issues, which proposed that "Having a list of low-level codes providing detailed insights into family members' behaviour involving existing technologies would enable researchers to identify usability issues and improve the design of these technologies" and Design Innovation, which proposed that "Having a list of low-level codes providing detailed insights into family members' behavior not involving existing technologies, would enable researchers to inform the design of novel technologies to support this behaviour".
The aim of the study was to investigate if the Introductory Interviews Design Map (the name of which was shortened to a 'Design Map' for the purpose of the study), could assist the participants with the completion of two tasks, related to those hypotheses, and elicit their opinions about the ways navigation through Categories, Themes and Codes could be improved. As hypotheses were too broad to test within the timeframe allocated per each group of participants (2 hours), the Lean UX approach (Gothelf and Seiden, 2013), previously explored in the context of software agency (Liikkanen et al., 2014) and learning (Dodero et al., 2014), was used to identify the parts of the hypotheses that were a priority to test.
First, a Business Assumptions Worksheet (Constable, 2015) was used to declare the following six User Assumptions: Then, the six Assumption Statements were created to answer these assumptions and a Prioritisation Matrix (Gothelf and Seiden, 2013, p.22), was used to determine the 'highest priority' assumptions to be tested (ii, iv and vi). Following this, three Hypothesis Statements: HSA2, HSA4 and HSA6, were created for each of these assumptions (Gothelf and Seiden, 2013). They each consisted of two parts, the first part outlining what is believed to be true for the Design Map and the second part outlining how this was going to be assessed. Hypotheses Statements HSA4 and HSA6 were still too broad, so had to be broken down into the following 'subhypotheses': HSA4sub1, HSA4sub2, HSA6sub1 and HSA6sub2.

Designing tasks and questionnaires
The study was designed in a form of two workshops, each consisting of two design tasks: Questions aiming to test the two subhypotheses derived from Hypothesis Statement HSA4 (HSA4sub1, HSA4sub2), which formed the basis for the two tasks, and evaluate the participants' ability to identify usability issues (Q11) and possibilities for design innovation (Q12) within the workshop materials provided.
14 question statements used 5-point 'Level of Agreement' Likert-type scale with the verbal qualifiers: 'Strongly Agree -Agree -Neither Agree nor Disagree -Disagree -Strongly Disagree' (Vagias, 2006), frequently used in social sciences research (Rohrmann, 2007). One question statement used 5-point 'Level of Difficulty' Likerttype scale with the verbal qualifiers: 'Very Easy -Easy -About Right -Hard -Very Hard'. Three questions were formulated as 'free text', allowing the participants to write their own comments by hand.

Conducting the study
Participants worked in teams of 4 or 5, to complete design tasks using a Design Map (printed on one page), A0 sheets of paper, colored pens, highlighters, post-it notes and a document containing a 'breakdown' of the Design Map with the full list of Codes (printed on 33 pages). At the beginning of each of the workshops, participants were shown how to navigate through the Design Map's Categories, Themes and Codes.

DISCUSSION OF RESULTS
26 out of 29 participants completed the Exit Questionnaires (12 male and 14 female). In their responses to Q17 (related to User Assumption ii and HSA2), only one of the participants noted a Design Map could not assist them with their current way of working. One participant noted a Design Map would be useful if they were "designing a new product", whilst two participants noted it would be useful for academic work like "assignments & projects" and "thesis".
The responses to questions Q11 and Q12 (related to User Assumption iv, HSA4sub1 and HSA4sub2), revealed that the majority felt they were able to identify usability issues (84.62%) as well as possibilities for design innovation (88.46%) using a Design Map.
The responses to questions Q2-10 (related to User Assumption vi, HSA6sub1 and HSA6sub2) further revealed that the majority also understood the relationship between Codes, Themes and Categories in the Design Map and the workshop materials (84.62%); were able to find Categories, Codes (73.08%) and Themes (69.23%) they were interested in, understood the meaning of Themes (84.62%) and Codes (73.08%) and felt their team members had the same understanding of Themes (73.08%) and Codes (73.08%) as they did.
The responses to question Q16 ('What research tools and methods do you currently use to inform your design process?'), revealed a list of 22 different tools, techniques and strategies participants used to inform the design process, with 'Mind Maps' and 'Thinking Hats' being used by most.

FUTURE WORK
This study, testing User Assumptions ii, iv and vi with participants studying design at an undergraduate level (and as such, being relatively inexperienced as designers), confirmed the usability of a Design Map.
The responses in the Exit Questionnaires also confirmed that research findings structured in a way that provides a high-level overview (through Categories and Themes) whilst also preserving low-level design detail (through Codes), have a potential for being useful in the process of design.
As a follow up, a study comparing the two thematic analysis approaches, the 6-Phase approach (Braun and Clarke, 2006) and the adapted thematic analysis approach (Brown and Stockman, 2013), will be conducted with a new group of participants studying design at an undergraduate level, so that usefulness of both approaches, for improving the functionality of existing technologies or informing the design of new technologies, can be evaluated.
Some of the responses in the Exit Questionnaires indicated the need for adding more in-depth questions about participants' research and design background, as well as questions exploring the correlation between the participants' understanding of the workshop materials and the time spent working within the same team. This will be taken into account when designing future iterations of the Exit Questionnaire.

ACKNOWLEDGMENTS
This research was made possible thanks to EPSRC DTA grant; Chris Porter from the Faculty of ICT, Vince Briffa and Christine Porter-Lofaro from the Faculty of Media and Knowledge Sciences (MAKS) at the University of Malta, as well as all the undergraduate students who participated in this study.