Qualitative Adaptation: Informing Design for Risk-based Decision Making

Research on decision making during risk and uncertainty facilitates risk-based decision making by understanding techniques decision makers use to arrive at informed decisions. Approaches to the research usually involve a mix of cognitive techniques for information discovery and sense-making; these were methodologically not intended to inform design. We detail our experience in applying qualitative techniques to elicit persona characteristics from risk-based decision making data.


INTRODUCTION
Risk-based decision making (RBDM) is an endeavour to make informed decisions under conditions of risk and uncertainty.The conditions may be the result of time limitations, insufficient information, and dynamic environments to mention a few.Understanding how these decisions are made is a step towards understanding how to facilitate RBDM through design.
To achieve this, the research aimed at identifying how persona characteristics may be grounded in risk-based decision making data.

APPROACH
A data elicitation exercise was carried out in Japan with a group of 30 industrial participants undertaking a cyber security course.Cyber security was chosen as its activities exemplify RBDM in action.Participants were drawn from 11 different sectors including Transport, Oil, Electricity and Manufacturing, and with experience ranging from 1 to 20 years.

Participants
were trained on the Risk Rationalisation Process (RRP) (M'manga, 2018), our adaptation of Boyd's (1996) Observe Orient Decide Act (OODA) and provided with a cyber security decision making scenario containing elements of risk and uncertainty.From this, they were asked to provide rationales for their decisions using RRP.Responses were thematically clustered into risk rationalisation variables using Nvivo, which were then exported to a spreadsheet and categorised as behaviour variables following the Persona Case technique (PCT) (Faily and Flechais, 2011).As a final step, the spreadsheet was imported into CAIRIS (Computer Aided Integration of Requirements and Information Security), a tool supporting the Persona Case technique for modelling personas (Faily, 2018).

THEMATIC ANALYSIS
In this section, we present a detailed overview of the thematic analysis approach taken during the research (illustrated in Figure 1).Details here are independent of tool usage and aim at highlighting the methodological approach.
The first round of thematic analysis was incorporated into the study's response form by dividing it into eight steps of RRP.Participants were expected to present the rationale for their decision in accordance to the eight steps.For example; Situation assessment information was provided for step 1 and Option validation information for step 7.
After responses were grouped into the eight RRP steps (themes), the second round of thematic analysis aimed at identifying sub-themes to form risk rationalisation variable.For example, the following risk rationalisation variables were culled from Situation assessment; analysing security policies, analysing security trends, seeking clarification, analyse past security trends, and monitoring network activity.Due to the translated nature of the data (discussed in Section 4.1), our approach to coding was based on similarities in statements (conveying meaning) as opposed to similarities in words (Saldana, 2015).For example, the sub-theme analysing security policies was based on statements such as "Identify information security policies", "confirm with the department which developed the information security policies", and "identify the history of information security policies" Unlike Grounded Theory (Corbin and Strauss, 2008), where a core category emerges during the final round of coding, our adoption of PCT (Faily and Flechais, 2011) for the third round of thematic analysis was meant to use the risk rationalisation variables in an argumentation (Toulmin, 2003) form that represents behaviour variables.To do this, the risk rationalisation variables were categorised as risk assessment activities, validation activities and goals.For example, "analysing security policies" was argued by the goal "provide security advice" and validation activity "policy effectiveness".

Cross-language research
Cross-language research occurs when there is a language barrier between researchers and participants (Squires, 2009).The language barrier was the first research challenge faced as the study was undertaken in Japan, with non-English speaking participants, and an English speaking main researcher.
Cross-language research is rare in computing but common in health and ethnographic studies where the objective includes studying cases related to minority groups (Wallin and Ahlstrom, 2006).We followed this approach due to the lack of local cyber security analysts willing to participate in research due to privacy and others issues; this issue was previously reported by Kotulic and Clark (2004).On the other hand, the study in Japan benefitted from multiple participants in a single place, with diverse experiences from multiple industrial sectors.
To tackle the language problem, we worked with a Japanese university; they provided a Japanese secretary proficient in the two languages.The secretary was a regular translator at the university, held a university degree, and possessed twelve years working experience in the USA.Because our data collection approach took the form of written responses to the decision making scenario, the secretary was given the responsibility of translating both the scenario and its questions from English to Japanese, and vice-versa for the responses.The secretary had limited cyber security knowledge, but we believe that the use of a single translator increased the secretary's immersion in the data and improved contextual translation.A Japanese cyber security lecturer assisted the secretary in cases where the security terminology was not understood, and the main researcher was contacted when English words were unclear.For example, the main researcher was asked to clarify the term "rationalisation" in relation to the scenario.
The disadvantage of the approach was that the data immersion process -essential in thematic analysis -was slow due to the main researcher's reliance on translation.Becoming lost in translation is inevitable in cross-language research, and we encountered a few false positives.For example, one of the participants professed experience as a SOC (security operations centre) analyst in a printing company, which we believed was an error in translation.Verification of this experience showed that this was a false positive.

Methodological differences
The second challenge the research faced, relates to the methodological gap between techniques used in decision making research and techniques used in specifying design requirements.Traditionally, decision making research has followed a descriptive to prescriptive approach; this is the modelling of how decisions are made to recommend facilitation techniques.In contrast, the approach to design is to elicit requirements; these are then modelled to inform design decisions (Fischer, 1991).
While there is undoubtedly more modelling in the two approaches than our simplified explanation, the main point is the difference in model use.Decision making models are primarily aimed at understanding the problem domain, while design models are aimed at supporting the solution.In a perfect world, this would simply mean that one serves as input to the other, however, methodological differences present incompatibilities.For example, RRP aims at understanding the rationale behind risk-based decision making; decision making may, however, be influenced by biases such as the overconfidence in one's own abilities (Kahneman, 2011).For example, none of our participants identified personal inabilities (experience or training) as a reason for erroneous assumptions in their responses to the scenario.While decision making research may bring this to light, relating it to design may not be as straightforward.
This problem is evident in decision making research, where design recommendations are usually presented in a textual format with no clear relation to design models (Groenewald et al., 2017;Gerber et al., 2016).We are, however, not stating that our approach is an all-encompassing solution, but rather that it provides the means to relate decision making to design models through the RRP to Persona mapping (Figure 1).

Seamful tool support
The final significant challenge we encountered concerned tool support in relation to seamful interaction.Seamless interaction is explained as a tool's ability not to intrude on one's consciousness but allows the user to focus on the task at hand (Weiser, 1994).Conversely, seamfulness is the lack of tool invisibility.While invisibility refers to the seamless integration of system components or their interaction (Chalmers, 2003), our experience refers to the work overhead resulting from passing data between tools during the research.
As indicated in Section 2, we used Nvivo (version 11) to thematically analyse the participant's responses to risk rationalisation variables.The findings were then exported to a spreadsheet (PCT template) for behaviour classification.However, in Faily and Flechais (2011) proposal, PCT relied on the qualitative data analysis tool ATLAS.ti;licensing restricted our research to Nvivo.
We identified that although Nvivo presents several data export functions, none adequately satisfied the PCT requirements.For example, one of the main features of PCT is the use of concept relationships for persona characteristics argumentation.However, it was not possible to export concepts and their relationships from Nvivo to a single spreadsheet.In light of this, we manually populated the spreadsheet.This, in turn, was a laborious process due to a large amount of data collected.Our experience illustrates how differences in Computer-Assisted Qualitative Data Analysis Software (CAQDAS) may affect qualitative research and lead to a lack of seamless interaction in tool use.

CONCLUSION
In this paper, we presented our experiences adapting qualitative methods to inform persona design for risk-based decision making.We presented cross-language research, methodological differences and seamful tool support as our three main areas of concern.The concerns raised are however not limited to qualitative research, but the wider research community as a whole.In summary, the areas of concern raise some of the following questions; what approaches should researcher take when subject matter experts are unavailable for empirical research?How should cross-discipline research approaches be adapted to provide meaningful findings?How should tool dependent research be approached when recommended tools are unavailable?

ACKNOWLEDGEMENT
The research was funded by Bournemouth University studentship DSTLX1000104780R_BOURNEMOUTH_PhD_RB DM, with the initial collaborative meeting between UK/Japan researchers facilitated by support from the Great Britain Sasakawa Foundation.We are also grateful to DSTL for their sponsorship of this work.