Towards an instrument for measuring sensemaking and an assessment of its theoretical features

This paper reports the result of a study that we conducted to develop an instrument for measuring sensemaking and to understand how sensemakers conceptualise the sensemaking that they are doing. To address these aims we conducted a review of how sensemaking is described in the literature and derived a series of features. These were used to construct a questionnaire, which we deployed in an experimental study in which participants performed a sensemaking task. The results help to validate the questionnaire and provide insights into how users think about the experience of sensemaking.


INTRODUCTION
Sensemaking has been described as a process of constructing an understanding when facing complex situations (Attfield and Blandford, 2009;Klein et al., 2006). In electronic environments, it can frequently involve a complex interplay of information foraging and information structuring, as well as reflection and hypothesising based on the structures created (Pirolli & Card, 2005).
Sensemaking frequently provides a context for the use of technologies for information seeking and exploration (such as information retrieval, information extraction, data mining, data visualisation etc.), as well as tools for visually structuring and reflecting on information. To take one example, Selvaraj et al. (2016) report a study of police analysts who made sense of crime data by iteratively querying information resources and using resulting information to construct elaborate link charts and timelines for the generation and testing of hypotheses. Operationalizing the concept of sensemaking within instruments for measuring it can therefore make a useful contribution to the design of such tools.
Sensemaking has been studied in various disciplines, including HCI (Russell et al., 1993), library and information studies (Dervin, 1998) organisational studies (Weick et al., 2005(Weick et al., , 1995, and naturalistic decision making (Klein et al., 2006). Various conceptions have been presented, for instance that it is a process that involves structuring and gaining insight (Pirolli and Card, 2005), or finding connections (Klein et al., 2006 We are interested in understanding how sensemakers conceptualise the sensemaking that they are doing, and also in developing an instrument for measuring sensemaking. To this end, we designed a study in which participants performed a sensemaking task.
As an experimental manipulation, participants were asked to use the information to create a visual structure or simply save the information. At the end of the task, participants were given a questionnaire, which asked them the extent to which they made sense of the information and probed them according to conceptualisation of sensemaking drawn from the literature. We then conducted a correlational analysis to understand how strongly participants associated the different conceptualisation with the idea of 'making sense' of some information.

BACKGROUND
In order to build a questionnaire of phenomena related to sensemaking, we first reviewed the literature, and included the most significant theories and definitions that cover different dimensions of sensemaking. We derived the following five dimensions:

Sensemaking as a process of gaining insight:
Sensemaking has been associated with cognitive processes such as comprehension (Klein et al., 2006) and the process of gaining insight. Pirolli and Card (2005) describe sensemaking as a process of gathering information, creating a representation (schema) of the gathered information, utilising the created representation to build insight, and using the gained insight to create knowledge or lead to action.
They summarised the sensemaking process with insight as one of its main outcomes (see figure 1).

Information
Schema Insight Product Figure 1: Stages of sensemaking process: Pirolli & Card (2005) 2.2 Sensemaking as a process of understanding connections: Sensemaking has been also defined as "a motivated, continuous effort to understand connections (which can be among people, places and events) in order to anticipate their trajectories and act effectively" (Klein et al., 2006). This can be seen, for example, in sensemaking efforts which involve externalizing information by gathering items (physical or digital) and arranging them into some coherent pattern with tangible connections (Kolko, Jon, 2010). The process of finding relationships among entities in large amounts of data has been described as a central part of the sensemaking process (Stasko et al., 2008). When attempting to gain insight into a subject, among other different procedures, sensemaking is considered a key process that people perform(Yi et al., 2008).

Sensemaking is about Structuring:
Sensemaking has been defined as, "the process of searching for a representation and encoding data in that representation to answer task-specific questions" (Russell et al., 1993). It is argued that creating representations to filter and understand data is central to sensemaking. During the process of sensemaking, people create representations, whether internally, externally or both. These representations then guide their decisions about what to look for and what else to include (Faisal et al., 2009). In the data -frame theory (Klein et al., 2007), sensemaking is described as a process of framing and reframing; where a frame is a structure used to filter the data and data is used to update a frame. The process of structuring can occur internally, as emphasised in the data-framemodel, or externally, as discussed by Russell et al., (1993) and Pirolli and Card (2005).

Sensemaking as a process of reducing confusion, uncertainty and ambiguity
"Sensemaking is about contextual rationality. It is built out of vague questions, muddy answers, and negotiated agreements that attempt to reduce confusion" (Weick, 1988 other in terms of the reasons why people react to each: when people are faced with ambiguity, sensemaking occurs due to the presence of multiple interpretations, while in the case of uncertainty, the sensemaking process occurs as a reaction to the of lack of knowledge. Thus, people may follow different strategies in order to make sense in each case. In this study, we adopted part of Wieck's definition of sensemaking, namely that "sensemaking is the attempt to reduce confusion" (Weick, 1988) whether this confusion involves uncertainty or ambiguity.

Sensemaking as a process of gap discovering and bridging
Brenda Dervin's sensemaking methodology emphasises understanding sensemaking from the sensemaker's perspective. The aim is to be centred on the user by considering what is real to them, understood in their own terms. Occasions of sensemaking are understood in terms of a metaphor of a discontinuity or gap. A gap is seen as the question or confusion that arises out of a situation (Dervin, 1983;Romanello et al., 2003) . It is the point at which sense runs out. These gaps act as barriers to progress and give rise to the sensemaking strategies and tactics that people use to overcome them. This leads to 'building a bridge' in order to resolve the gap.

1 Design:
Participants (postgraduate students from Middlesex University), performed a mock investigation task using a collection of documents. The task involved constructing queries over a data set, searching for documents and reviewing them to decide on individual document relevance (document triage).
There was a single independent variable (structure) with two levels (structuring vs. non-structuring). For the structuring condition, participants were asked to "create a representation" of the key events; in doing so, they could use any feature of OneNote to organize the information they selected. For the non-structuring condition, participants were asked to "collect relevant files in a folder". They could not change the file names or determine an order. At the end of each condition, participants were asked to complete a questionnaire.

Materials:
We operationalised the models and definitions that we chose to cover different features of sensemaking. The questionnaire was designed to measure sensemaking overall at the same time as measuring different possible features of sensemaking. Each theory or definition contributed to a subscale in the questionnaire.
A single 'root' question, "To what extent do you think conducting the given task under this condition helped you to make sense of the available information", was used to address sensemaking directly. The questionnaire then incorporated five additional subscales: comprehension and insight, understanding connections, gap discovering and bridging, structuring, and reducing confusion, uncertainty and ambiguity. Each of the 16 sub questions incorporated a statement to foreground a feature drawn from theory, for example, for the structuring subscale: "To what extent do you think conducting the given task under this condition helped you to find a way to organise information". Participants rated their agreement on a Visual Analogue Scale.

Results
The initial task in terms of evaluating the results was measuring the reliability and the validity of the instrument, after which we conducted a correlational analysis to find which among the subscales correlated more strongly with the main question.

Reliability of instrument
Cronbach's alpha coefficients were estimated to determine the internal consistency of subscales and the overall instrument. Table 1 shows the alpha values in the last two columns, corresponding to each part of the instrument. According to these data, the instrument proves to have a high internal consistency (Cronbach's alpha = 0.97) . The indicator of reliability across all the subscales is also high.

Exploratory factor analysis
An exploratory factor analysis was conducted to examine the minimum numbers of latent factors underlying the data. Table 2 shows the eigenvalues and the percentage of variance explained by each factor.
According to the eigenvalue criteria, the two initial factors are candidates for extraction as their values were greater than one (12 and 1.32, respectively). The first factor explained 75% of the total variance, whereas the second factor only did so by 8.27%.
Because of the small percentage of total variance accounted for by factor 2, a decision was made to   As can be seen in figure 2, there was a clear main point of inflexion before the second factor, which supports the argument to retain only one factor. Therefore, if the 16 items are included in the scale, the structure of the single latent dimension on sensemaking seems suitable for the data. Table 3 shows that, across all items all the loading values were positive and all above 0.5. To be more precise, one unit increase in the factor score is associated with more than half a standard deviation increase in each individual item. This suggests that the 16 items are highly associated. That is to say that the factors strongly explain the values across all the 16 items. Therefore, this is strong evidence for assuming that all items are measuring the same construct i.e. sensemaking. Gap discovering and bridging .687 Finally, we conducted a correlational analysis to find which subscales correlated more with the root question ("To what extent do you think conducting the given task under this condition helped you to make sense of the available information").

Factor
All of the subscales correlated very well with this question, as coefficients were greater than 0.6. Therefore all subscales strongly related to sensemaking. In this context, it is worth mentioning that the Structuring subscale had the highest correlation with the root question (0.89), whereas the gap discovering and bridging subscale had the lowest (0.68).

DISCUSSION
We have established, by means of exploratory factor analysis and Cronbach's alpha coefficients, that the questionnaire is a valid and reliable instrument for measuring sensemaking. As such, it can be considered an effective tool for evaluating the various theories of sensemaking that the questionnaire draws on. The results indicate that sensemaking involves all of the processes under consideration: gaining insight; reducing confusion, uncertainty, and ambiguity; finding connections; structuring; gap-finding and gapbridging. However, it is apparent that people recognise structuring most prominently; gap-finding and gap-bridging, on the other hand, appears to be the least recognisable.
The low number of people who consider that terms relating to gap-finding and bridging describe their process of sensemaking suggests that it is a less useful formulation. Using this same principle, the high correlation found between structuring and sensemaking strongly supports this particular formulation of the process. It is important to note that the way people responded may have been an artefact of the specific task. As participants were asked under one condition to "create a representation" of the key events, this may have encouraged them to identify structuring as being related to sensemaking. However, it is encouraging to see that the correlation with structuring was also found in the other condition, which was an unstructured triage task. Future application of the questionnaire to a range of tasks would enable us to assess whether these correlations can be found in other circumstances.