An Exploratory Study of Computer Supported Sensemaking: Relating Representational Structure to Triage

This paper reports part of the results of an exploratory study that investigated how the use of external representations alters the process of sensemaking. The results show that there are significant correlations between the level of structuredness and the document triage process, in terms of its performance and its efficiency. Sensemaking. Document triage. External representations.


INTRODUCTION
Sensemaking has been thought of as the process of constructing an understanding when facing complex situations (Attfield and Blandford, 2009;Klein et al., 2006). Representation creation is central to the process of sensemaking, whether internally, with a mental model, or externally, by creating for example maps or tables. When solving complex sensemaking problems people often create external representations of information. Where this is conducted using computers, representations may be created using spreadsheets, mind mapping or argumentation software, or more specialist applications such as IBM I2 which is used fairly extensively by the police and military. The significance of representations has been acknowledged in many of the sensemaking models (Klein et al., 2006;Pirolli and Card, 2005;Russell et al., 1993). It also tends to be assumed that creating external representations help the sensemaking process although little is known about precisely how external representations alter the process of sensemaking. Answering this question, we suggest, can help in designing tools to support the use of external representation in the process of sensemaking. We conducted a study to assess the influence of the manual creation of an external representation during a computer supported sensemaking task. We report specifically on the relationship between qualities of the external representation and triage performance.

Sensemaking
The concept of sensemaking has been investigated by prior studies in various disciplines such as HCI (Russell et al., 1993), library and information science (Dervin, 1998(Dervin, , 1992, organisational studies (Weick, 1995;Weick et al., 2005), and naturalistic decision making (Klein et al., 2006). Sensemaking has been described as the process of comprehension (Klein et al., 2006), or the process of constructing an understanding when facing complex situations (Attfield and Blandford, 2009).

Sensemaking and external representations
During the process of sensemaking, people structure information into representations that help them to make sense of a situation. The significant role of searching for structure in the sensemaking process has been emphasised in some previous research (Klein et al., 2006;Pirolli and Card, 2005;Russell et al., 1993). In the notional model of sensemaking (Pirolli and Card, 2005), in a stage called "Schematize", the analyst uses the data from the "Evidence File" stage in order to structure a representation to aid the analysis process. In the Learning Loop Complex theory, sensemaking is defined as "the process of searching for a representation and encoding data in that representation to answer task-specific questions" (Russell et al. 1993). Searching for structure appears in the 'generation loop', where the sensemakers search through the data to build an initial representation in order to make sense of the domain. The Data-Frame theory of sensemaking by Klein et al. (2006) considers sensemaking as a process of framing and reframing, where the frame is a cognitive structure used to filter the data or use the data to update the frame. The frame is used as a metaphor for the representation people construct in their mind and use to comprehend the world.

Document triage
The term "triage," primarily used in a medical context, refers to the process when the "triage nurse" decides the patient's order of treatments based on the degree of emergency of their case. The information seeking community adopted this term to refer to the essential step in the information seeking process when people evaluate the results of a search they conducted to make a decision regarding what to consider relevant to their information needs. This step of an assessment can occur during any stage, whether initial or advanced, of the information seeking process (Loizides, 2009). The judgment of relevance in the document triage process can be done in less than one minute (Buchanan and Loizides, 2007).

Experiment design
Participants were asked to perform a mock investigation using a collection of documents. The task involved constructing queries over a data set, searching for documents, and reviewing the results to decide on individual document relevance (document triage). There is a single independent variable (structure) with two levels (structuring vs. non-structuring). Participants either structured the documents using OneNote software (structuring condition) or simply put the documents into a folder (non-structuring condition). The experiment involves two conditions: structuring information and nonstructuring information. A repeated measure design was followed.

Data set
The data sets in the study were collected by the researcher from many different news sources; each file includes a news story. During the qualitative analysis, besides looking for emergent themes and patterns, the process model figure (1) was used to help detect changes in participants' information-seeking behaviour and interaction with the data, before and during the process of constructing representations.
The analysis of the data indicated that "structuredness," which is the amount of structuring created in the representations in the "structure" step of the model, had an impact on triage and constructing queries. This variable, "structuredness," is considered high if the amount of structuring in the representations is enough to build a good understanding of the domain. For example, it is not about the number of files that participants collected and judged as relevant in OneNote canvas, but also about whether there was a strong relationship between them, for instance, timelines or themes. Having a link between files is very important, because some participants were observed using OneNote canvases as a space (or a folder) to save as many relevant documents as they can without building relationships between these files.
One of the findings was that during this process, participants judged the relevance of documents by evaluating specific parts of the document that relate to the created representation, without reading the entire document. For instance, participants who created a chronological representation evaluated the relevance of documents by looking at the date in the file first, while those who created thematic representations concentrated on the parts related to the theme. Based on that analysis, the following sets of hypotheses were developed: 1-Structuredness correlates with query specificity. 2-Structuredness correlates (inversely) with triage time. 3-Structuredness correlates with triage quality. 4-Structuredness correlates with perceived sensemaking. 5-Structuredness correlates (inversely) with perceived uncertainty. 6-Perceived sensemaking correlates (inversely) with perceived uncertainty. In this paper we only report the third hypothesis.

Significant different cross conditions
A paired-samples t-test was conducted to compare the effectiveness of the document triage process across the two conditions (structuring condition) and (non-structuring condition).To measure the effectiveness of the document triage process, recall and precision were calculated in both conditions. Recall refers to the proportion of all relevant documents that were judged as relevant by the study participant, while precision refers to the proportion of documents judged relevant by the study participant that, in fact, proved to be relevant.

Measuring structuredness
The concept of structuredness was operationalised based on the assumption that, similar to language analysis, when people create external representations during sensemaking tasks, there are visual languages of production rules the users follow in creating representations (Okoro and Attfield, 2016). These visual representations can be described in terms of parse trees that consist of entities which are embodied within other entities. The grammar is interpretively defined by interpreting the external representation in terms of what we think the sensemaker's intent is and describing the underlying structure in a parse tree. The length and depth of the created parse tree, which represents the level of structuredness of the external representation, is then measured using a metric that we developed. Based on the idea that the type of the representations that people create impose some constraints on the information, for example: a timeline relation imposes a chronological constraint, a metric was developed to measure the level of the structuredness. The metric sums up the number of different relations (such as: theme, timeline) within a representation and multiplies that by the dimensionality and variable type of each relation.

Figure3: a scatterplot with a regression line showing the linear relation between the level of structuring and recall.
Pearson correlation was run to assess the relationship between levels of structuredness and recall. There was a strong positive correlation between the two variables [r =.915, n =13, p =.000]. The value for Pearson Correlation is 0.915. It is very close to +1. This means that there is a very strong relationship between structuredness and recall. It means that as structuring is enhanced, recall rises as well. A change in one variable affects the other. The Pearson Correlation 0.904 is close to 1. The implication is that there is a strong positive relationship between structuring and precision. It means that as structuring is enhanced, precision rises as well. A change in one variable affects the other. The Sig. (2-tailed) is 0.000. This is less than 0.05. It implies that there is a statistically significant correlation between structuring and precision.

CONCLUSION
We found significant correlations between the level of structuredness and the document triage process, in terms of its performance and its efficiency. Document triage refers to the essential step in the information-seeking process when people evaluate the results of a search they conducted to make a decision regarding what to consider relevant to their information needs. This step of an assessment can occur during any stage of the information-seeking process, whether initial or advanced (Loizides, 2009).
Using a similar paradigm to the first study, the current study will replicate the findings of the previous study and expand on them by assessing a set of additional hypotheses. It will look further into the role that external representations play in sensemaking by assessing new dependent variables: specificity of query, perceived sensemaking and perceived uncertainty. Specificity of query will be measured through calculating the number of returned documents by each query; perceived sensemaking will be measured using the same questionnaire as the first experiment; uncertainty will be measured using a questionnaire adapted from previous studies.