Towards an Approach for Analysing External Representations Created During Sensemaking Using Generative Grammar

During sensemaking, users often create external representations to help them make sense of what they know, and what they need to know. In doing so, they necessarily adopt or construct some form of representational language using the tools at hand. By describing such languages implicit in representations we believe that we are better able to describe and differentiate what users do and better able to describe and differentiate interfaces that might support them. Drawing on approaches to the analysis of language, and in particular, Mann and Thompson’s Rhetorical Structure Theory, we analyse the representations that users create to expose their underlying ‘visual grammar’. We do this in the context of a user study involving evidential reasoning. Participants were asked to address an adapted version of IEEE VAST 2011 mini challenge 3 (interpret a potential terrorist plot implicit in a set of news reports). We show how our approach enables the unpacking of the heterogeneous and embedded nature of user-generated representations and allows us to show how visual grammars evolve and become more complex over time in response to evolving sensemaking needs.


INTRODUCTION
During the process of sensemaking, users often find it helpful to create external, structured representations to help them make sense of what they know, and what they need to know (Wong and Varga, 2012).In doing so, they necessarily adopt or construct some form of representational language to enable them to record and interpret knowledge and ideas.
The kinds of representations that sensemakers create can be diverse, drawing on forms such as concept maps, mind maps, argument maps, regular spatial maps and narratives.If sensemaking research is to understand this diversity then it would be helpful to have a way of describing and differentiating cases and trends.This may be helpful, for example, for understanding what happens under different conditions of task, tool and user.
Moreover, software may present an ideal environment for creating and communicating such representations, but designing such tools necessarily implies decisions about representational language including decisions about primitives and relations and the way in which users can interact with these to create more complex structures, in other words, decisions about a representational grammar.
We assume that such decisions are significant.They affect the kind and range of meanings that users can generate as well as the time and effort needed to do so.We note trade-offs between complexity of representational primitives on the one hand, and representational flexibility and efficiency in use on the other.For one thing, by understanding, and describing what users do and what they find helpful to do, we might be in a position to optimise such trade-offs.And so it may be helpful to have a way of characterising, discussing and comparing the different kinds of representational schemes that users develop in different kinds of sensemaking tasks and the tools that are designed to support them.
In this paper we develop an approach to analysing user-generated representations created during sensemaking in terms of their underlying grammar.We do this in the context of an exploratory study looking at the free-from external representations created by a group of participants conducting an evidential reasoning task based on information within a set of news reports.

Towards an Approach for Analysing External Representations Created During Sensemaking Using Generative Grammar Okoro • Attfield
Our approach involves creating generative grammars to account for the interpreted, emergent structure of the reprentations and was inspired by techniques in the analysis of natural language and in particular by Rhetorical Structure Theory (Mann and Thompson, 1987).
In section 2 we review the role of external, structured representations within sensemaking and different representational forms that might be important for evidential reasoning; in section 3 we describe the method for the study; in section 4 we describe the analysis; in section 5 we present findings revealed by the analysis; and in section 6 we discuss the approach.

LITERATURE REVIEW
Sensemaking is an ongoing process described as the deliberate effort to understand events (Klein, Phillips, and Peluso 2007).It occurs mostly when one is faced with new events in unfamiliar situations and current knowledge is inadequate (Zhang et al. 2008).One of the most recognized models of sensemaking is the notional model of intelligence analysis by Pirolli and Card (2005).
Their model decomposes sensemaking in intelligence analysis into two loops of activity: foraging and sensemaking.The foraging loop includes those processes directed at information seeking; the sensemaking loop includes those processes directed at schematization and sensemaking.
Schematization is a significant part of the notional model.It involves the structuring of information in some way, perhaps as a model of the domain, and has a function rather like the frame in Klein, Moon, and Hoffmans' (2006) Data/Frame theory.According to Data/Frame theory information is organized in schematic ways, such as diagrams, stories, maps, hypotheses etc. (Klein, Moon, and Hoffman 2006) so further questioning can be made or conclusions easily drawn.The schema may be held in the analyst's mind (Klein et al., 2007) or aided by a paper and pen or computer based system i.e. external to the analyst (Russell et al., 1993).Whatever the case, representation is at the heart of sensemaking (Attfield and Blandford 2011) and it is sometimes easier to perform sensemaking externally (Kirsh 2009) than internally.Representations and interactions with them can make processing less demanding, with better speed and accuracy than by working internally alone (Kirsh and Magilo, 1994).
No doubt, different kinds of sensemaking problem call for different kinds of representation.And there are many kinds to choose from differentiated by the entities that are depicted and how these are related (Blandford, Faisal, and Attfield 2014).In the case of reasoning with evidence, in which we take a particular interest here, the literature points in part to the significance of narrative or story as a natural cognitive form involved in evidential reasoning where a story consists of chains of events (actions, happenings) and characters and settings (Chatman, 1980).For example, Pennington and Hastie (1986, 1991, 1992) developed their Story Model based on extensive research on how participants acting as jurors constructed an understanding of a murder trial based on evidence.Others have noted that constructing and communicating cases in a narrative form is both natural and compelling in a legal setting (see for example, Chapin, Attfield and Okoro, 2013;McElhaney, 2009 andWagenaar, van Koppen andCrombag, 1993).
Beyond narrative, literature also points to the importance of the representation of argument in evidential reasoning where the case tends to be made on the grounds of normative reasoning.Twardy (2003) andvan Gelder (2009) for example claim that argument improves critical thinking.Arguments can be used to develop and present cases (Cyra and Górski 2011).They can be described as structures of claims in either inferential or evidential relationships (Sbarski et al. 2008) in order to support and/or counter a claim.
An argument map is a visual representation of an argument (Sbarski et al., 2008).Argument mapping was made popular by Wigmore (1913) and Toulmin (2003).The rationale for mapping an argument is to uncover its structure with an interest in, for example, identifying unstated assumptions or evaluating the support an argument provides to a conclusion (Fisher 2004).Le et al. (2013) and van Gelder (2002) described a typical argument structure to be a "box and arrow" diagram where the nodes correspond to propositions and the links show their relations, whether they be evidential or inferential, where each named circle represents a proposition and lines represent inference (Macagno, Reed, and Walton 2007).
Based on a study of how judges reason with evidence, Wagenaar, van Koppen and Crombag (1993) developed the notion of anchored narratives.Anchored narratives are essentially hybrid forms in which stories are anchored through simple argumentational links to evidence (as well as through our common sense understanding of how the world works).Bex (2009Bex ( , 2010) ) adopted the idea to develop an approach for police analysis to represent and work with cases.
The work of Wagenaar, van Koppen and Crombag and of Bex open up the possibility for complex hybrid forms in evidential sensemaking.When it comes to what users actually do or what they actually find useful, why should we expect that the relations that they use or find helpful to use correspond to one form or another?Commercial, investigatory tools such as IBM I2 for example provide sufficient flexibility for representing cases in terms of entities and events within network graphs of propositional relations (i.e.concept maps), timelines for appreciating narrative and simple argumentation or inferential links.
We have argued that the representational language that people find helpful may vary from problem type to problem type, and representational needs may be complex and hybrid.To understand user needs, of course, it is worth studying what they do in practice.And given that the forms that they use may be complex and diverse, we need a way to study user-generated representations which has descriptive flexibility.The approach that we adopt in this paper is motivated by approaches used in the analysis of language.Specifically we develop an approach of representing the visual grammar of a representation based on Rhetorical Structure Theory (Mann and Thompson, 1987).
Rhetorical Structure Theory is a theory of text organization which provides a framework for an analysis of text.The theory is based on the premise that a coherent text is not merely a string of clauses, but consists instead of hierarchically organized groups of clauses that stand in various relations to one another.These rhetorical or discourse relations can be described functionally in terms of the purposes of the writer and the writer's assumptions about the reader.They hold between two adjacent parts of a text, where, typically, one part is "nuclear" and one a "satellite."An analysis of a text consists in identifying the relations holding between successively larger parts of the text, yielding a natural hierarchical description of the rhetorical organization of the text.Here we apply this idea to the analysis of user-generated visual representations.

STUDY METHOD
For the study task we used data from the IEEE Visual Analytics Science and Technology (VAST) 2011 mini challenge 3. The data were a corpus of news reports about a fictional city called Vastopolis.The task, as per the original VAST challenge, was to analyse the documents in order to identify and provide evidence for potential terrorist threats.The original dataset contains over 4000 plain text documents, manually generated or modified from an existing corpus of news reports.Each report is a plain text file containing a headline, date of publication, and content.Since our interest was to gather data about how users structure information to solve the task, we reduced the search problem by using a subset of 30 articles handpicked to include 13 documents that were judged as relevant to the threat, as determined by the VAST challenge committee, and an additional 17 irrelevant documents randomly selected to add noise.
A PC running Microsoft OneNote was used as the task environment (see figure 1).OneNote was used as it provides a canvas on which users could readily import and organise documents (shown as icons), links to documents and document snippets (extracted from a text editor and shown as text).It also includes tools for annotation and drawing elements such as lines/arrows etc. plus elements can be organised and related in fairly ad hoc ways relatively free of a priori semantics, allowing users a good deal of freedom to generate the representational conventions that they find most helpful.
The documents were loaded into a folder and shown at the interface using Windows File Explorer.This allowed users to see the documents as an ordered list of icons, to search them using Windows Search, to open documents in a text editor and to drag complete documents or text snippets into OneNote.Screen capture software was used to record movies of the session for later analysis.
11 postgraduate students took part in the study.6 were female and 5 male with an age range of 21 to 36 years (mean = 30 years, standard deviation = 5.2 years).Participants signed an informed consent form and were given some training on OneNote and the search tool.For the task, they were asked to: (i) identify any imminent terrorist threats in the Vastopolis metropolitan area; (ii) provide detailed information on the threats (e.g.who, what, where, when, and how); (iii) provide a list of the documents to support their answer.
They were asked to conduct the investigation in whatever way suited them and were told to use the representation platform in any way they wanted.

ANALYSIS
We approached the analysis with the aim of understanding the representational conventions that participants had generated in the process of solving the problem, and to understand how these took shape over time.Focussing on relation types, an informal review indicated that participants had each combined a range of conventions to create hybrid representations.For example, a single representation might apparently use spatial organisation to indicate thematic grouping, arrows to indicate inferential or argumentational links, and vertical order to represent sequence or narrative.In other words, the representations seemed to be heterogeneous.Another characteristic was that these conventions appeared to be hierarchically embedded.In other words, structures of one type might be embedded within a structure of another.
The third thing that we noted was the presence of repeating surface level sub-structures, suggesting an attempt to adopt consistency in the ways of representing particular kinds of relation.
These characteristics, which are also characteristics of natural language, suggested the potential value of characterising the conventions that participants had created in terms of some form of grammar.The approach that we developed uses this idea and is also derivative of Rhetorical Structure Theory or RST (Mann and Thompson, 1988).RST is an approach to describing the organisation of a text in terms of its discourse structure relations.Discourse structure relations are functional relationships that an author intends between elements of text, such as elaboration or cause or justification.RST describes texts by characterising them in terms of a set of predefined relations such as these (Mann & Thompson, 1988).
The particular relations of RST, however, are not terribly important here; what is important is the general approach.RST analyses text as hierarchically structured and is based on the assumption that any coherent text ought to ultimately be analysable as a single relation.This

Towards an Approach for Analysing External Representations Created During Sensemaking Using Generative Grammar
Okoro • Attfield kind of structural assumption is broadly applied to many kinds of linguistic analyses.
Importantly, an RST analysis represents an account of an author's communicative intent as interpreted by an analyst.RST is intended to capture the way in which an author intends relations between text elements to be understood by the reader.
Similarly, although the representations we analysed were not what one might think of as prototypical examples of communication, nevertheless they were imbued with a communicative intent that we aimed to capture.
Related to this, is the question of whether there are particular cues that can be taken as indicating a particular relation.No doubt, within any representation, communicative intent or meaning supervenes on surface structure.In RST analysis, however, no particular commitment is made to surface level cues, presumably, since this would introduce a requirement for defining an exhaustive set of cues and contexts and overly constrain the analysis approach making it inflexible and insensitive to context.What is important is the meaning or intent behind the relation not how it is realised.We adopted the same approach in our analysis.A refusal to reduce analysis to a predefined set of cues or indicators means that the analysis is essentially interpretive.We wanted to capture user intent based on an interpretation of intended meaning, recognising that a given meaning or intent may be represented in any of a number of ways.To anticipate our findings slightly, one thing that this enabled us to do is to detect the way in which users apparently adapted the mapping between surface form and meaning based on pragmatic considerations of representational space and user-cost.
Viewing the screen recordings, we developed an emergent coding scheme which we could use to account for the kinds of relations that participants created.As our analysis proceeded (and in common with RST) we defined the relations that we saw formally, resulting in a coding dictionary (See table 2).The relations, which are as follows, focused on functional characteristics of the relations and their elements.

Information object surrogate relation
An information object surrogate relation is a combination of elements which act as a proxy for a document or some other container of information.The information object could be a document, video or sound file etc.It has a mandatory element summary which reminds the sensemaker of the central idea communicated by the information object or an idea within it that is important for the current task.The summary can be a title, a gist or both.An information object surrogate can also include a date indicating when it was published or when an event that it reports occurred.It can also include a source which is some form of reference or pointer to the information object which allows the representation user to have a ready access to it.The source might be a clickable icon or a text string with an information object identification number or a combination of both.

Timeline relation
A timeline relation is a combination of two or more information object surrogates sorted in order of publication date or date that events took place.The timeline relation allows the user to review multiple events in chronological order.

Themed grouping relation
A themed grouping relation is a combination of information object surrogates which are about some common theme or topic.The themed grouping relation allows the user to review multiple events of the same theme.

Explanation relation
An explanation relation is a combination of two information object surrogates linked through an explanatory relation.This relation has two mandatory elements: an explanans, which is the thing that does the explaining, and the explanandum, which is the thing that is explained.There may be multiple explanans.The explanation relation allows the user to express that an event explains why another event occurred.

Justification relation
Justification is an argumentation relation.It has mandatory elements of evidential support and claim and indicates that the claim follows from the evidential support.The evidential support element can be a representation of an information object.For example the document icon or a text string with the document identification number (e.g.03040) or a combination of both.The justification relation allows the user to express a claim and provide support to increase the belief in the claim i.e. to show the argumentational support for the claim.

FINDINGS
In this section we describe some findings enabled by the analysis approach.

Summary Findings
Beginning with some summary observations.Figure 2 shows the distribution of the relation types as they were used across participants.Of the 11 participants, 2 used 3 kinds of relation, 4 used 2 kinds, and 4 used just 1.In other words, participants generally developed schemes that were a hybrid of more than one convention.The most prominent structuring convention was timeline which was used by all but 1 participant.

Figure 2: Grid showing participant number with structuring relations used
We also analysed the sessions in terms of the distribution of different kinds of structuring over time.This was done by noting the different types of structures that were created during each minute for each participant.On average, 65% of the minutes featured timeline construction.This was followed by justification (18%), explanation (4%) and theme grouping (4%).

The Evolution of Representational Grammars
The relation definitions in section 4 evolved as we analysed each of the screen recordings.For each apparent steady state of the representation, we developed the definitions described above, used the definitions and the representations to create syntax trees to describe the representation and adjusted a developing set of production rules that could describe the tree.At each point, we aimed for the most parsimonious grammar possible.
This allowed us to describe and examine how the representational conventions developed by each participant changed over time.Below we work through an example to show this process and to show how the approach led to interesting findings.The participant in our example began the task by dragging a document icon (source document) from the collection to the OneNote canvas.We can describe the representation at that point as consisting of a source where source is a reference or pointer to an information object.It can be described by the simple syntax tree shown in figure 3. The tree means, simply, that the representation consists of a source document.

Figure 3: Syntax tree showing one information object
We can describe this tree using a single production rule: <Representation> → <source> In the next step, the participant added another document icon to the canvas, only this time they gave it a title and a date, possibly because they now had two documents and anticipated the need to differentiate.They chose to indicate the relationship of the title and date to the document icon using a blue arrow (see figure 4).
This new arrangement is a composite sub-structure which appears to have something of its own identity.Blue arrows, it appears, now have a role of linking a source with its corresponding title and date.In this configuration, the title and date appear to qualify the source.In our analysis we gave this composite sub-structure the name information object surrogate.Across the analysis of all the representations the information object surrogate was a fairly common sub-structure, although it did come in a number of variants in terms of both its constituents and the surface-level cues.Variations in constituents were accommodated by making small adjustments to the definitions and by specifying both mandatory and optional elements.Although, the definitions were unaffected by variations in surface structure these were interesting to observe, and supported the strategy of abstracting away from specific cues.
In figure 4 we see the document surrogate relation indicated by an arrow.This association could also be indicated using proximity and perhaps we could have interpreted the title and date combination as an entity in itself.However, in the long run this would have led to a more complex grammar.As we will see, by not having this intermediate structure we will be able to more easily accommodate subsequent variants more parsimoniously.
We now move forward in the example.Having established the convention of signalling an information object surrogate relation using an arrow, this was later dropped in favour of the using proximity and the object merge tool within OneNote.We interpret this as a time and space saving move.The participant, we assume, having realised the implications of how they were indicating document surrogate relations in terms of the time it takes to create them and the space they use up, decided to change the way this was done.
In other words, they projected forward the implications of a choice they had made and decided it would be more effective to change their strategy, even though this required the revision of previously created relations.This is a frequent kind of dilemma that arises in many sensemaking tasks, where realisations suggest changes to strategies but that these have implications for redoing work.
Moving further forward, the participant began grouping information object surrogates in terms of two emerging themes.In order to visually differentiate them they added yet another representational convention of signalling a theme using a combination of proximity and colour (see figure 6).
Throughout, we noted that the participant followed the current set of conventions relatively strictlyuntil, that is, they changed the convention.Each document icon they added led to the creation of an information object surrogate.Once they had established the convention of associating documents icons with titles and a date, this was done for every new document icon added to the representation.
We also observed that within these themes the participant sorted the information object surrogates chronologically by publication date from top to bottom.We took this to indicate that the participant was creating a timeline within each theme relation.

DISCUSSION
In this paper we have described an approach to analysing the representations that users create in order to solve sensemaking problems.We adopted an approach to the analysis that looks at the constituent structure of elements in a representational convention taking linguistic analysis as a model.
We describe user-generated representations using production rule grammars and show that doing so can reveal some interesting features of what users do that would not otherwise have been easy to see.These include that users create complex heterogeneous representations consisting of multiple relation types; that these entities and multiple relation types are embedded within others so there is a need for a hierarchical theory to unpack them; that when users develop representational conventions on-the-fly they introduce additional complexity to meet evolving representational needs; that they have a desire for consistency; and that they sometimes change the surface form of representational components to meet demands of space and the effort required to create them.The reason why such an approach is helpful, we argue, is because it can help us to study what users do under different circumstances of sensemaking, understand through this what they might need, and help us to specify new kinds of support tools tailored to the kind of task that a user wishes to perform.
We recognise that there are potential limitations in the approach which should be addressed.For example, it is well suited to decomposing structures which can be readily analysed as hierarchical, but some structures are not like this.Examples include matrices and networks.It is not clear how these would be dealt with, although alternatives might include treating such structures as two-dimensional indices or as basic and undecomposable.This would probably not be such a terrible thing given the aim of abstract characterisation.
In future work we plan to address questions like this and also to see how the approach applies across variations of sensemaking task and support tools.One interesting question would be to see how changing the user-cost structures implicit in creating and modifying different kinds of representational element systematically change how users appropriate tools to support their representational needs.

Figure 1 :
Figure 1: The setup used for the study with the search tool to the left and the OneNote canvas to the right.

Figure 4 :
Figure 4: Representation showing one information object surrogate linked with an arrow Now we have a representation that consists of a source document and an information object

Figure 5 :
Figure 5: Syntax tree showing one information object surrogate and one information object

Figure 6 :
Figure 6: Representation showing colour coded themesFigure7shows the final syntax tree created for this participant and below it is the set of production rules that describe it.Within the syntax tree, IOS, S, Su and Dt refer to Information Object Surrogate, Source, Summary and Date respectively.In the production rules, bar "|" acts as an OR operator.

Figure 7 :
Figure 7: Syntax tree showing all the constituents of the user's representation

Table 1
shows the documents used in the study.

Table 1 :
The document serial numbers for the relevant and irrelevant documents used in the study

Table 2 :
Relations and elements used for analysis