A Frame Signature Matrix for Analysing and Comparing Interaction Design Behaviour

Protocol studies are an established method to investigate design behaviour. In the context of a project to investigate novice interaction design (ID) behaviour across protocols and cultures, we found that existing design behaviour analysis frameworks did not provide reliable results. This paper describes the development of a new approach to analyse and compare ID behaviour using verbal protocols. We augment Sch ö n’s basic design and reflection cycle with construction of a frame signature matrix and analogical categorisation coding. We demonstrate this approach by comparing two protocols of novice interaction designers in Botswana. The initial findings indicate that this approach increases consistency and accuracy of coding, and that there are different degrees of reframing for the design problem and solutions.


INTRODUCTION
Protocol studies are an established method to investigate design behaviour (e.g.Cross et al, 1996;Gero & McNeill, 1998) in which individual designers, designer pairs or designer groups are recorded solving a time-constrained design task.Studies have been conducted in domains including Product (Dorst & Cross, 2001), Architecture (McDonnell & Lloyd, 2009 and Engineering Design (Valkenburg & Dorst, 1998).There is also a wealth of methods to analyse the protocol data (see McDonnell & Lloyd, 2009).This work is part of a larger project that will compare two cohorts of students studying the same Interaction Design module, one in the UK and one in Botswana.We intend to use the outputs from this project to improve design education practice.The Department of Computing at the Open University has been partnering Botho College, Botswana to deliver a degree programme of Computing and its Practice, which includes the module "Fundamentals of Interaction Design".This is concurrently delivered in the UK using the same teaching materials.In this context we sought an appropriate analysis method to be able to confidently and accurately analyse and compare interaction design behaviours across cultures.However the approaches mentioned above have not been applied to interaction design (ID), nor do they have an explicit focus to compare protocols across cultures.For this purpose we looked back at Schön's seminal work on design behaviour (Schön, 1983).Schön observed that design activity iterates through different phases of naming, framing, moving and reflecting.Completing such a design and reflection cycle leads to the reframing of the design problem or solution space.Valkenburg and Dorst (1998) used this cycle to analyse design behaviour in Engineering Design.We intended to replicate their approach in ID, but found it insufficient, producing results with low reliability due to code assignment variation across coders and low validity due to disagreement on coding definitions.
This paper introduces a new approach to analyse and compare data from protocol studies of interaction design (ID) behaviour.It augments analysis of protocols in terms of Schön's design and reflection cycle with frame signature matrices (Gamson & Lasch, 1983) and analogical categorisation coding (Christensen & Schunn, 2007).
We demonstrate this new approach using recently gathered empirical data from novice interaction designers in Botswana working in their preferred local languages (Setswana and Kalanga).We briefly describe data collection, illustrate the approach's use with this data, and discuss initial findings including the approach's applicability to analyse and compare ID behaviour across protocols and potentially across cultures.

BACKGROUND
To understand design behaviour we need to know what designers are thinking about when designing.Verbalisation and subsequent protocol analysis is routinely used in design studies, and according to Coley et al (2007) is, "the most popular method in design research for the capturing, understanding, and analysis of design thinking".There are three methods for obtaining this data: 1. Concurrent (often called "think aloud") 2. Retrospective 3. Constructive interaction (also called "Codiscovery" or "Paired think aloud") We decided to use the method of constructive interaction to elicit verbal protocols (O'Malley et al 1985, Miyake, 1986); this involves paired participants collaborating on a task and talking about what they are doing and thinking.
Constructive interaction has been found to overcome some problems of concurrent protocols, i.e. silence and inhibition; for example, Wildman (1995) used paired user testing.This approach avoids the possible cultural influences on concurrent protocol verbalisation (Clemmensen et al 2008) and the focus on the design product rather than process of retrospective protocols (Kuusela and Paul, 2000).It also aligns with a personal communication from Winschiers-Theophilus which stated that concurrent protocols would not be effective with African students.
A variety of methods have been used to analyse protocol data, such as conversation analysis (McDonnell, 2009), grounded theory (LeDantec & Do, 2009) or semiotic analysis (Lloyd, 2009).In these studies, only one or two protocols are analysed, resulting in a detailed linguistic analysis.Valkenburg and Dorst (1998) worked with protocols in Dutch and stated that a "faithful translation of a transcribed protocol is nearly impossible" (pp.256).
We concluded that a detailed analysis based on linguistic structural or semantic features in translated protocols is not viable in our case due to the need for translation.Furthermore, whilst we were interested in the use of metaphor and analogy by our student designers, we did not require a precise understanding of it.Finally, analysis frameworks for protocols are very specifically tailored to the research question.None of the existing design behaviour analysis frameworks fitted our goal of analysing interaction design behaviour across protocols and cultures.We decided to use Schön's design and reflection cycle as an approach to coding design conversations (as did Valkenburg and Dorst).

Setting
For several years, The Open University has been partnering with Botho College, Botswana to deliver a degree programme of Computing and its Practice.This programme includes the module "Fundamentals of Interaction Design" which is delivered in the UK and Botswana using the same teaching materials (Sharp et al, 2007).

Protocol study
A protocol study of novice interaction design behaviour was held in Botswana in January 2012.Thirty participants were selected from a pool of volunteer students taking the Interaction Design course at Botho College.Participants were paired and told that the study was to investigate design behaviour in different cultures They were given the design problem, which was concerned with how to support sick people in administering the right medication at the right time.We used an empathetic problem description that emotionally involves participants, inspired by Chavan's (2005) 'Bollywood' usability testing method.An excerpt is given below: You have just been appointed by the Minister of Health to design a new interactive product which will help ensure sick people take the right drug at the right time.This is an important role.You also have a personal interest in this subject because you have a close relative who is living with you who often forgets to take her medication.
The participants were given an hour to design solution(s) or partial solution(s) to the medication design problem working closely with each other and talking out loud about what they were doing and thinking.The study was conducted in the participants' chosen language (Setswana or Kalanga) and a facilitator was present to answer their questions.
To capture the protocol, we used two video cameras, an audio recorder and existing CCTV recording equipment.For analysis a team of translators who spoke Setswana and Kalanga translated and transcribed the recordings.

DATA ANALYSIS
This section describes the initial analysis, uncertainties that arose, and two augmentations to Schön's basic design and reflection cycle.

Initial coding
Initially, each of four coders coded two translated protocols using Schön's cycle of reflection and following the approach described by Valkenburg and Dorst (1998).There was overlap such that each coder had both of their protocols also coded separately by two of the other coders.Table 1 shows an extract from our data and its analysis.Initial coding led to poor agreement between the coders over what constituted Framing, Naming and Moving, although Reflecting seemed more readily identified.In an attempt to increase agreement the coding team refined the definitions; Naming: Names identify specific things you have to design, such as functions and components Framing: A frame provides a focus for the discussion.
Frames generate new design requirements or maybe give a set of constraints Reflecting: Reflection is the evaluation of a design idea, concept, requirement.Is this a good thing to be doing?Why is it good?Moving: Exploring the problem space; expanding the brief; adding detail.
Agreeing a set of definitions was helpful to align the team.However there was still significant lack of concordance with regard to identifying a Frame, the extent of its influence on the participants, and when reframing took place.There were two particular areas of concern: (i) How to code the use of metaphor or analogy in the design discussions.The designers frequently used this in their discussion, e.g "it's like a mobile phone, it could be worn like a bracelet."Does this signal a reframing of the problem?(ii) How to code the discussion when an entity to be designed for, or to be designed, is named (e.g the environment) which leads to further elaboration of the design concepts, e.g.Before: Naming a switch off button After: They are combining analogies -bracelet and watch.Naming a switch off button to be designed (part of the framing device) In order to define frames more clearly and consistently, we augmented our analysis with: a frame signature matrix and analogy categorisation.

Signature matrices
Rein and Schön (1996) consider how to address multiplism (multiple viewpoints) in political and social policy.They view this as important because individuals must work out agreements in the face of divergent world views.To address this, following Gamson and Lasch (1983) they clarify a frame as being a strong narrative structure constructed by beliefs and meanings of individuals.Frames guide both analysis and action in practical situations.They are diagnostic and prescriptive narratives that tell what needs fixing (diagnostic) and how it might be fixed (prescriptive)., Gamson and Lasch (1983) identified two devices in any particular frame: "framing devices" suggest how to think about an issue (i.e.what needs fixing); the "reasoning devices" suggest what should be done about the issue (i.e.how it might be fixed).
Applying this notion to recognising frames in the design process suggests that a frames' distinctive signature will be revealed by studying the beliefs and meanings of the designers, and interpreting them in terms of their professional practice and their culture.Thus the signature matrix for an interaction design problem will be instantiated by the specific design goals, knowledge, beliefs, external elements, methods, solution components and conceptual frameworks deployed by the designer (as revealed through their "designing" narrative in verbal protocols).To identify the signature matrices from our data, the matrix elements and attributes were defined and agreed between two coders in alignment with knowledge from the domain of interaction design and grounded in the data after repeatedly reading through one protocol each.The coders were in high agreement about these.Adding values according to each protocol allowed us to identify whether one or more frames were in play and how a new frame was signified and characterised.
Elements of the ID signature matrix which constitute a framing device are: • the designers' goal for the system and core constructs of focus from the ID domain: • the centrality of the user and the importance of their goals and characteristics; • the nature of the environment of use, • the nature of the user tasks and the instantiation of these via entities and relationships drawn from the situation to be considered (e.g deaf patients, time, hospital) and revealed in the verbal protocols.Interaction design also explicitly proceeds via the use of methods, employing metaphor and analogy, and incorporating in the designed product notions of branding, fashion and other culturally determined outcomes.Elements of the ID frame structure which constitute a reasoning device are: • knowledge about methods to understand the situation and possible solutions (e.g storyboarding) • knowledge of solution components (e.g a screen, a power off button), and • conceptual frameworks for identifying solutions (e.g analogy).
Augmenting our analysis approach with the signature matrix helped us to recognise more consistently when reframing took place.Tables 2  and 3 show the signature matrices and their instantiations into two solution-based frames for each of our pairs.Table 2 is based on the transcript of which an excerpt is presented in Table 1.The elements in bold in Table 2 indicate the differences between the two identified frames, and hence the basis for saying that reframing has occurred.In this example, the Bracelet frame introduces a new reasoning device (Bracelet).Table 3 illustrates a further example (extracted) where a new goal is introduced along with other changes

Analogy categorisation
Analogy was identified as a major source of our coder variability.Christensen and Schunn (2007) studied the use of analogy in engineering design.They found analogy served three functions; identifying problems, solving problems, and explaining concepts.We added this categorisation of analogy to our signature matrix as part of the reasoning device elements (shown in Tables 3 and  4  This can be detailed by looking at Tables 1 and 2 together which come from the same transcript.The re-coded "after" extract in Table 1 shows that ambiguity of frame determination has been reduced by illuminating the intended purpose of the analogy.The 'before' analysis in row 1, Table 1 shows that previously we have been uncertain about which frame 'bracelet' should be assigned to.In the revised coding, mobile phone is coded as an explanatory analogy, and bracelet as a candidate solution analogy which clearly marks the start a new frame (see

DISCUSSION
When we reviewed the coding process and outcomes we identified that some reframing seemed of more significance than others.For example, a change of analogy from say, alarm clock to bracelet (Table 2), whilst generating a new design discourse and new entities to design for, did not seem as radical as a change of goal from reminding patients to take their medication,to training them in taking it (Table 3).This also seemed to result in a more prolific period of "naming" new attributes and values.
According to Rein and Schon (1996), frames are ways of determining what needs fixing and how it might be fixed.Explicitly in our protocols, the designers' problem was that patients did not always take the correct medicine at the right time, and the strongly implied design goal was to support patients, "to remember to take medication".
We found that design goal changes appeared to be synonymous with framing device changes and probably signalled a radical reconceptualisation of the problem.Essentially the designer is asking "are we fixing the right thing?".Changing reasoning devices represented a more subtle change of frame and required the judgment of the coder.
We therefore see different degrees of reframing.These are demonstrated by comparing signature matrices of two pairs from our analysis.For each pair there are two frames and therefore two signature matrices.Pair 10 (table 2) reframed by changing reasoning devices.Specifically they changed their solution analogy from "Mobile phone" to "Bracelet".New elements are shown in bold in Table 2.The change to a Bracelet solution allows them to introduce cultural elements as they have more design control over the form.They also consider the powering options for this device (presumably that was a given for a mobile phone).Pair 12 (Table 3) reframed by changing framing devices.Specifically they changed their design goal from supporting "remembering to take medication" to "training patients" (so that they do not need support).New elements are again shown in bold.The change in design goal spawns new elements in both framing and reasoning devices indicating a significant change in point of view.
Based on these examples, our analysis so far indicates that reframing occurs when: • a new solution analogy as part of the reasoning device is introduced • a new goal as part of the framing device is introduced.

CONCLUSION
The basic design and reflection cycle is too open to use as sole coding scheme to analyse ID behaviour.Rein's and Schoen's use of a frame matrix, derived from Gamson and Lasch's work is more fine grained and can elements that align with characteristics of the domain of ID i.e. users and context.The addition to this of analogical coding (Christensen and Schunn, 2007) reduces the ambiguity in the data set.The approach shows potential for comparing ID behaviour across protocols and potentially also across cultures.This will be our next step.
as Reasoning Device (Metaphor/Analogy): Attributes: Solution, Problem, Explanation).The addition of analogy categorisation supports us in identifying more clearly and consistently what attribute and value is named in which device (simply called Naming in Schön's cycle), It does this by showing the addition and loss of elements, and by supporting the location of the reframing in either the framing device attributes and values, or in the reasoning device attributes and values.

Table 1 :
Protocol analysis before and after introduction of signature matrix and analogy coding.Our initial coding is labelled "before", and the revised coding is labelled "after".The "before" comments illustrate the uncertainty regarding how to code this extract.All transcript elements in bold are clearly part of the design discourse but there was poor agreement about which phase of the design and reflection cycle they represented.Transcript AnalysisA.Something like a bracelet, having a vibration mode for deaf patients and functions just like a mobile-phone.

.For deaf patients….
B. It should always show time when vibrating."Showing the time" named as a function A. It should have audio and time.Have you seen those kinds of mobile-phones?I meant to say the small watches designed like a ring, something of that sort... B. Some buttons showing time.It should have one button to avoid many buttons, so that…it can be easily pressed to switch it off.

Table 2 :
Pair 10 -Mobile Phone frame and Bracelet frame

Table 3 :
Pair 12 -Reframing based on a new goal