Learning Cognitive Task Analysis

In this paper we identify and describe a number of issues and difficulties faced by HCI postgraduate students when they learn about cognitive task analysis and how to carry it out. Cognitive task analysis is a set of techniques used to elicit descriptions of the considerations and thinking strategies invoked by people, say, during the use of a piece of technology or system. Such techniques include think aloud methods or in-depth interviews such Critical Decision Method. This work is based on a preliminary pilot assessment to review how teaching of CTA is carried out and how it can be improved.


INTRODUCTION
Among the many tools that a student of human-computer interaction (HCI) needs to have learnt about are the methods and tools to carry out a cognitive task analysis, or CTA.According to Chipman and her colleagues, cognitive task analysis is a set of methods designed to elicit information about the "… knowledge, thought processes, and goal structures that underlie observable performance" [1].Task analysis methods such as Hierarchical Task Analysis [2] focus on identifying observable behaviours such as the sequence of key strokes to send a text message on different mobile phones to compare efficiency of use, or the differences in the observed behaviours between experts and novices in performing a complex task such as the actions taken to diagnose and rectify a failure in a nuclear power plant.In interactions with more complex systems such as the nuclear power plant, where there can be multiple pathways to resolving a problem, traditional task analysis can provide useful information about the actions taken between the novices and experts.However, based on such traditional task analysis alone, the investigator will not have enough information to infer intention and reasons for adopting that particular path of actions.These traditional task analysis approaches are useful for providing insights for the ergonomic design of the physical interfaces, e.g.where buttons and indicators need to be placed to minimize, say, hand movements, or to reduce the likelihood of errors because the switches that carry out dangerously different things are placed together such as 'shut down and erase system' and 'save all work'.
Traditional task analysis techniques are not so appropriate for understanding the rationale of the users' behaviours.Cognitive task analysis, on the other hand, have techniques that are designed to elicit clues about mental processes that underlie our actions, such as how information is considered, recognized, assessed and transformed, basis for judgments, in order to arrive at a diagnosis of a problem, which can then lead to decisions that enable actions and problem resolution.Cognitive task analysis techniques such as the Critical Decision Method [3], Goal-Directed Task Analysis [4], and Work Domain Analysis [5] are CTA techniques for investigating the reasons that underlie behaviours, and for identifying the underlying functional and structural relationships that are essential for problem diagnosis and promoting expert behaviours in process-based systems.
The insight about human actions and the nature of the cognitive work derived from such analyses will then guide the design of information and visualization interfaces, and the supporting methods for interacting with the information.For example, the techniques needed for interacting with a graphic or graphically rendered data, will be different from interactions needed to control input and presentation of form-based displays.CTA also provide important insight about how information is used: is it visually compared?Are the data logically ordered and how?Are data from different sources combined in some way, e.g.multiplied or subtracted, or applied to some mathematical formula?Such insights can then be guided by display design principles such as Wicken's Proximity-Compatibility Principle [6], so that the information needed for a task could be rendered in a way that is compatible with the way the cognitive task is performed.
The general approach described above has been used to design interfaces or organizations that interact with technology in some way to control operational processes (for example, see [7][8][9][10]), and in some cases, have led to as much as 40% improvements in users' ability to identify appropriate geographically-dispersed emergency ambulances to activate [8].
CTA is a useful tool that graduates from HCI courses should know about and be competent practitioners of.However, from our initial review of how CTA is learnt, it seems that while knowledge and awareness of CTA can be readily acquired from tutorials, books and the internet, developing the skills to competently carry out a CTA is still very much a black art.Perhaps it is the nature of expertise that must be honed through practice.In this paper, we describe some of the initial outcomes from this review.

METHODOLOGY
This preliminary review was carried out using an emailed questionnaire to 10 postgraduate students at our university who studied and used CTA for the first time in their research projects.Four of the 10 students responded.The questionnaire is composed of five open-ended questions.The questions followed the pattern of probes used in the CTA technique known as the Critical Decision Method.Question 1.As with the CDM, the intention is to identify and select a specific and memorable instance that one has personally experienced in order to retrospect on it.However, as this was a written survey, it was necessary to elaborate on the question.
( Question 2. In this question, we establish the issue of whether the investigator had personally used the technique.Once that is established, we then ask about how did the student learn to use it, and to then ask to reflect on it to describe some of the difficult or challenging aspects of the CTA technique to learn.Then by asking them to identify some useful 'tips and tricks', it allows us to look at the same issue from a different perspective in order to articulate the problem of learning as they would see it as an expert -to turn their experience of the difficulties into strategies that would help a novice overcome the problem.This is in line with the technique of multi-perspective analysis which is again, embedded in the CDM technique. (

2) In learning about this CTA technique and how to use it in research: (i) how did you learn this technique? (e.g. self-taught by watching a video, read 25 different papers on the technique and they all said different things!) (ii) what were the some of the most difficult or challenging aspects to learn about and to understand, and why? (iii) what were some of the most useful 'tips and tricks' you picked up, and why? (iv) what would have made learning about this CTA technique easier?
Question 3. Continuing in line with the multi-perspective approach, the next question attempts to identify the areas of the technique that presents the greatest difficulty to implement in practice, and to therefore consider how the learning of this aspect should occur to overcome it. (

3) In using what you learnt about that CTA technique, what were some of the greatest difficulties and challenges you encountered in using the technique? Consider the different phases of preparation, data collection and data analysis.
Question 4. Asking someone directly to identify the mistakes or mis-understandings he or she has made in using the technique often leads to answers that skirt the issue.Instead, by posing the question in a different way: by asking them to imagine they were experts coaching novice users, and to anticipate the likely mistakes and the areas these novices are most likely to misunderstand.This provides us with pointers as to where our teaching might be lacking or need clarification.
(4) Now having learnt and are fairly expert with its use, what mistakes or omissions, mis-understandings or incorrect assumptions, would you most likely to see someone just starting out make? Question 5. Retrospection and hindsight are useful tools in trying to understand expertise, which by definition is something difficult to articulate.However, good advice requires a sound understanding of the problems being faced, and is therefore a useful technique from which we can infer the strengths of their expertise, and in this case, what they have learnt about the learning to conduct a CTA. (

5) From your experience, what advice would you give someone new to CTA trying to learn how to carry out a CTA?
In this small study, we have adapted the CDM for use as an evaluation mechanism.We have previously attempted to use such an adaptation of the CDM for the exploratory evaluation of a novel, first-of-a-kind prototype that could be deployed in future ATC systems sometime in the future (cir.2025) under work conditions and operational assumptions that are expected to be very different from that of the present ATC work domain [11,12].

RESULTS
Bearing in mind the small number of respondents to the survey, we will treat the results as exploratory and only use the results to indicate the variety of learning difficulties experienced by this group of learners in our preliminary assessment.The responses to the questionnaire were collected and subsequently collated into common themes that emerge from the data.These themes are reported next.

Learning approach.
The primary learning approach adopted was directed self-learning.The students were initially provided with basic papers and books on various CTA techniques.The basic learning strategy used by the students was to initially read papers that were more theoretical treatments of CTA before moving on to applied papers reporting on actual field studies.This is a typical learning strategy -learn the theory and basic concepts before practicing it.However, one problem with this approach is that of matching the theory to the practice and an appreciation of where one is at in his or her skill development, especially in a directed, though self-learning approach.P1 reports: "I found the books/papers too theoretical and descriptive.Books usually go too much in depth into the theory and they leave the applied work for later.By the time, I managed to get to some example, I already forgot about the theory, or found it so dull that couldn't relate them."P3 augmented this learning approach by watching a few videos of how CTAs are conducted during the interview process, available from YouTube and Dailymotion.He found it valuable to "… get a feel for what's involved -meaning you might [understand] what each method incurs, but you can't really [appreciate] it since it's your first time.A video brings you closer to that experience." In addition, P3 did not stop there in learning how to carry out a CTA.He re-visited the "… theoretical papers during post-analysis of [his pilot study]; customised the technique for final field studies; apply and compare results with existing literature and feedback from other practitioners; [and then] decide on the final form of the technique for that specific context."

Prior experience and learning CTA
None of the respondents had prior experience or knowledge of conducting a CTA, although one had some experience in carry out observational study of user behaviours in an emergency control centre.
"I did not start from scratch as I had already experience on observational study of users' behaviours.I think this boosted the learning process [for learning the Critical Decision Method]."(P4) Because of his prior knowledge in the observation technique, he was able to extend and modify the CTA technique he learnt to allow him to understand not just the decision strategies and goal structures, but was able to combine the findings to arrive at a more comprehensive description of the nature of work of the approach controller in an air traffic control centre.P4 also integrated "… the use a A3 sketch pad where I invited the controller to sketch his spatio-temporal thinking.Without this … it would have been very difficult to follow the complex dynamics of their scenario descriptions."

Difficulties encountered in learning
Most CTA methods were developed to study users or operators working in their natural environments, usually time pressured, safety critical work domains.However, when attempting to use the CTA to study users in non-safety critical domains, P2 expresses the frustration, "The lack of studies in domains that are not defined as [safety] critical makes it difficult to relate with my actual study [of understanding information search behaviours of low literacy users of large data sets]".P1 explains that, "I found the books/papers too theoretical and descriptive" because CTA "… is a lengthy process, the books or papers are not written in an informative way".She proceeds to suggest that a way to accelerate the learning could be short seminars based on case studies "that could be used to emphasize the procedure of each stage, from knowledge elicitation to data collection and analysis", and practical exercises so that "one would be more aware of the errors that one might make and the importance of some questions over others depending on the case".P4 suggests apprenticing as an alternative to seminars, "working as assistant, e.g.note taker of a CDM expert for a couple of interview would have helped.It is very important to observe the tone and the manner an expert maintains when probing a user", to learn first hand the practical skills needed by a CTA interviewer.
Another difficulty reported concerns understanding the purpose of the questions used in the CTA interviews, "the challenging bit is to learn the questions and understand their intention.It is not so difficult, but it require some time to get familiar with them".P4 provides examples of why this is important: understanding the purpose of the questions "helped me to stay focused, since … the presentation order of the questions can change during the interview" as he sought to understand "… the complexity of the scenario dynamics".Because he understood the purpose he was trying to achieve with each set of questions, he was able to introduce another technique during the interview, "… asking the controller to sketch his spatio-temporal thinking on a A3 sketch pad", and was able to modify and "… to formulate new questions as my understanding progressed".P4 refers to this issue of staying focused as "keeping the core principles of the technique running in one's head while in the field", i.e. rather than memorizing the questions, one needs to understand the purposes of the categories of questions.P4 learnt the list that provided such a breakdown of the different categories to stay focused yet flexible during interviews.He then learnt to keep track of that while asking the questions, taking notes, considering the answers, facilitating further responses, and keeping the interview on track.However, he warns that "…breaking eye contact when the person is answering your question can be a rude non-verbal gesture and remind the subject that s/he IS a subject" which will make it hard to establish the rapport needed for successful CTA interviews.
As a piece of advice on how to overcome this, P4 explains, "Stay focused as … you are about to enter the mind of you user.You really need to grasp his view of the problem he was trying to solve during a particular incident.So please listen, listen as carefully as you can and remember: it is the user who has to talk most not you".Such advice is unfortunately are not commonly found in readings on CTA as they refer to very practical aspects of the methods.

Suggestions to enhance learning CTA
A number of suggestions to facilitate the learning of CTA are summarized next.
In terms of books and teaching materials, P1 suggests that CTA books targeted at beginners should "…not have to explain every detail of the theory at the start, but divide it into manageable chunks that relate a piece of theory with a clear example of a "real-case" study.So it will be easier to understand the model …".In this way, the reader can trace the route taken that led to insights that subsequently guided the development of designs.When using 'real-world' cases, we should avoid making them "… too descriptive" or taking a "… too theoretical approach, making it difficult to find the relationships between the theory and application of it".
Another suggestion is the preparation of freely available videos that demonstrate the CTA interviews in real situations.These documentaries or tutorials would be annotated, providing explanations during the progress of the interview (P4).There should be more reports in the literature that describe "… contextualized applications of the techniques" and the changes and adaptation of methodological choices that take place during real-world CTA analysis, rather than the often clinical and clean rationalization of the final methodology (P4).
Additionally, in learning about CTA, students should learn "…different methods [so that they can] adapt to a real-world situation …" which requires students to "… identify methodological limitations" of the techniques.As students build up their toolset of CTA techniques, they need to learn them with the following in mind, "CTA is one of the many approaches for studying work -is it the most appropriate one?Are you comfortable with its [philosophical] implications?",and a final suggestion, P4 advises students learning CTA to "Read at different applied levels of the technique; practice it -rinse and repeat".

CONCLUSION
In this paper we have reported on a preliminary assessment of how a small number of HCI postgraduate students learn about CTA in a distance learning and directed, self-study context.We adapted a CTA technique and re-phrased the questions for use as a self-assessment instrument.Our initial findings suggest that students find it difficult to learn about CTA from books and published papers, preferring annotated videos that show and explain at key points, how a CTA interview is carried out.
As in most CTA studies, the interviews are often recorded either to audio tape or to video, recordings of real interviews are available.The educator can annotate the video using sub-titling techniques in movie making software such as iMovie, indicating points to note or mistakes to avoid, places where the interview is likely to go wrong.The annotations can also include discussions of why certain techniques may work well in a particular situation, and not in another.While this is useful for early learning and preparation, it is no substitute for having actual hands-on practice, which can be recorded, and then debriefed with the educator later on.Annotations can also be carried out using qualitative data analysis software such as HyperResearch.It has capabilities for direct indexing of videos, audio and textual transcriptions.So, instead of indexing and tagging and describing themes or conversation fragments, we use HyperResearch to annotate learning moments.
Students with prior knowledge of some form of task analysis will be able to bootstrap their learning of CTA and how they can flexibly combine and extend on the techniques to suit the work domains they will investigate.As most CTA investigations have been carried out in safety critical domains and much fewer in nonsafety critical areas such as information search behaviours of low literacy users, it is difficult for students find examples from which to learn about how CTA can be adapted for use in such areas.
Another area that is useful for educators is for them to communicate that students should learn about the purposes of the interview questions.This will give them flexibility and the ability to stay focused on the interview's goals despite digressions.The easiest way we have taken to teach the purposes of the interview questions is often to present the questions (or probes, as they are known in the CDM literature) and to explain what they are intended to elicit.In hindsight, while this appears appropriate, it will not support subsequent 'knowledge-based reasoning' strategies, especially when the student is facing the interviewee.
Another approach that has been adopted is instead, to identify the goals that the different sub-groups of probes or questions are intended to achieve, and to then link them back to the underlying model that the CTA technique was originally intended to address.For example, in the case of the the CDM, it was intended to elicit information to provide evidence for or against a model of naturalistic decision making, called the Recognition Primed Decision Model.Thus, when the student understands the relationship between the questions and what aspect of the decision model it was intended to probe, the student will no longer be executing the interview by rote or by procedure, and will be able to operate at the Knowledge-Based Behaviour level of problem-solving and reasoning.This will then enable the student interviewer to change his or her questioning strategy as interesting or un-anticipated situations arises as the interview progresses.
Finally, in learning to be a competent practitioner of CTA, one should learn several techniques and to then use them as the situation demands.However, as mentioned above, while we can learn about different CTA techniques, one should learn them in relation to their underlying models.We can expand our repertoire of techniques by learning and practising several techniques.We can also achieve this by adapting and thereby increasing the scope of one main technique by adding new features or questions or areas of investigation from other techniques.We have adapted, for example, the CDM method originally designed to elicit expertise in decision making, for use to understand learning difficulties in acquiring CTA skills as well as for use in evaluating the utility and effectiveness of new user interface designs.
This initial study has highlighted a number of shortcomings in how we help students learn about CTA.We plan to further investigate the extent to which these difficulties are experienced in other institutions.