Use of a team collaboration model to identify candidate knowledge management and collaboration technologies

Motivation - We used a model of team collaboration as a framework to assist in the requirements stage of knowledge management (KM) and collaboration system development. Research approach - The model emphasizes the macrocognitive processes entailed in collaboration and includes major processes that underlie this type of communication. Data was analyzed for several 9/11 efforts and air-warfare scenarios. Findings/Design - Data captured from teams performing their tasks provided insight into areas where collaboration might be lacking. Research limitations/Implications - Findings suggest our approach can identify candidate KM and collaboration tools types. Further analyses and other types of collaborative tasks are necessary to generalize our results. Originality/Value - Our work focuses on the requirements stage of development while others generally focus on assessing current technology. Take away message - Our research seeks to identify the right types of technology tools early in the development process. Cost savings is generally more dramatic in the earlier stages of system development.


INTRODUCTION
Government, military, and business teams/organizations increasingly use collaboration and knowledge management (KM) technology to collaborate and share information and task perspectives to aid in decision making. Rapid access to current, accurate, and relevant information and the ability to engage in real-time collaboration with other geographically distributed decision makers have become indispensable elements of the command and control planning and decision-making process. Structured data, such as relational databases, and unstructured data such as documents, graphics, and audio/visual files have increased dramatically while the technology to make sense of and provide access to this data has fallen behind. Data is increasingly being distributed through mobile devices such as personal digital assistants (PDAs), Blackberries, and iPhones. While there is no lack of various collaboration products or developers who will customize a suite of collaboration tools, what is lacking is the use of a user-centered design approach to assess the right tools for the diverse KM and collaboration requirements needed in each organization or team.
General KM and collaboration categories for teams and organizations can include, but are not limited to, content management, portals, workflow, collaboration suites, and information retrieval. Under these broad categories can include types of collaboration tools such as instant messaging, discussion groups, web conferencing, virtual workspaces, Real Simple Syndication (RSS), Geographical Information Systems (GIS), analytics (business intelligence) and sometimes decision support systems (DSS). Content management software enables users to retrieve and view all types of data, including "unstructured" documents, graphics, presentations, and audiovisual files as well as structured relational databases, and produces databases that are interoperable, eliminating "stove-piped" data. Web portals are used to import diverse sources of data to one location, which is dynamically updated; another key feature is that it can "push" time-critical information to users using RSS technology. Analytics and "business intelligence" refers to techniques for displaying relationships between data in different ways for viewing and analyzing the data in order to gain insight regarding relationships between elements of that information. GIS systems can provide a synchronized graphical representation of data and information in layers on an electronic map similar to a GPS and was used for example, during the hurricane Katrina recovery operations as a means for first responder teams to coordinate operations and reduce redundancy of team effort. DSS systems may be used to reduce the reliance of subject matter experts on the team during time critical operations.
The focus in this paper is on how the model of team collaboration can be used to make recommendations regarding the selection of these types of candidate collaboration tools and then to be used in the design of "proof of concept" prototypes using agile methodology such as rapid application development (RAD). This is only the first step in the process of implementing the optimum set of collaboration tools. Other researchers have focused on evaluating existing collaborative tools using such measures as situation understandings, plans, and decisions (Noble and Kirzl, 2003). Researchers typically compare the delta in team performance after the implementation of collaboration technology. O'Dea et al., (2007) identified 29 indicators for assessing collaborative technologies but underscore that prior to this, "… the selection of appropriate metrics for assessing the effectiveness of collaboration technologies must be preceded by an in-depth understanding of the collaborative task and thus, the requirements of the system." Thus our framework and associated data provides at least a part of the understanding needed for the requirements stage prior to the selection or evaluation of collaboration technologies addressed in previous research.
The standard methodology for the design of information systems includes use of the universal modeling language (UML) use cases, workflow analysis, and strategies such as spiral development to understand user requirements. UML use-case studies entail observing a person while performing the task to gain insight for developing software for that specific task. In a use-case study, UML is used as a diagnostic tool however one drawback to is it typically ignores the cognitive components and socio-technological environmental aspects which influence the process. While use-case studies often provide valuable inputs, human factors techniques are also needed to cover these other aspects of the overall complex process the software is designed to support. User surveys may prove to be inadequate since few users may be aware of the whole decision process of the team, organization or inter-agency collaboration processes. Such design strategies ignore the contributions provided by naturalistic decision-making (NDM) methodologies and how technology designed based on techniques such as the crucial incident technique and cognitive task analysis could produce tools that provide enhanced support for the macro-cognitive processes involved in task performance.
The lack of a clear methodology for technology selection and development of the "right" KM and collaboration technology (CT) applications from a NDM perspective can have dire consequences. For example, of the eight missed opportunities cited by the Columbia Accident Investigation Board (2003) that might have avoided the Columbia space shuttle tragedy, at least six were related to the use of email -typically a poor collaboration tool. Emails were often sent via "local channels," therefore no one had a macro-view of (1) the scope of the problem or (2) all of the relevant information. The Board reported emails that were not followed up on: This particular problem might have been resolved by using workflow management software. The Board also identified the problem of trying to access different types of data such as Power Point slides and structured data from multiple databases. This would suggest considering use of content management technology that uses open standards to overcome "stove-pipes." The board pointed out the "flawed safety culture" that had to be dealt with before these technology solutions would be effective. As with any complex problem, developing solutions usually requires a multi-faceted approach (see, for example, Hocevar, Thomas, and Jansen, 2006). Similar after-the-fact studies on the 9/11 performance of New York police and fire departments (McKinsey, 2002) and the performance of NORAD/FAA on Sept 11, 2001, (National Commission on Terrorist Attacks, 2004Routt, 2008) indicated that in both cases deficiencies in collaboration tools played a role in team performance. For example, in the World Trade Center, the magnetic status boards were lost when the towers fell. Critical situation awareness was lost, thus impairing command and control, fundamental to managing such a complex situation. Use of a networked electronic status board would help prevent the loss of essential information, and would provide the additional benefit of distributing this critical information to team members at all locations where this information can be useful. In line with the theme of this year's conference -NDM and Computers -we focus on the challenges associated with making decisions in demanding situations and discuss how these decisions can be facilitated by incorporating computing technology, with examples of situations where the existing technology does not adequately support the decision-making situation.

Model of Team Collaboration
A model of team collaboration was developed that emphasizes cognitive aspects of the collaboration process and includes the major processes that underlie this type of communication: (1) individual knowledge building, (2) developing knowledge inter-operability, (3) team shared understanding, and (4) developing team consensus (Warner, Letsky, & Cowen, 2004). Analysis of team communications data from three collaborative decision making tasks in the context of this collaboration framework provides clues to team performance problems. The analysis therefore focused on how the collaborative team problem solving process was enhanced or hindered by CT, the appropriateness of CT tools, and how the lack of technology hindered the macro-cognitive processes, and therefore team performance. Team communications that transpired during three complex problem-solving situations were analyzed and coded: NORAD/FAA collaboration during 9/11, air-warfare decision-making scenarios, and communications that transpired between firefighters when they were responding to the fires in the World Trade Center that occurred as a result of the terrorist attack on 9-11.
The first problem situation analyzed the 9/11 efforts of the Federal Aviation Administration (FAA) in communication with Northeast Air Defense Sector (NEADS) and North American Aerospace Defense Command (NORAD) collaborating to scramble fighter aircraft to escort the airliners. The air-warfare scenario involves a US Navy Aegis cruiser combat information center team identifying air tracks in a Persian Gulf scenario. The firefighters were conducting search and rescue and working to extinguish the fire. All three problem situations involve team collaboration to solve complex, one-of-kind problems. The objective for this research is to better understand the cognitive processes employed when teams collaborate to solve problems and to make recommendations to improve decision-making in information-rich, time-compressed collaborative problem-solving situations.
The types of problem-solving situations this model describes are ill-structured decision-making tasks, characterized by time pressure, dynamic information, with high information uncertainty, high cognitive workload (i.e., a large amount of knowledge is brought to bear), and human-system interface complexity. The model focuses on three tasks: (1) team data processing, (2) developing a shared understanding among team members, and (3) team decisionmaking and course of action selection. Four unique but interdependent stages of team collaboration are included in the model. As depicted in Figure 1, the stages include knowledge construction, collaborative team problem solving, team consensus, and outcome and evaluation and revision. Cognitive processes within each stage are represented at two levels: meta-cognitive processes, which guide the overall problem-solving process, and macro-cognitive processes, which support team members' activities within the respective collaboration stage. The model's macro-level definition of the cognitive processes permits empirical assessment of these cognitive processes with currently available measurement techniques, e.g., verbal protocol analysis, communication analysis .
Team types described by the model include teams who operate asynchronously, whose members are distributed, and may be culturally diverse, where members possess heterogeneous knowledge, due to the unique roles played by each team member, and operate in a hierarchical organizational command structure, and in some situations involve rotating team members . The model consists of general inputs (e.g., task description), collaborative stages that the team goes through during the problem-solving task, the cognitive processes used by the team and final team outputs, such as the selected course of action.

METHOD
Verbatim transcripts were analyzed from two series of experiments and two real-world situations where teams collaborated to solve a complex problem. In all three problem-solving tasks, assessment is particularly difficult due to time pressure, high workload, and because the available information is often incomplete or ambiguous. Transcripts included communications that transpired between all team members as well as with decision makers at the distributed sites. Our approach was to analyze and code team communications data using the cognitive process definitions developed by . The three tasks also provide insight on the possible collaboration tools needed or the deficiency in CT/KM/DSS computer tools.
Two coders coded a practice set of team communications and then discussed their respective coding to calibrate their use of the definitions of the macrocognitive processes included in the model. Following this training period they coded independently and then reviewed their coding and resolved any differences. An inter-rater reliability score was calculated to measure the degree of subjectivity involved in the process.

Experiment: Air Warfare Decisionmaking
Air warfare decision making is conducted in the combat information center of a Navy ship. The team is responsible for identification of a large number of air tracks under high time pressure. These air tracks can fit multiple hypotheses regarding the level of threat they pose to the battlegroup due to the high level of ambiguity associated with the data. Incoming information arrives via various sensor systems and various verbal reports. In a series of speech turns, five separate contacts may be discussed at various levels -initial reports, updated reports, sharing information on the response, or lack of response, by the contact to some action taken by the ship, etc. The high degree of ambiguity associated with contact information can often make threat assessment very difficult because many pieces of data can fit multiple hypotheses regarding threat assessment.

Transcripts from 9-11: Firefighters
The third communications analysis was conducted on the data in the 9-11 first responder transcripts between the Fire Department of New York dispatcher and various field units during the 9-11 attack on the New York City World Trade Center. Firefighters on 9-11 were performing search and rescue and working to extinguish the fire.

Transcripts from 9-11: NORAD
The forth communications analysis was conducted on the data from September 11, 2001, between the North American Aerospace Defense Command (NORAD), air traffic controllers in New York, Boston, Washington, and Cleveland when they discovered that four American commercial airliners had been hijacked. We were interested in the collaboration that occurred between NEADS, their counterparts at the Federal Aviation Administration (FAA) and various other air traffic control centers in order to provide military air support and ground civilian air traffic control over the United States. Transcripts of recorded audio from the command and control center at NEADS were coded and analyzed in an effort to use a real world example to empirically validate the slightly modified model of team collaboration, developed by the Office of Naval Research. Table 1 presents the percentage of speech turns for air warfare, 9-11 firefighting, and the NORAD/FAA communications coded as representing each of the macrocognitive processes included in the model. The high percentage of speech turns coded as categories 1-4 reflects the huge emphasis on individual knowledge construction that is required for all three tasks. The percentage of speech turns coded as one of the four macrocognitive process categories in the knowledge construction phase are 39%, 71% and 34% for air warfare, 9-11 NYC firefighters, and NORAD respectively. Individual task knowledge development (#3, itk) is defined as a team member asking for clarification to data or information, or a response to a request for clarification. The large number of speech turns coded as itk reflects the high degree of uncertainty inherent in these decisionmaking tasks. (Note: For the NORAD coding we used a modified version of the model so the coding for this event is not shown in Table 1.) Evidence was found for all six cognitive processes that occur during the collaborative team problem solving phase (7-13), where teams integrate individual knowledge to develop a team common understanding, indicating the role these cognitive processes play for teams who engaged in all three tasks. Far fewer speech turns were coded as representing processes included in the team consensus phase of collaboration (14-18). One striking difference between the firefighter and NORAD data and the air warfare data is the difference in the percentage of speech turns coded individual task knowledge development, "itk" (team member asking for clarification to data or information). Twenty-five percent of the total number of speech turns in air warfare scenarios were coded "itk" compared to a significantly larger percentage, i.e., 41%, for firefighter data (p<.01) and 41.2% for NORAD (p<.01). This, most likely, reflects the extraordinarily high level of ambiguity inherent in these completely unanticipated events. Another significant difference is in the Collaborative Team Problem Solving phase differences among three collaboration groups as shown in Table 1. The air warfare scenarios percentage was significantly higher (p<.01) with 23.7%, compared to both the firefighters (8.5%) and NORAD (11%). The small number of speech turns that were coded as cognitive processes that occur during the team consensus stage (14-18) and outcome evaluation and revision stage (19)(20) indicates that outcome evaluation and revision for air warfare, NORAD, and firefighting tasks is not conducted in a collaborative manner.

RESULTS
The focus of the collaboration is to keep all team members apprised of the situation so that all team members can maintain overall situation awareness, especially when dealing with as large a problem situation as the attack on the WTC. As was the case with air warfare, the 9-11 firefighters collaborated more about the "front end" of the problem: "What's going on?" A much smaller percentage of communications were devoted to collaboration during the team consensus and outcome evaluation and revision phases of the model. If this same pattern holds for other examples of team collaboration it has implications for designing collaboration systems.

9/11 Firefighters: ITK Communications
A review of the Fire Department of New York (FDNY) transcript revealed that many of the "itk" speech turns that transpired between team members were related to team members verbally attempting to clarify information including the scope of the problem, determining locations for ingress/egress routes, staging areas, etc., and questions regarding the application of limited resources at the right time and location. The need was to develop team member's individual knowledge and ultimately a team shared understanding of the courses of action needed to deal with this novel situation.
One typical example involved FDNY Engine 8-3 asking for assistance from the dispatcher to clarify the location of the station and directions under a very stressful situation. This particular communication thread was interrupted repeatedly by other units competing for the attention of the dispatcher including reports of people jumping off the World Trade Center Towers. This example only lasted about three minutes but contained potentially many life and death decisions interspersed with important but lower priority items regarding coordination and collaboration. The requirement to filter out "noise" (which may be knowledge to others) hindered individual and team knowledge development.

9/11 Firefighters: Technology Solutions
The FDNY team collaboration suffered from both communication system and working memory overload of team members. In a redesigned system, a good deal of the data might have been distributed by other means to decrease the overloaded communications channels and reduce the cognitive demands on human working memory so that higher-level cognition collaboration could take place. First responders typically use mobile data terminals (MDTs) while en route, in their vehicles, and one solution would be to include a GIS combined with a global positioning system (GPS) on the MDTs. Review of the transcript suggests a web portal with standard protocols (TCP/IP/HTTP) with GIS and common portal features, which would provide alerts (RSS newsfeeds, as one example) on diverse platforms which can operate on a range of platforms. The McKinsey report did suggest a similar course of action including standard protocols to reduce "stove pipes" and improve interoperability. A web portal could provide these improvements to reduce information overload by eliminating, or reducing, the number of lower-priority information requests flooding the communications system and allow team members to collaborate on higher and more complex tasks. CapWIN (Capital Wireless Information Net) is a specific portal example discussed later.

9/11 NORAD/FAA: Reducing the high level of information requests
While the tasks are different than the 9/11 tasks, the NORAD transcript shows similar types of speech turns. As with the firefighters, NORAD/FAA devoted much of their time to data verification such as aircraft tail numbers, telephone numbers, and track numbers and not as much time on understanding the problem and developing solutions. In addition to a portal, specific features would include access to a directory that would expeditiously provide the information such as tail numbers, tracks, and points of contact. The problems faced NORAD is similar to the air warfare experiment. From a human factors perspective this suggests a possible requirement for a GIS (Geographical Information System) to facilitate situational awareness development and understanding of the problem.

9/11 NORAD/FAA: ITK Solutions
In addition to a portal, specific features would include access to a database. The problem however, is deeper than technology. The 9/11 Commission found organizational dysfunction and those issues would have to be resolved before effective use of KM technology could be implemented. The FBI Information System, called Trilogy, tried to solve the technology problem before organizational problems were resolved and wasted millions on Trilogy (National Research Council, 2004).

DISCUSSION
Although two different knowledge domains, the NORAD and firefighter results obtained by using the cognitive process definitions included in the model of team collaboration to code team communications indicate that a large number of firefighter and NORAD communications involved clarifying information and attempting to develop knowledge interoperability. There were several instances where locations, units, tail numbers, and tracks were incorrectly identified and information on staging areas and flight zones was vague and incomplete. These communications might have been streamlined through better procedures and technology.
The 9-11 Report listed some of the deficiencies of NORAD/FAA collaboration: • NORAD heard nothing about the search for American 77. Instead, the NEADS air defenders heard renewed reports about a plane that no longer existed: American 11.
• NEADS never received notice that American 77 was hijacked.
• NORAD had heard nothing about United 93.
• No evidence that, NORAD's top commanders, in Florida or Cheyenne Mountain, coordinated with their counterparts at FAA headquarters to improve awareness and organize a common response.
While the NORAD/FAA deficiencies had more to do with organizational and political problems, the high percentage of requests for clarification of data or information clearly indicates there is another dimension: technology and how it could aid decision makers in these information-rich, complex, dynamic tasks.
Recommendations of the McKinsey report were to have more collaborative exercises as well as the development of better command and control and decision support system technologies. Results obtained through coding team communications by using definitions of cognitive processes included in the model seem to validate those conclusions. The model has potential diagnostic value when it can show where there may be problems with the collaboration that transpired between the team members. This would explain part of the difference in the percentages of speech turns coded as "itk" but many of the speech turns were related more to data (numbers and places) than higher order communications. Any reduction of these lower-level kinds of communications, i.e., repeated requests for data and information, would allow more time for the team development of a better understanding of the unfolding situation and how to respond in a time compressed environment. One example is the Capital Wireless Information Net (CapWIN), a web based, interoperable first responder data communication and information sharing network partnership between the State of Maryland, the Commonwealth of Virginia, and the District of Columbia (see http://www.capwin.org). This web-enabled collaboration tool provides maps (GIS), graphics, and other information that can provide a better understanding of the situation. It can also reduce the lower-level coordination communication requirements, thereby reducing the cognitive load on team members to provide more time for higher-level collaboration between teams and organizations. The McKinsey report (2000) commissioned by New York City noted that first responder communications protocols needed standardized radio procedures to "improve the flow of vital information among agencies to ensure it is clear and unambiguous, appropriately prioritized, and reaches the appropriate parties in a timely fashion during incidents and in day-today operations."

CONCLUSIONS
Analysis of data captured from teams performing their tasks in a collaborative environment can provide valuable insight into what constitutes effective collaborative performance. This understanding can then be used to develop technology to support this cognitive activity, develop tools to reduce cognitive workload, and techniques and processes to improve information exchange among collaborating members. Clearly a "one size fits all" set of collaboration tools is unlikely to meet the diverse problem domains of teams and groups.
Too often the design process is done without the benefit of human factors practitioners and the many effective techniques they employ. The typical information technology approach focuses on user desires without considering the cognitive components that need support. Improving the quality of decisionmaking is a multi-dimensional challenge involving technical, socio-technical, and organizational perspectives. Our model can be used in this context to provide a direction as to the types of the collaboration tools needed for each problem domain through a review of the statistics in conjunction with an analysis of the speech turns and how they fit into the team collaboration model. In addition to the traditional workflow and UML use case analysis, the techniques employed by NDM practitioners should be part of the analysis of technology selection for team collaboration. This approach can act as a bridge between the human factors professionals and the IT/KM system designers. The approach helps the human factors professionals point the KM developers in the general direction of the KM tools required without dictating a specific KM solution that could hamstring the KM developers.