Combining Naturalistic and Mathematical Decision Aids to Support Product Design

Motivation – To increase the benefits that product designers derive from use of computer decision aids in their daily design work. Research approach – The process followed by product designers was observed through a combination of ethnographic and protocol studies to identify needs and constraints in a typical work context. Findings – The results suggest that design decisions are tightly intertwined with information seeking activities, and require great exploration flexibility. Research Implications – the benefits and appeal of decision aids might be greatly increased for product designers by additionally supporting information seeking activities that inform decisions. Originality/Value – This approach represents a paradigm shift towards a broader view of decision making as flexible set of activities that includes assessment of information adequacy and information seeking. Take away message – Practical decision aids for design need to augment rather than replace human judgment, and support a range of intertwined activities while limiting the data entry burden.


INTRODUCTION
Rapid, high quality product design is essential to competitiveness in the global economy; and effective decision making is essential for producing high quality product designs.The long-term objectives of this work are to improve product designers' decision making abilities by developing a new decision-aid paradigm that synergistically combines mathematical and naturalistic decision methods.There are a wide range of mathematical decision methods available which aim to help people approach decision making in a more systematic and rational way than they might, if left to their own devices.In contrast, naturalistic decision making is an approach which focuses on understanding the needs of designers and problem solvers in the environments in which they would normally work.This approach is important because it brings to light practical constraints which might other wise go undetected if decision making is considered to be an idealized activity taking place under idealized conditions in which complete information, or statistically accurate stigmatizations, are available.
Most mathematical decision methods were developed with using the assumptions that goals are fairly well defined, and the criteria for making decisions are well understood, and information, or good estimates (or statistical distributions) can be obtained.These assumptions rarely hold in any reasonably interesting or complex design problem.They are illdefined; goals and criteria may only be come to be understood through a process of creative exploration that is the heart of the design process.Information may be unavailable, and estimates used may not be accurate.Design decisions may not fit the assumptions of many mathematical decision making approaches.However, this is not to say that mathematical decision methods are not useful, quite the contrary.They are used beneficially in a wide variety of design applications.However, they are underutilized in product design.Designers in industry report they may use mathematical methods to verify major choices after they have made those choices, particularly when justifying them to management.The explanation offered is that they are often "too busy" to use these methods in all their design work.They perceive them as being burdensome.The assumptions underlying this research are that perceived "burden" of these tools results in part from the mismatch between the assumptions of mathematical decision models and the practical needs of designers in the workplace, and that the usability, appeal, and practical utility of such methods can be increased by combining naturalistic and mathematical approaches.This work uses naturalistic decision making methods of study including ethnographic studies and protocol analysis (think aloud studies of problem solving) to understand how product designers go about their work.Additionally, we will use practices of user-centered design to develop and test decision aids which embody our emerging combined mathematical and naturalistic approach to decision making.These methods involve rapid prototyping and testing and user testing with realistic tasks.The decision aids will focus on simple approaches that support flexibility, complement rather than replace human judgment, minimize data entry.The investigator has identified information seeking as a critical design activity that needs to be integrated with decision making.The contributions include the introduction of a naturalistic decision making approach to increased practical acceptance for mathematical decision making methods in product design and other complex decision making contexts, as well as an increased understanding of decision making activities in product design.

RELATED LITERATURE
Following is a short summary of literature on 1) mathematical decision making methods, 2) naturalistic decision making and 3) the nature of product design processes.

Mathematical Decision Making Methods
Multi-criteria decision making (MCDM) techniques (Saaty, 1980;Klein, 1993) are a broad family of mathematical methods that compare alternatives in a set, using multiple criteria.For example, a prospective car buyer might compare his or her car choices by criteria such as fuel efficiency, cost, and comfort.The criteria used may vary from buyer to buyer depending on what is most important to that particular person.One common MCDM method is the weighted sum method (Hayes, J. R., 1981) in which each term in the sum represents how well an alternative fulfills a given criterion, and the term's weight represents that criterion's importance to the decision maker.Variants of the weighted sum method are popular because they are relatively easy to understand and use.
One can further divide MCDM (and weighted sum) methods into deterministic and non-deterministic methods.Deterministic decision making methods are those that do not explicitly incorporate a representation of uncertainty, for example, the cost of an option may be represented as a specific number or point value.While the decision maker may understand that this number is not exact, the degree to which it is not exact is not represented.In contrast, nondeterministic decision making methods are those that incorporate some explicit representation of uncertainty or unknowns.For example, the uncertainty in the cost of an option may be represented as a range of possible costs or as a function describing the likelihood of various costs.
Vagueness and ambiguity can be modeled by many techniques including those based on fuzzy set theory (Thurston & Carnahan, 1992).The merit of fuzzy techniques is that imprecision (Bellman & Zadeh, 1970) is recognized as an element of the decision model.The drawbacks of such techniques are the relatively high computational effort required for modeling the decision situation and processing the input information (Law, 1996).However, while much research has focused on the development of formal decision making methods, relatively few studies have assessed their practical utility and impact in complex tasks.In the laboratory study summarized later, deterministic and non-deterministic (fuzzy) decision making methods were compared against designers' typical, informal methods (Hayes and Akhavi, 2008).

Naturalistic Decision Making
Goals.Naturalistic decision making (NDM) aim to understand and improve decision making in field settings, so that better tools, training and supports can be provided which will work in those settings (Salas & Klein, 1991).In this approach, decision making is studied in the context of actual tasks and the environments in which they are typically carried out.These environments are often work environments (Salas & Klein, 2001;Klein, 1993;Suchman, 1987).Some of the premises underlying research in naturalistic decision making are that traditional, mathematically-oriented decision making research focuses on only one part of decision making, the decision event (Orasanu & Connolly, 1993).In a decision event, a single decision maker compares a fixed set of alternatives using a fixed and well defined set of goals.Additionally, if precise information on the performance of each alternative is not available, statistical estimates can be obtained.
However, this is a rather limited view of decision making (Klein, 1993).In real situations, few of the assumptions above may hold (Dym, 1994;Thomas & Carroll, 1984;Ullman et al., 1988).Furthermore, decision making cannot be studied in isolation from related processes such as problem solving, uncertainty management, or the development of expertise (Salas and Klein, 2001).
Methods.Methods used for studying naturalistic decision making and include a variety of methods including cognitive task analysis, ethnographic studies and other observational methods (Montgomery et al., 2004).These methods focus on studying decision makers in environments as close to "natural" as possible, so that environmental constraints that impact decision making, such as stress or lack of information, can be understood.Ideally, the results of "field" studies can be combined with more controlled laboratory studies to provide a variety of perspectives on the task.
Naturalistic Decision making and Product Design.Because each task and environment has different characteristics and brings different challenges, each needs to be studied to identify its unique challenges.Naturalistic decision making has been studied in a wide range of domains from traffic management and mission analysis, to health care and space exploration (Salas & Klein, 2001;Schraagen et al., 2008).However, product design has not yet been greatly studied through naturalistic decision making approaches although it is an excellent example of a complex task that is not well modeled as a decision event.Alternatives are constantly being added, modified, or refined, as are design goals.Much information is simply unknown or hard to obtain.Additionally, there are practical time considerations; the sheer number of decisions that must be made in creating a design necessitates that most must be made very rapidly.Finally, many important design activities are not captured by a decision event, for example, information seeking behaviors which precede the selection of an alternative, as will be described later.Thus, one of the contributions of this work is to bring naturalistic decision making approaches to product design.

Product Design Processes
Because decision making cannot be fully understood without also understanding the larger process of which it is a part, the following section will summarize some of the existing literature describing the salient properties and structure of design processes.The models described were developed primarily in the context of mechanical design processes.However, the general characteristics of most complex design processes are essentially similar.

Uncertainty in design.
Uncertainty is present in all designs (Aughernbaugh & Paredis, 2006), from hand-held computer devices to space station systems.Even when a design is considered to be complete, there may still be uncertainty concerning issues such as the performance of the design under all the conditions to which it may be exposed in its working life, its manufacturing feasibility, or the final cost.Uncertainty is most prevalent in the early stages of design, also known as conceptual design, when the alternatives under consideration may be little more than quick sketches or brief outlines (see Figure 1).
Descriptions of the design process.At all stages of the design process, designers must repeatedly choose the most promising alternative(s) for further development.These could be major choices such as, "To best assist a quadriplegic man in his office, should one design a wheel-chair mounted robot arm, a desk mounted robot arm, or a device he can hold in his mouth to push objects on his desk?Alternatively, they could be more minor choices about how to implement the details in a particular design, for example, "Should the robot arm be made of aluminum or a composite material?"This process of winnowing alternatives is known as down selection; this is the step that most people think of as "decision making" because alternatives must be compared and the best one(s) chosen.Conceptual design, down selection, and their relationship to the overall design process are shown in Figure 1.The model of the design process shown in this figure represents the investigators' integration of several design process models (described below) with their own observations, to create a unified design process model.
There is an overall progression in the design process from conceptual design to detail design as shown in Figure 1, but there is no distinct dividing line between these stages (Pahl & Beitz, 2006).The design is gradually transformed from a set of sketchy alternatives into one (or more) fully detailed final design.The transformation occurs through many iterations in which designers explore, develop, and eliminate many alternatives (Simon, 1985;Blanchard & Fabrycky, 2006).The activities performed in each iteration or the design process are represented by the labels in Figure 1, such as requirements gathering, design review, and down selection.
However, design activities rarely proceed in a precise and orderly progression.Ullman, Dietterrich & Stauffer (1998) performed protocol studies of mechanical design processes in which designers were asked to "think aloud" as they created designs to solve engineering requirements.They found that, in practice, there is much jumping back and forth between steps.From these studies, they developed the Task/Episode Accumulation (TEA) model in which designers incrementally refined and patched design alternatives in a series of design episodes.The models of the design process, described above, set the larger context for the proposed work; decision making (down selection) is tightly intertwined with the other design activities and cannot entirely be separated from them.The work reported in this article deepens prior work by providing a detailed study of down selection in the laboratory and in the context of actual design tasks.

A STUDY: COSTS AND BENEFITS MCDM DECISION AIDS IN DESIGN TASKS
An essential issue in this research was to first assess some of the benefits and costs of mathematical decision making approaches in naturalistic design situations.Once these can be understood, one is in a better position to know where improvements are needed and where they are not.Most work in mathematical decision making methods simply assumes that the methods will improve decision making (almost by definition) without examining their suitability or impact in practical contexts.Towards this goal, in preliminary work the principle investigator and colleagues studied expert and intermediate-level designers as they solved actual design problems in order to assess (Hayes & Akhavi, 2008): • What are the costs and benefits of various decision approaches when used in product design?
• Do decision aids that represent uncertainty help more than those that do not?
• Do expert and intermediate-level designers derive different benefits from decision aids?
Experimental set up.The task studied was design of a manned lunar excursion vehicle.Examples of concept designs (i.e. initial sketches) for this domain are shown in Figure 2. The goals were to design a vehicle that would provide occupants with protection, life support, mobility, towing capability, communication, sufficient power, etc., for an average excursion.The excursion vehicle must also fit inside the launch vehicle for deployment at the landing site.Twenty six subjects participated in the study.Eighteen senior engineering students were enlisted as intermediate-level designers (they had too much design experience to be considered novices), and eight faculty with experience in design of space vehicles served as expert designers.Each subject was asked to use three decision making methods to rank sets of four design alternatives, like those shown in Figure 2, from best to worst.The methods used were: 1) A non-deterministic computer decision aid based on a fuzzy, multi-criteria weighted sum method, in which values for each alternative and criteria were entered as single number (e.g. the price of this alternative is about $10) 2) A deterministic computer decision aid based on a standard multi-criteria weighted sum method, in which the values were represented as a range (e.g. the price is between $6 and $17).
3) No decision aid.(This was the control condition).
We wanted to compare at least one deterministic with a non-deterministic method because we assumed that the latter would be well suited to the uncertain nature of design decisions, and might help designers in the reasoning about it.Twenty six subjects participated in the study.Eighteen were senior undergraduates in a capstone design course, and eight were engineering design professors.The students were considered to be intermediate level designers (not novices) and the professors were experts in lunar vehicle design.Results are summarized in Figures 3 and 4. Figure 3 shows the degree to which subjects' ranking of alternatives agreed with the average rankings of the experts in the study.This was used as an indicator of decision quality (while recognizing that it is not a direct measure, which can be difficult to obtain for the reasons described below.)Both decision aids improved the experts' decision quality, but neither significantly improved the intermediates' decisions.Furthermore, contrary to our expectations, the non-deterministic (fuzzy) aid, did not appear to provide improvements over the deterministic aid.

Inte rme diate s' Ave rage Spe arman
One definition of decision quality is the degree to which a decision maker's rankings accurately reflect rankings computed from empirically measured performance data gathered after the design alternatives are actually built and deployed (Akhavi & Hayes, 2008).However, most design alternatives are never built, so one can rarely assess quality directly.As an alternative, one can have experts to assess the quality of alternative designs that have not yet been built.While experts may not agree which is the "top" alternative, they tend to agree on what are good alternatives, and their rankings tend to be highly correlated (Hayes & Parzen, 1997).Thus, if one takes this as evidence that experts are indeed able to judge quality, then the correlation of a decision maker's rankings with those of experts is an indicator of quality.Figure 4 shows that both decision aids required significantly more time to reach a decision than did no decision aid, with the fuzzy (non-deterministic) aid requiring the most time.1 Observations of the subjects during the experiment indicate that much of this additional time resulted from data entry required by the decision aids, where the fuzzy aid required the most data entry.

In summary:
• Both decision aids simultaneously helped and hindered expert designers in terms of decision quality and time, respectively.This explains reports from industrial designers that while they sometimes use mathematical decision making methods, they often feel that they are "too busy" to use them all the time.Because of the extra time involved, they perceived it as burden to use decision aids in many situations.If ways could be identified to reduce the data entry burden, designers might use them in more situations.• Neither aid appeared to help the intermediate-level designers.This may have been because the intermediates were not as good as the experts at estimating the likely values of the various parameters.For example, when they estimated costs, their estimates were often quite unrealistic.Nor did they appear to have any concept of how inaccurate their estimates were, or when they should spend some time gathering more information to improve their estimates.• Furthermore, the non-deterministic aid did not appear more beneficial than the deterministic one, contrary to expectations.It was not clear why this was so, but we hoped to gain more insights through the observations of the design process described in the next section.

OBSERVATIONS OF DESIGN DECISION MAKING IN CONTEXT
To gain insights into these issues, we used protocol studies and ethnographic observations.Prior to the study described above, we observed the four lunar excursion vehicle design teams as they created designs for lunar excursion vehicles in their twice-a-week design meetings, over a period of several weeks.Observations were taken in their regularly scheduled design meetings.The protocol studies were done concurrently with each trial of the experiment described above.Subjects were asked to think aloud as they worked through each decision making task.
The ethnographic observations revealed that designers continually generated and evaluated alternative designs throughout the design process, some of which were entirely new concepts, while others were minor variants on existing alternatives.To keep the number of alternatives under current consideration manageable, designers continually engaged in down selection to prune out the less promising ones.They appeared to follow at least two different approaches to down selection that we call rapid elimination and considered comparison.Additionally, they were often observed to engage in information seeking if they did not feel comfortable with the amount of information available on each alternative.

Rapid Elimination
In this variant of the down select process, design alternatives were only briefly considered before being discarded on the basis of rapid, informal assessments.Alternatives were often discarded based on a single criterion.For example, a designer might say, "this option is way too expensive for our budget" or "that option is far more complex (mechanically) than our other options.I don't see the need to consider it further unless we are desperate."Options eliminated by this method were usually those that were clearly dominated by others (e.g., worse in all major criteria).
Rapid elimination is by nature imprecise and may sometimes lead to inappropriate elimination of alternatives (some of which were later revisited).However, it is also a very pragmatic approach and probably necessary given the enormous volume of down select decisions that must be made during a typical design process.If designers considered all decisions in depth, design progress would rapidly come to a stand-still.

Considered Comparison
In this variant of down selection, designers were observed to compare several options during group discussions, often comparing all alternatives by one criterion, then another, possibly revisiting one, and so on.They frequently added additional criteria discovered through discussion or revised initial estimates of criteria importance.Thus, unlike formal decision methods in which the criteria and their importance are pre-determined, designers mixed discovery and determination of criteria with the decision making process.It was only occasionally that designers performed this type of in-depth comparison.It occurred most commonly as a deadline was approaching at which time the team must select and justify a single "best" alternative which they would develop and prototype.In many but not all cases, the alternatives compared were not obviously inferior or superior to each other than those pruned through rapid elimination.This process might best be described as a very flexible and very approximate method in which criteria, importance weights, and values are continually added, subtracted, or modified.

Information Seeking
Choosing an alternative in the down selecting process is tightly tied with information seeking.At many points in the design process, designers lacked sufficient information to make informed comparisons between alternatives, particularly during conceptual design.Designers seek information through many methods.Sometimes they create the information themselves by developing more detailed drawings of targeted areas of a design or by building and testing prototypes.For example, information about the likely performance of the Mars Rover on Mars was obtained by building and testing prototypes in the harsh conditions of the Anaconda desert.Sometimes information is produced through analytical methods such as calculations and simulations.And sometimes it is collected from external sources, for example by searching the library and web or by calling multiple vendors to gather a range of price quotes.
Some information seeking activities require significant effort, knowledge, and cost (Gonzoles et al., 2005, Lipshitz et al., 2001).Designers must make judgments about when the cost of information seeking is likely to pay-off in the final product.An issue observed in senior undergraduate designer teams was that they did not always know when to seek more information or when to stop.Deadlines were very important in forcing them to think critically about what information was most important and to focus their information seeking efforts.

SUMMARY AND RECOMMENDATION
The laboratory studies showed that while some designers (experts) benefited from use of mathematically-based decision aids, they took far more time than using no aid.The extra time involved is consistent with industry designers' reports that they are hesitant to use these tools except to justify major decisions, because they take too much time.
Additionally, the ethnographic observations indicate that the mathematical decision making methods, as formulated, are not well matched to the approaches that product designers typically use which involve very flexible exploration of criteria and alternatives, many short cuts to focus search and information collection, and a tight intertwining of information seeking and other decision making activities.We feel that this suggests an opportunity; that decision support in product design be extended to support not just selection of an alternative, but also a range of activities that prepare the decision maker for task, including decisions about the adequacy of available information (is there enough information here to make an informed decision?), and the likely benefit of seeking more information (Bradley & Agogino, 1994).Through these adaptations, it may be possible to reduce the time required to use such MCDM decision aids, while increasing their usefulness to product designers, and their willingness to employ them in their everyday work.

Figure 1 .
Figure 1.An iterative model of the design process [Hayes and Akhavi, 2008].

Figure 2 .
Figure 2. Four design alternatives for a lunar excursion vehicle.

Figure 3 .
Figure 3. Average correlation of subjects' and experts' rankings using various decision methods.

Figure 4 .
Figure 4. Time required to rank alternatives using various decision methods.