Visualizing Temperature Forecast Uncertainty for a Non-expert Web Audience

Motiviation - The goal of the studies reported here was to determine the optimal way to present web-based uncertainty forecasts for temperature. Research Approach - We focused on the 80% predictive interval, bounded by the temperatures at the 90th and the 10th percentiles of the probability distribution for temperature. Findings/Design - We found that non-expert users can understand novel forecast uncertainty information presented in a web-based format and they were able to put the information to good use, improving the quality of their decisions over those with the deterministic forecast alone. Research Limitation/Implications –Presentation format affected the ease and accuracy with which this information was extracted. Originality/Value – To date there has been very little empirical testing on the value of uncertainty visualizations. Take away message – Our research shows that non-experts can understand the benefit for forecasts with uncertainty and some presentation formats facilitate these.


INTRODUCTION
Decisions involving weather conditions can have enormous safety and economic consequences.However, at present, the general public is provided only part of the relevant forecast information upon which to base such decisions.They are given a single, "deterministic" value that fails to communicate the essential uncertainty involved or the full range of potential outcomes.
Although the methods for modeling weather forecast uncertainty have improved in recent years, most uncertainty information is withheld from public forecasts due to the fear that it will be misunderstood and improperly used by nonexperts.Indeed, there is evidence that probability of precipitation, the only uncertainty regularly included in public forecasts, is misunderstood decades after it was introduced (Gigerenzer et al. 2005).
These concerns are compounded by the fact that most people get weather information from mass media sources such as television, radio or the World Wide Web (NRC, 2006) where complex explanations of the statistics would be inappropriate and likely ignored.Furthermore, web-based forecasts, the focus of the studies reported here, must be useful for a wide audience with varying needs, concerns and levels of sophistication.Consequently, the information must be presented in a manner that is clear and easily grasped by a wide range of users who spend on average a minute viewing such websites.1

METHODS
The study reported here was designed to test expressions of the 80% predictive interval to determine which is most successful in conveying an accurate understanding of uncertainty and leads to the best decisions.All of the experimental forecasts tested had two parts, a visualization and a textual key.We tested two visualizations (bracket and plus/minus signs) and three keys (verbal, probability, frequency) (see Figure 1) and compared them to a condition with no visualization.
The purpose of the key was to explain the 80% predictive interval.It was initially selected because it has the following simple definition: There is an 80% chance it will contain the observed temperature.The same idea could be expressed as frequency ("8 times in 10").Frequency expressions are believed to be a better reflection of natural experience because they describe uncertainty as number of events experienced (Cosmides & Tooby, 1996;Hertwig & Gigerenzer, 1999), whereas probability or chance is an abstract theoretical construct.A third way this idea could be expressed omits numbers: It is likely that the interval will contain the observed temperature.Some argue low numeracy skills among the general public make non-numeric expressions of uncertainty preferable (National Research Council [NRC], 2006), although there is evidence that much precision is lost with verbal expressions (Mosteller & Youts, 1990).
The three keys (verbal, probability, frequency) combined with each of the visualizations (bracket, plus/minus) to create six experimental conditions.In addition, each of the visualizations was presented without a key to create three additional control conditions.These were compared to the conventional deterministic forecast.In all conditions, participants were given the same agricultural scenario involving protecting crops against freezing conditions and a twoday forecast.Participants made decisions based on the forecast and answered questions that probed their understanding of the forecast and the uncertainty in the forecast.

FINDINGS AND RECOMMENDATIONS
Results suggest that uncertainty visualization had a significant impact.Participants made better decisions with uncertainty forecasts, issuing a freeze warning when that was appropriate (high likelihood of 32ºF).All participants understood that temperatures less similar to the deterministic forecast were less likely to be observed.However, the answers to several questions indicated that those with the uncertainty visualization had the best understanding of the amount of uncertainty involved.In addition those with uncertainty forecasts were able to select the most reliable forecast, while those with the deterministic forecast were not.Finally, we found increased errors in the "verbal only" and "plus minus" conditions suggesting that they are not optimal for presenting uncertainty in this context.

CONCLUSIONS
This study demonstrates that non-expert users can understand novel forecast uncertainty information presented in a web-based format.The evidence also suggests that participants naturally anticipate the uncertainty associated with predictions of future weather, which may prepare them to understand the information provided.However, presentation format affects the ease and accuracy with which this information can be extracted, suggesting that presentation is critical for this medium.Importantly, not only did participants understand the amount of uncertainty involved in individual forecasts, but also they were able to put the information to good use, improving the quality of their decisions over those with the deterministic forecast alone.