Moving toward effective estimation and communication of uncertainty information has broad and deep implications for the Enterprise and the community it serves. Because of the immense breadth and depth of this challenge, detailed solutions are beyond the reach of a single committee. Consequently, this report provides general ideas for consideration by the National Oceanic and Atmospheric Administration’s (NOAA’s) National Weather Service (NWS) and the entire Enterprise.
The committee presents nine overarching recommendations in this final chapter, all with equal priority. In addition, detailed recommendations appear in the preceding three chapters that add further specificity and breadth. All recommendations should be considered in the context of NOAA’s Policy on Partnerships in the Provision of Environmental Information. In cases where a recommendation states that “NWS should …” it is the committee’s intention that the recommendation also applies to any relevant group or activity within NOAA, such as the Office of Oceanic and Atmospheric Research (OAR).
All prediction is inherently uncertain and effective communication of uncertainty information benefits users’ decisions (e.g., AMS, 2002; NRC, 2003b). The chaotic character of the atmosphere, coupled with inevitable inadequacies in observations and computer models, results in forecasts that always contain uncertainties. These uncertainties generally increase with forecast lead time and vary with weather situation and location. Uncertainty is thus a fundamental characteristic of weather, seasonal climate, and hydrological prediction, and no forecast is complete without a description of its uncertainty.
Nonetheless, many products from the Enterprise do not include uncertainty information, or they include it ineffectively. The deterministic forecasts that dominate the offerings from the Enterprise are often truncated representations of much richer information. Moreover, the information in deterministic forecasts is sometimes confusing or misleading, which can lead to poor decisions and undesirable consequences (NRC, 2003b). Decisions by users at all levels, perhaps most critically those associated directly with protection of life and property, are being made without the benefit of knowing the uncertainties of the forecasts upon which they rely.
Albert Einstein wrote, “[the right to search for truth] implies also a duty; one must not conceal any part of what one has recognized to be true.”1 Because the Enterprise has increasing capability to estimate forecast uncertainty and the ability to provide products that communicate this knowledge, it has the duty to do so in a way that is valuable to the users it serves.
Finding 1:2 Hydrometeorological services in the United States are an Enterprise effort. Therefore, effective incorporation of uncertainty information will require a fundamental and coordinated shift by all sectors of the Enterprise. Furthermore, it will take time and perseverance to successfully make this shift. As the nation’s public weather service, NWS has the responsibility to take a leading role in the transition to widespread, effective incorporation of uncertainty information into hydrometeorological prediction.
Recommendation 1: The entire Enterprise should take responsibility for providing products that effectively communicate forecast uncertainty information. NWS should take a leadership role in this effort.
Letter, March 14, 1954.
See Section 1.5 for further discussion on this topic.
Product Development Incorporating Broad Expertise and Knowledge from the Outset
Finding 2:3 Understanding user needs and effectively communicating the value of uncertainty information for addressing those needs are perhaps the largest and most important tasks for the Enterprise. Yet, forecast information is often provided without full understanding of user needs or how to develop products that best support user decisions.
Parts of the Enterprise (e.g., within the private sector and academia) have developed a sophisticated understanding of user needs. In addition, there is a wealth of relevant knowledge in the social and behavioral sciences that could be more effectively incorporated into product research and development. Currently, this variety of resources is not being fully tapped by NOAA,4 and user perspectives are not incorporated from the outset of the product development process.
Recommendation 2: NOAA should improve its product development process by collaborating with users and partners in the Enterprise from the outset and engaging and using social and behavioral science expertise.
Education on Uncertainty and Risk Communication
Finding 3:5 Enhanced Enterprise-wide educational initiatives will underpin efforts to improve communication and use of uncertainty information. There are three critical areas of focus: (1) undergraduate and graduate education; (2) recurrent forecaster training, and (3) user outreach and education.
Recommendation 3: All sectors and professional organizations of the Enterprise should cooperate in educational initiatives that will improve communication and use of uncertainty information. In particular, (1) hydrometeorological curricula should include understanding and communication of risk and uncertainty, (2) ongoing training of forecasters should expose them to the latest tools in these areas, and (3) forecast providers should help users, especially members of the public, understand the value of uncertainty information and work with users to help them effectively incorporate this information into their decisions.
Finding 4:6 The ability of NOAA to distribute and communicate uncertainty information is predicated on the capacity to produce post-processed probabilistic model guidance on a variety of spatial scales. Currently, NOAA maintains long-range (global) and short-range ensemble prediction systems. However, the short-range system undergoes no post-processing and uses an ensemble generation method (breeding) that may not be appropriate for short-range prediction. In addition, the short-range model has insufficient resolution to generate useful uncertainty information at the regional level. For forecasts at all scales, comprehensive post-processing is needed to produce reliable (or calibrated) uncertainty information.
Recommendation 4: NOAA should develop and maintain the ability to produce objective uncertainty information from the global to the regional scale.
Ensuring Widespread Availability of Uncertainty Information
Finding 5:7 NWS, through the National Centers for Environmental Prediction (NCEP), produces a large amount of model output from its deterministic and ensemble numerical weather prediction models. The ensemble forecasts and output from statistical post-processing (i.e., Model Output Statistics) already produce a wide variety of uncertainty information. However, both the model output and statistical information regarding its skill are difficult to access from outside NCEP. Thus, NWS is missing an opportunity to provide the underlying datasets that can drive improved uncertainty estimation and communication across the Enterprise.
Recommendation 5: To ensure widespread use of uncertainty information, NWS should make all raw and post-processed probabilistic products easily accessible to the Enterprise at full spatial and temporal resolution. Sufficient computer and communications resources should be acquired to ensure effective access by external users and NWS personnel.
Broad Access to Comprehensive Verification Information
Finding 6:8 To make effective use of uncertainty products, users need complete forecast verification information that measures all aspects of forecast performance. In addition, comprehensive verification information is needed to improve forecasting systems. Such information includes previous numerical forecasts, observations, post-processed uncer-
See Sections 2.4, 4.2.6, 4.2.7.
Recognizing that private-sector entities gain a competitive advantage through knowledge of user needs, there is, nonetheless, some opportunity for information sharing that could significantly improve the effectiveness and efficiency of product development.
See Section 4.2.8.
See Section 3.5.
tainty information, and detailed verification statistics (for raw and post-processed probabilistic forecasts).
Verification measures and statistics need to evaluate all aspects of forecast performance that are relevant for use or improvement of the forecasts. Single scalar measures of forecast performance—commonly presented on NWS Web sites—are not adequate to meet these requirements. In many cases, the verification statistics that are provided by NWS are difficult to obtain, overly aggregated, or inappropriate for probabilistic forecasts. Thus, only a small fraction of the information needed by users and model developers is available to them.
Recommendation 6: NWS should expand verification of its uncertainty products and make this information easily available to all users in near real time. A variety of verification measures and approaches (measuring multiple aspects of forecast quality that are relevant for users) should be used to appropriately represent the complexity and dimensionality of the verification problem. Verification statistics should be computed for meaningful subsets of the forecasts (e.g., by season, region) and should be presented in formats that are understandable by forecast users. Archival verification information on probabilistic forecasts, including model-generated and objectively generated forecasts and verifying observations, should be accessible so users can produce their own evaluation of the forecasts.
Effective Use of Testbeds
Finding 7:9 Testbeds are emerging as a useful mechanism for developing and testing new approaches and methodologies in estimating and communicating uncertainty.10 The effectiveness of testbeds is limited when all appropriate sectors of the Enterprise are not included.
Testbeds are multipartner collaborations that create prototypical environments where innovative approaches can be tested before being applied more generally. They allow the community to evaluate new modes of cooperative research, development, training, and operations. Testbeds allow (1) the operational community to benefit from strengthened connections to academia (including decision and social sciences) and the private sector; (2) academia to benefit from exploring new research areas at the interfaces of physical, social, and decision sciences, and from hands-on student training; and (3) the private sector to provide input on science and technology development and ensure that new approaches are responsive to private-sector requirements. In moving toward improved characterization and communication of forecast uncertainty information, testbeds can, for example, play a critical role in developing the technology of probabilistic prediction, evaluating ways to communicate such information and garnering active input and interactions with user communities. Although NOAA has successfully participated in testbeds in areas such as hurricane prediction and hydrology, no testbed activity currently exists in the critical area of probabilistic prediction using ensemble techniques.
Recommendation 7: To enhance development of new methods in estimation, communication, and use of forecast uncertainty information throughout the Enterprise, and to foster and maintain collaboration, confidence, and goodwill with Enterprise partners, NOAA should more effectively use testbeds by involving all sectors of the Enterprise.
Enterprise Advisory Committee
Finding 8:11 Only through comprehensive interaction with the Enterprise will NWS be able to move toward effective and widespread estimation and communication of uncertainty information. One mechanism for engaging the entire Enterprise on this and other Enterprise-related topics is an independent NWS advisory committee with broad representation. Such a committee is under consideration by NOAA in response to a recommendation in the Fair Weather report (NRC, 2003a).
In 2003, the National Research Council recommended that “NWS should establish an independent advisory committee to provide ongoing advice to it on weather and climate matters. The committee should be composed of users of weather and climate data and representatives of the public, private and academic sectors, and it should consider issues relevant to each sector as well as to the set of players as a group, such as (but not limited to)
improving communication among the sectors,
creating or discontinuing products,
enhancing scientific and technical capabilities that support the NWS mission,
improving data quality and timeliness, and
disseminating data and information.”
Recommendation 8: The committee endorses the recommendation by the Fair Weather report to establish an independent advisory committee and encourages NOAA
See, for example, Joint Hurricane Testbed (http://www.nhc.noaa.gov/jht/), Weather Research and Forecasting (WRF) Developmental Testbed Center (http://www.dtcenter.org/index.php), NOAA Climate Testbed (http://www.cpc.ncep.noaa.gov/products/ctb/), NOAA Hydrometeorology Testbed Program (http://hmt.noaa.gov/).
See Section 4.2.6 and overarching recommendation 2.
to bring its evaluation of the recommendation to a speedy and positive conclusion.
Finding 9:12 Incorporating uncertainty in forecasts will require not only the attention, but also the advocacy of NWS management. Given the scope of this challenge, the level of effort involved will demand a “champion” within the NWS leadership—an individual who can effectively organize and motivate NWS resources and engage the resources and expertise of the entire Enterprise.
There is recent precedent for such an approach at NWS, admittedly at a more technical level: the NWS WRF model coordinator is to implement the WRF program.
Recommendation 9: NWS should dedicate executive attention to coordinating the estimation and communication of uncertainty information within NWS and with Enterprise partners.
See Section 1.5.