From Research to Operations in Weather Satellites and Numerical Weather Prediction: Crossing the Valley of Death (2000)

Operational weather forecasting in the United States has evolved considerably since the formation of the U.S. Weather Bureau in 1870 as a component within the U.S. Army Signal Service. From an initial effort that was largely observational, increasing scientific understanding and technological advances have permitted the development of a forecast system that is based on models that incorporate physical understanding of the earth's coupled atmospheric and oceanic system. As a result, the nation's weather services now have the ability to forecast weather and climate events and patterns. Until the end of World War II, the federal government provided virtually all weather services. Since then, however, the demand for specialized meteorological services has led to the development of effective private sector organizations that offer forecast products and services.

The U.S. National Weather Service (NWS) still plays the central role in the provision of operational weather data and services in this nation. It is responsible for the maintenance of the fundamental observing networks, the core forecasting infrastructure, and the provision of severe weather and flood warnings necessary for the protection of life and property. The NWS consists of a network of 120 Weather Forecast Offices spread more or less uniformly across the country, 13 River Forecast Centers covering the major river basins, and the National Centers for

Environmental Prediction (NCEP) providing guidance products⁷ for the rest of the NWS, other government agencies, and the private sector.

NCEP is composed of several centers that provide specialized products to the meteorological community (see Appendix B for a description of the NCEP structure and the functions of its major units). This chapter focuses on linkages, actual and potential, between research and operations at NCEP's Environmental Modeling Center (EMC), which has the major responsibility for implementing and operating the numerical weather and climate prediction models that serve as the basis of modern forecasting operations. Both government and private sector activities use EMC products.

The mission of EMC is to “Develop, enhance, and maintain numerical forecast systems in support of national and international forecast requirements” (Lord, 1999). EMC operates numerical forecast systems for weather prediction (global and regional, 1-15 days), ocean prediction (global, daily to annual) and climate prediction (seasonal to interannual). It is responsible for enhancing numerical forecasts through improvements in data assimilation techniques, model physics, and numerical methods. EMC is also responsible for adapting the production programs to new hardware systems. It must remain responsive to changes to the quality and quantity of the input data and to the changing operational requirements for products and services. EMC's mission is accomplished through a blend of in-house research and development and cooperative alliances with the external research community. Forty-seven percent of the funding for the EMC staff is provided by the base budget, and 53 percent comes from soft money.⁸

Although the mission appears well defined, previous reports, as well as presentations to BASC, reveal prioritization and staffing issues that indicate EMC capabilities are already stressed, and therefore, the task of meeting society's growing expectations of the forecasting enterprise is already compromised⁹. These concerns were pointed out as major shortcomings in the Dorman report:

The danger signs at EMC include lack of effective prioritization processes, insufficient base funding which compounds the work allocation problem, and an insular concept of operations that fails to adequately access the national talent and resource base (Dorman, 1999).

NCEP management clearly recognizes the value of the Dorman report recommendations, and they indicated to BASC that many of the criticisms are being addressed. For example, EMC activities have been prioritized to permit a rational framework for resource allocation decisions (Lord, 1999). EMC also is now a major player, with other institutions, in the development of the next generation of mesoscale forecast models (Lord, 1999). The fact that the base funding for EMC still covers less than half of the EMC staff, however, creates extreme stress on the system. Soft money funding may be appropriate for some research activities, but not for operational organizations. The increasing difficulty in continuing to collect the soft money necessary to operate the center has been a major concern for EMC leadership. The lack of stable base funding makes it difficult to plan the ongoing operations. At the same time, the organizations that serve as the sources for the soft money may have interests that differ from the principle mission of EMC and may place conditions on the receipt of the funding resulting in tensions in priority setting. The solution to this problem may well lie at levels in NOAA considerably above EMC or NCEP.

Advances in computer power play an important role in NWP accuracy increases as noted in Figure 3.1. Computer power, as measured by

millions of floating point operations per second (MFLOPS), is depicted in Figure 3.2 for NCEP (Lord, 1999) and the European Center for Medium Range Forecasting (ECMWF) (ECMWF, 1999). Note that the scale for computing power is logarithmic. Although both centers have benefited from a rapid increase in computer power, the disparity between EMC and ECMWF increased during the 1990s. One important consequence of the disparity is that, relative to ECMWF, EMC lacks adequate resources to conduct experiments to test and evaluate model improvements and work effectively with external users to build a more effective analysis and forecast system. Likewise, these computational limitations hinder EMC's ability to develop new products and services (NRC, 1999c). However, improvements are being made in this situation with the recent acquisition of a new IBM system for NCEP. The planned increase for 2000 and 2001 is currently in the NCEP budget, and this addition should provide EMC with the computer power necessary for improved operations. There is no plan at this time for a system on which to perform parallel testing and demonstrations (Lord, 1999)¹⁰.

The lack of resources affects EMC's ability to adequately interface with the user community to evaluate and improve the operational prediction models. The value of cooperation among interested user groups has been demonstrated by the National Center for Atmospheric Research (NCAR) community climate model (CCM) and the Penn State/NCAR mesoscale model, version 5 (MM5). The MM5 model is now used not only at NCAR, but also at 44 U.S. universities, 28 federal and state agencies, 60 foreign organizations, and 28 private sector companies. In the past decade, the CCM user group has grown to over 200 individuals and organizations. These community models benefit from the number of scientists that are using, evaluating, and making suggestions for improving them. EMC has not used the MM5 for operations, but instead developed an independent model, the ETA 11.

Although the ETA model has some advantages, it is not widely used in the community. EMC needs to benefit more from the active involvement of the community in the design, testing, implementation, and evaluation of new models, something that EMC management has recognized from their ongoing work with the broader meteorological community in generating the next generation weather prediction model (Lord, 1999).

The ability to encourage researchers from other agencies to spend periods of weeks to months at EMC is important in leveraging national investments in research. Such visits have been shown to be a significant component of the success enjoyed by ECMWF. Discussions with EMC personnel indicated that, currently, these visits are seriously hindered by the lack of adequate EMC facilities. Space is so limited that there is no room to accommodate visitors, and adequate temporary lodging is not readily available in the vicinity.

Mechanisms exist to implement obvious improvements in forecasting capability to an operational mode. For example, NSF supported the doctoral dissertation of a student from the University of Oklahoma for several years through the NSF/NCEP Joint Program, yielding an advanced prognostic cloud scheme for the operational ETA model. Post degree, the student became a Visiting Scientist at NCEP/EMC, where he spent two additional years developing and testing his scheme for a wide variety of meteorological situations in the operational environment. The scheme was implemented operationally, resulting in significant improvement in the operational prediction of precipitation. This outcome was the result of NSF's support to enable the interaction and EMC's support for testing and implementation.

Such cases, however, appear to be relatively unusual rather than part of a systematic process. Several recent specific examples involving the transition from research to operations illustrate the problems associated with this inadequacy. Each case was discussed by the EMC representatives at the summer study.

• As part of their ongoing research program, scientists at NOAA's Geophysical Fluid Dynamics Laboratory (GFDL) developed an advanced model for hurricane prediction. NWS agreed that EMC would implement the GFDL model after a planned supercomputer upgrade (it was too computationally expensive for the computer then in use). The

transfer to operations took place over three years. While testing using the diverse examples characteristic of operational use revealed some areas of weakness, the new model resulted in a major improvement of operational hurricane forecasts with the largest reduction in track prediction error obtained in many years. However, this transition is far from complete. The GFDL research staff is still occasionally maintaining the program and is now working on converting the model code to run on a new computer system. Even though it is a vital part of the operational run-stream, there is a lack of sufficient programmer staff at EMC to effect fully the code transition.

• Over many years, NOAA's Forecast Systems Laboratory (FSL) developed a system designed to produce a quick, accurate analysis and short range forecast over the continental United States. Following an agreement between FSL and EMC, this model was implemented operationally at EMC, a major effort requiring about two years of joint development and operational testing. The model provides a rapid (1 to 3 hours) update of the atmospheric evolution essential for severe weather forecasting. While the outcome of this transition was a success, the process was not expedient because of the difficulty in incorporating capabilities developed outside of EMC. A formal management agreement and considerable time were required to ensure cooperation.

• In December 1992, EMC and ECMWF were the first centers to implement operational ensemble predictions which provide useful guidance to the forecasters about forecast reliability. When the ensemble members are very different from each other, the forecast is very uncertain. This observation led to the development of the adaptive observation concept in which additional observations are taken to reduce uncertainty. Two sets of research experiments, FASTEX and NORPEX, demonstrated both the moderate cost and the success of this approach in improving severe storm forecasts. Despite the success, the transition to operations has not occurred because funding necessary for aircraft operations to implement the adaptive observational process has not materialized.

• Almost $1 billion has been invested in a national Doppler weather radar network as part of the modernization of the weather service. At the heart of the network is the NEXRAD radar that provides critical information needed for severe weather warnings issued by the NWS. Although the radars have been in existence for nine years, the detailed data from the radars, which include radar reflectivity, precipitation rate and characterization (e.g., rain/hail/snow), and wind velocities,

have not been incorporated into data assimilation models routinely used at EMC for operational numerical weather prediction. The lack of new resources and higher priority demands on available staff time at EMC have created an impediment to the development of an assimilation algorithm that would enable enhanced predictions and have prevented researchers from evaluating the inclusion of the radar data (NRC, 1999d).

• Accurate observations and model output are central to both the research and operations enterprise and critical for advances in research and improvements in operations. There are many examples of this, including use of model output as observational surrogates in those regions where the observational network does not permit detailed analysis of weather, use of gridded surface meteorology over the ocean in deploying instrumented buoys and ships, and use of atmospheric variability measures to assess changes in the temperature, salinity, and velocity structure of the upper ocean. The NWP model output permits exploration of new, improved methodologies and applications by the research community. However, resource limitations have prevented these data from being made available routinely to the research community. The lack becomes a major obstacle in linking operations and research.

• The observations that are currently used in weather forecasting are taken from airborne, ground-based, and satellite platforms. In general, planning for each set of observations or platform is done independently. This design inadequacy sometimes results from difficulties in assembling a national network of observational instrumentation across many federal agencies. Some progress has been made in developing a national plan for an integrated observing system for weather observations (NRC, 1998a); however, with respect to climate, a national integrated observing network is not in place (NRC, 1999a). With the anticipated suite of future forecast products, the need for a national plan for the required observation network is critical. Such a plan would require consideration of the necessary hardware, maintenance and operation, and assessment of methods to assimilate the measurements into the models. EMC is beginning to address a portion of this requirement through computer-based studies, but again, the lack of sufficient computer resources for parallel testing of data assimilation scenarios and measurement and sampling strategies is slowing the efforts.

• For the last two decades, EMC's computational resources have been a small fraction of the capability at other international facilities

such as the ECMWF (Figure 3.2). This lack of resources has had a demonstrated negative impact. For example, specific cases have occurred in which upgrades to the U.S. forecasting capability could not be executed simultaneously with the existing operational forecast codes in order to provide careful assessment, debugging, and systematic changes in the newly developed code. As a consequence, at the appearance of a problem, new model versions have been replaced by older model versions to prevent an interruption in service. In contrast, ECMWF is capable of simultaneous execution of test and operational codes.

• Computational power is not the only computational issue. EMC purchased an alternative architecture based on parallel processing that computer experts predict will be the architecture of choice in the future (a Department of Commerce ruling prevents the sale of advanced vector processing capability from foreign suppliers even though U.S. suppliers are not competitive in terms of performance [NRC, 1998b, 1998c]). By “adapting to the future early” it is hoped that while the U.S. capability may lag for a period, it will then leapfrog current vector machine capability. However, the EMC transition to this architecture occurred before the software and model programming expertise required to ensure efficient utilization was available. Consequently, the transition to a new architecture is likely to result in significant stress to an understaffed facility. This issue will affect many U.S. laboratories in addition to EMC, arguing for community model infrastructure as well as programming and software alliances (Reed et al., 1999).

These examples were discussed at the summer study and contribute to the following assessment.

BASC's assessment of the strengths and weaknesses in the current transition from research to operations follows from the criteria described in Chapter 2:

1. BASC considers that the overall atmospheric and oceanic research community in the United States is strong (NRC, 1998a). Improvements in weather and short-term climate forecasting will involve

leveraging this research strength to expand the quality and diversity of services provided to the nation.

2. Some major research programs are taking steps to link research activities and forecast products to society's needs. For example, the national and regional assessment programs under the USGCRP are addressing the societal impacts of climate variability and the resulting potential environmental changes (USGCRP, 1999).

3. BASC could find no evidence of a comprehensive effort to address the societal benefits and economic impacts of various research and transition strategies. For instance, while the social science community dealing with societal benefits and economic impacts is becoming more fully engaged in the USWRP, few resources are being targeted for this activity. There is also little USWRP funding that promotes transition to operational products (NRC, 2000b).

1. EMC does not have the ability to test and evaluate new algorithms and forecast capabilities and obtain feedback from the user community. Insufficient computational resources prevent simultaneous testing of new model versions while operational forecasts proceed. Computational capability that allows simultaneous testing and operational execution of model codes does not exist. The ability to address the special needs associated with assimilation of a large volume of new satellite observations is limited.

2. Several aspects of the U.S. weather observational capability have improved considerably in the past decade. The NEXRAD weather radar system, the new Geostationary Operational Environmental Satellite (GOES), and the increased number and quality of moored and drifting ocean buoys all add to the potential for significant improvements to the forecasting enterprise.

3. Human resources at EMC receive only 47 percent of their funding from the base budget; the remainder is supported by soft money. Although EMC has been successful despite this limited base funding, BASC is concerned that the level of base funding is too low for an operational agency.

4. When new sensors are developed, the budget to develop the algorithms and tests to introduce those sensors into the operational system is frequently inadequate.

5. EMC is located in a substandard and unsafe facility. The building is not suitable for hosting quality technical activities with modern computer and communications requirements. Muggings in the parking lot and bullet holes in the windows create an environment that is not conducive to the recruitment of scientific talent on a short-or long-term basis.

6. EMC is not co-located with any other forecasting enterprise, data assimilation facility, or research and/or operational team. It is geographically isolated from the supporting NOAA laboratory structure, such as GFDL and FSL, and from the other national capabilities such as NCAR and NASA Goddard. While NCEP recognizes the advantages of co-location with complementary research facilities and has developed plans for co-location with NASA Goddard, these plans have never materialized.

7. EMC lacks a clear delineation of responsibility for the transfer of research results to operations. No one at the EMC or the weather-related NOAA labs is specifically assigned the responsibility for this linkage.

8. The NWS model development process is becoming more open to the research community. EMC is participating with other organizations to develop a community mesoscale prediction model that will enable greater participation by a large number of researchers in the process of incremental improvements in the model performance. With its present computer and human resources, EMC cannot rerun a forecast situation to evaluate the cause of major forecast errors in an attempt to improve the performance.

9. EMC does not have a capable and accessible data and model archiving system that would enable the research community to address ways of improving EMC's forecasting models.

1. Limitations on resources have forced the elimination of routine forecast verifications at EMC (Lord, 1999), adding to the difficulties in making improvements and limiting the information on the expected improvements in forecast capability.

2. The lack of greater understanding of societal benefits and economic impacts of forecast products limits effective decisions in resource allocations.

The international data exchange experience tends to be mixed. Some international programs, such as TOPEX/POSEIDON and TRMM, are quite effective in exchanging data. However, commercialization issues within the national meteorological services are increasingly hampering the exchange of more conventional meteorological data, resulting in a reduction of the amount of data archived in the World Data Centers (NRC, 1998a). If it continues, this trend could have a considerable detrimental effect on both research and forecasting operations.

There is limited capability at EMC for continuous evaluation of operational effectiveness. The NCEP director acknowledged at the BASC summer study that forecast products are no longer routinely verified because of personnel and resource limitations.

The United States has a strong atmospheric and oceanic research community. However, resources designed to enhance the delivery of products useful for society have not fully supported this investment. There is an impressive opportunity to leverage a small investment into great accomplishments if key issues and problems that limit the transfer of knowledge and capability are addressed. Improved leveraging of this investment is essential to providing services and to expanding the forecasting products provided to the nation. The sense of the recommendations that follow is very similar to that of Recommendation 4 from the ‘Road Map report' of the National Weather Service Modernization Committee; however, they focus on NCEP and EMC rather than the more general NWS-wide nature of the Road Map recommendation (NRC, 1999c).

The recommendations are divided into three elements: (1) improvements needed to ensure that the United States avoid degradation or failure of current capabilities, (2) improvements needed to ensure that the nation can address future product demands and needs, and (3) management changes that have little associated costs, but will facilitate needed

improvements. Alternate mission definitions or organizations were not considered by BASC.

1. A credible development, testing, and integration facility within EMC with a strong connection to the research community. The inability to evaluate and test new algorithms while maintaining current operational forecasts is a major limitation (Emanuel and Kalnay, 1996).

The facility personnel would test, within a quasi-operational environment, ideas that have already been developed and shown to be promising in an academic environment. They would collaborate with outside researchers on tasks such as routine forecast verification, code conversion, and optimization. Such tasks cannot be handled by an individual researcher and constitute major barriers for the transition from research to operational implementation. Adequate computer resources should be provided to support this testing and to test new algorithms needed for massively parallel computers. The test facility could also facilitate the archiving of operational products in real time, make them available to researchers, and transition successful experimental projects such as Reanalysis, Seasonal to Interannual Prediction, and the GEWEX/GCIP Land Data Assimilation System into regular operational execution.

Such a test facility would require proper understanding of the user needs, as well as technological capabilities including communications expertise. A model archiving system should be closely connected to this model testing facility. From the initial creation of NCEP, the development and implementation of atmospheric (and eventually ocean) models and analysis systems has been conducted primarily within the organization, at first using civil servants, and later supplemented by contractors and visitors. The pace of progress is slow in meeting the requirements of the National Weather Service. The complexity of model and analysis systems science and technology has outgrown the single-site development methodology that has been in place for the past 40 years. No single center anywhere in the world can afford the staff necessary to permit internal development of all the upgrades required to continue improving models and analysis systems at the required pace. Even ECMWF relies on the meteorological research capabilities of its member states for partnership activities (ECMWF, 1999). Resources should be identified that ensure active collaboration and participation. Methods that will effectively open the NWS model development process to the research community outside EMC are required. Improved visitor programs, graduate fellowship programs, and funded collaborative efforts using designated funds are essential.

Currently, EMC modeling plans are presented and reviewed annually by the NWS users (the NCEP Service Centers and representatives of the forecasters in the NWS regions). This is a very comprehensive procedure that has led to better communications and faster response to forecasters' needs. NCEP should expand this review to include external users as well.

NOAA should provide the computational resources on the development side to simultaneously perform test simulations, reprocessing, and ongoing operational forecasts. Expertise in parallel processing and efficient code development should be a part of the facility. Collaborative partnerships to address the programming challenges of new computer architectures will be essential to the success of the enterprise and should be developed. Necessary staff should be added to enable archiving of data and model products and to interface with researchers. Both space and sufficient workstations should be available to enhance research collaboration. The capability to increase the number of long-term visitors (5 to 10 senior researchers, young investigators, and graduate fellows) is required to open the NWS process to the research community.

Staff should be budgeted to assist and support visiting researchers. Designated funds are needed to support mission-related research projects.

2. Staffing requirements.

Many of the issues and bottlenecks involve a lack of staff at EMC. The NWS and NCEP cannot even sustain their present level of development activity, and this level is viewed by BASC as inadequate. It is interesting to note that other agencies are willing to provide funds to EMC for certain activities, demonstrating the importance of NCEP for NOAA and other federal agencies. Yet this requirement for extensive outside funding (soft money) also clearly defines the nature of the human resource problems in an operational enterprise, an issue which will be exacerbated with any expansion of the forecast products.

3. Co-locate forecasting capabilities.

Operational predictions of weather, water, and climate should be integrated in order to make optimal forecasts to serve the nation. The NWS has already recognized that co-location of forecasting centers with universities and research centers has led to faster progress through synergy. The benefits of co-location or formal interaction are clear. For example, the institutional and physical separation of EMC, which devel-

ops operational atmospheric and ocean forecast models, and the NWS Office of Hydrology, which develops operational river models for the River Forecasting Centers, has hampered this integration. Similarly, the NWS Techniques Development Laboratory develops user products through techniques such as Model Output Statistics, which improve the utility of the forecasts by correcting error bias and relating model output variables to variables of user interest such as visibility. The integration of the atmospheric, water, and climate forecasting functions is important to allow for improvements in the usefulness of the forecasts.

Several options for co-location are available, including co-location with NASA Goddard Space Flight Center, with universities with strong atmospheric science capabilities, or with other national laboratories. If resources offered by NASA and/or universities are leveraged, this could result in net savings to the government over a period of 5 to 10 years. Further, prompt action should be taken to improve the safety and adequacy of the EMC facilities. The current location is viewed as unacceptable for attracting staff or visitors or for expansion of capability. NSF, NASA, and NOAA should support visits by university and other researchers to EMC under improved site conditions to accelerate the transition from research to operations, as is so successfully done in Europe.

4. Continued modernization and improvement.

NOAA's National Weather Service (NWS) began a major modernization program in 1989. Before the modernization, the NOAA weather radars were using vacuum tubes that were no longer manufactured in the United States and replacements were being obtained from the former Soviet Union. $4.5 billion has been spent on this modernization—all new radar equipment has been installed, all field offices staffed, and the forecasting and warning operations have been modernized throughout the NWS. Given the pace of technological change, however, it will not be long before this current equipment becomes obsolete. As we enter the twenty-first century, it will be impossible to meet tomorrow's needs with yesterday 's technology. An annual budget designed around life cycle costs should enable the forecasting system capabilities to be in place to meet the evolving needs of the nation for weather and climate forecasts.

5. National and international research potential in the atmospheric sciences.

The USWRP provides one mechanism to improve the balance of resources involved with the transfer of research to operations via its joint agency grants program. The program has been justified in terms of transferring research into improved products for the nation, so it is appropriate that USWRP activities are structured to enable this transition (NRC, 2000b). The community prediction model approach, as illustrated by the experience with the MM5 community model, enables the rapid assimilation of research results into the operational systems.

6. National and international research potential in the ocean sciences.

It is important to recognize that the USWRP is not the only user community engaged in research applicable to the NCEP mission. BASC noted how poorly EMC's model verifies over the oceans. Forecast improvement could come from improvement of model performance over the oceans. Some NSF, NOAA, Navy, and NASA funded research and research communities have little to do with USWRP but should be key players and partners with EMC. The air–sea interaction and coupled ocean–atmosphere–land climate communities can contribute research results to the EMC mission.

7. Clear responsibility for transition.

With the important exception of DOD¹² the nation's forecasting enterprise lacks a clear delineation of responsibility for the transfer of research to operations, although there does exist clear responsibility for both research and operations. In NOAA, this transfer responsibility is currently assigned in an ad hoc fashion, a situation that frustrates effective transfer both within NOAA and between other agencies and NOAA. In contrast, DOD designates responsibility for transfer to a particular person in its research and development process. NOAA has in the past utilized a process called the “troika ” to facilitate the transfer of research to operations. This process, while important, is outside the formal institutional lines of authority and budgeting. Transition responsibility requires an end-to-end advisory involvement, from involving operations people in the planning of research agendas and research people in operational planning.