National Academies Press: OpenBook
« Previous: Executive Summary
Suggested Citation:"1 Introduction." National Research Council. 1998. Peer Review in Environmental Technology Development Programs. Washington, DC: The National Academies Press. doi: 10.17226/6408.
×

1
Introduction

The U.S. Department of Energy's (DOE's) Environmental Management (EM) Program was established in 1989 to address the health, safety, environmental, and regulatory challenges associated with cleanup of the nation's nuclear weapons complex. Within EM, the Office of Science and Technology (OST) was created to promote the development of improved technologies to lower cleanup costs and risks (to workers, the public, and the environment) and to improve cleanup capabilities. OST supports the entire range of technology development activities—beginning with basic research through the EM Science Program (NRC, 1997a) and extending through development, demonstration, and (with the assistance of industrial partners) deployment into the cleanup program.

In fiscal year 1998 the annual budget for EM is about $5.6 billion, of which about $220 million is devoted to technology development activities within OST. The importance of technology development to EM's mission has been recognized in Accelerating Cleanup: Paths to Closure (DOE, 1998a), which describes EM's plan for cleanup of the weapons complex. This plan discusses the importance of technology development to reduce the "mortgage" at the complex—the long-term costs of maintaining contaminated buildings, equipment, and sites until they can be remediated.

OST sponsors 226 active research and development projects at universities, national laboratories, and private-sector companies on topics ranging from the remote detection of contaminants in the subsurface using geophysical techniques to the development of melters for waste vitrification. The major research and development program units within OST are listed in Table 1.1. OST uses various types of reviews (e.g., programmatic reviews, technical assessment reviews, peer reviews1) in its technology selection process

1  

These peer reviews are termed "technical peer reviews" by OST.

Suggested Citation:"1 Introduction." National Research Council. 1998. Peer Review in Environmental Technology Development Programs. Washington, DC: The National Academies Press. doi: 10.17226/6408.
×

(see Appendix B for a description of the different types of reviews). These reviews are used to assess the merit of individual projects as well as the merit of all technology development thrusts within the office.

TABLE 1.1: DOE-OST R&D Program Units

Office of Technology Systemsa

Office of Science and Risk Policy

Decontamination and decommissioning focus area

Environmental management science program (EMSP)

Mixed waste characterization focus area

Risk policy program

Radioactive tank waste focus area

 

Subsurface contamination focus area

 

Characterization, monitoring, and sensors cross cut

 

Efficient separations program cross cut

 

Robotics cross cut

 

Industry and university programs

 

Technology integration

 

a OST's focus areas, cross cuts, and supporting technology areas are administrative units used by OST to manage and coordinate its technology development activities, and are based on DOE-EM's major problems.

OST's Peer Review Program

Several recent National Research Council (NRC) reports evaluated OST's technology selection process and recommended that OST develop an independent, external review process and apply it to all technology development programs. The report Improving the Environment recommended that "technology selection should incorporate a Knowledgeable independent review group that has no vested interests in the outcome and that includes people from outside the Department who work in the commercial use of technologies" (NRC, 1995c, p. 104). The NRC's Committee on Environmental Management Technologies (CEMT) also evaluated this technology selection process in its 1994 and 1995 annual reports (NRC, 1995b, 1996). In particular, these reports

Suggested Citation:"1 Introduction." National Research Council. 1998. Peer Review in Environmental Technology Development Programs. Washington, DC: The National Academies Press. doi: 10.17226/6408.
×

recommended that OST develop a standardized, rigorous, and independent external peer review process and apply it to all technology development programs. These findings were echoed in a subsequent General Accounting Office (GAO) report, which concluded that "although the lead sites used significantly different systems to select projects, none of them used disinterested reviewers to determine the technical merit of the proposed work" (GAO, 1996, p. 7).

In response to these NRC and GAO reports, OST recently instituted a peer review program that uses the American Society of Mechanical Engineers (ASME), with administrative and technical support provided by the Institute for Regulatory Science (RSI), to conduct peer reviews of technologies (or groups of technologies) at various stages of development. According to the OST, the objective of this new program is to serve as a "management tool for assuring that the technology is of high quality and effective, that critical needs have not been overlooked, and that the technology has the best chance possible for implementation."2

Charge to the Committee

OST asked the NRC to convene an expert committee to evaluate the effectiveness of its new peer review program and to make specific recommendations to improve the program, if appropriate. In particular, the committee was asked to review the following:

  • internal procedures used by OST to identify the need for timely peer review of projects and programs;
  • structures, protocols, and procedures for obtaining peer reviews of OST projects and programs, including who decides what will be peer reviewed, what criteria for peer review are used, and when in the R&D process peer review is requested; and
  • feedback of peer review results into program management and development decisions.

In performing this assessment the committee was asked to compare OST's practices to generally accepted norms for scientific and technical peer review, including practices for selection of peer reviewers and screening for bias and conflict of interest. The assessment has been made more challenging by the

2  

Presentation to committee by Anibal Taboas, DOE, Washington, D.C., February 24, 1997.

Suggested Citation:"1 Introduction." National Research Council. 1998. Peer Review in Environmental Technology Development Programs. Washington, DC: The National Academies Press. doi: 10.17226/6408.
×

fact that OST has continued to improve its peer review process throughout this study, based on input from the committee's interim report advice from the ASME Peer Review Committee, and OST's peer review program staff.

OST has chosen to establish a relationship with two outside organizations, ASME and RSI, to carry out parts of its peer review process. In performing this assessment, the committee has focused on the structures, protocols, and procedures of OST's peer review program, including those parts carried out by ASME and RSI, but has avoided an explicit evaluation of the performance of these organizations because they are acting simply as agents of OST. The responsibility for establishing and executing an effective peer review program lies entirely with OST. The committee also has not evaluated, or endorsed, the organizational arrangements among ASME, RSI, and OST. The committee's findings and recommendations are directed at OST, which as noted above, has ultimate responsibility for all aspects of the peer review program.

Parallel Efforts

Another NRC Committee, the Committee on Prioritization and Decision Making in the DOE-OST, is currently conducting a parallel evaluation of the decision-making processes throughout OST's technology development program. One aspect of its work will be to examine the role and importance of peer reviews (and other types of reviews) in OST's decision-making processes. Our committee therefore has focused its work on OST's peer review program itself, including an evaluation of how peer reviews, if conducted, could be made more useful as an input to OST's decision-making processes, but has not evaluated OST's decision-making processes explicitly.

Study Process

The committee was briefed on the newly instituted peer review program by DOE staff at six committee meetings and has reviewed the recently developed Implementation Guidance for the Office of Science and Technology Technical Peer Review Process (DOE, 1998b), which documents OST's revised peer review procedures. Members and staff of the committee also observed peer reviews conducted on the In Situ Redox Manipulation project in Richland, Washington; the Decontamination and Decommissioning (D&D) Large Scale Demonstration projects and three D&D Technology Development projects in

Suggested Citation:"1 Introduction." National Research Council. 1998. Peer Review in Environmental Technology Development Programs. Washington, DC: The National Academies Press. doi: 10.17226/6408.
×

Morgantown, West Virginia; the MAG*SEP technique3 in Atlanta, Georgia; the Small In-Tank Processing Modules and Small Modular In-Can Vitrification projects in Columbia, Maryland; several High Temperature Melter and Characterization projects in Idaho Falls, Idaho; and the Vortec Combustion Melter, Catalytic Chemical Oxidation, and Steam Reforming of Low-Level Mixed Waste projects in Columbia, Maryland. Committee members and staff also attended the Annual Meeting of the ASME Peer Review Committee in November 1997 and the ASME Peer Review Committee meeting in January 1998. In addition, the committee reviewed all peer review reports produced under the program from its initiation in October 1996 through April 1998.

In its work, the committee reviewed the literature on peer review and placed an emphasis on comparison with peer review in other organizations—an informal type of benchmarking. To aid in the comparison, during its meetings the committee was briefed on current peer review practices at the Department of Defense (DOD) Strategic Environmental Research and Development Program (SERDP), the National Institutes of Health (NIH), the National Science Foundation's (NSF's) Small Business Innovation Research (SBIR) program, the National Institute of Standards and Technology (NIST), the Environmental Protection Agency (EPA), EPA's Science Advisory Board, the U.S. Nuclear Regulatory Commission (USNRC), DOE's Office of Basic Energy Sciences (BES), Information Dynamics, Inc. (which conducts reviews for the National Aeronautics and Space Administration's [NASA's] Office of Life and Microgravity Sciences and Applications [OLMSA]), and the American Institute of Biological Sciences (AIBS). The committee also drew on its members' personal knowledge of these and other peer review programs.

Reports

This is the second of two reports prepared by the committee. In October 1997, the committee completed its interim report, which provided OST with a preliminary assessment of its new peer review program (NRC, 1997b). The interim report described the essential components of a credible peer review process and provided a preliminary assessment of OST's peer review program and the status of its implementation. In the interim report, the committee examined broad issues and tried to offer constructive recommendations to assist OST in successfully implementing this program, focusing on how OST could most effectively establish the basic elements of a sound peer review process.

3  

The MAG*SEP technique is a means of recovering selected radionuclides and heavy metals from water and other liquids through sorption onto specially coated particles.

Suggested Citation:"1 Introduction." National Research Council. 1998. Peer Review in Environmental Technology Development Programs. Washington, DC: The National Academies Press. doi: 10.17226/6408.
×

OST has moved expeditiously to implement many of the recommendations from the interim report.

In this final report, the committee first reviews the current status of OST's peer review program (as of April 1998) by reevaluating the specific policy changes that have been implemented by OST in response to issues raised in the committee's interim report. The committee also addresses three specific issues that OST has identified as particularly important at this time:

  1. How should OST deal with the large number of technology projects at late stages of development that have never been peer reviewed? (See Chapter 6.)
  2. Can the fundamental principles of peer review outlined in the committee's interim report be applied to other types of OST reviews? (See Chapters 3 and 7.)
  3. How can OST measure the effectiveness of its peer review program? (See Chapter 7.)

In addition, in Chapter 7 the committee discusses a number of approaches that OST could use to continue to improve the peer review program in the future.

Suggested Citation:"1 Introduction." National Research Council. 1998. Peer Review in Environmental Technology Development Programs. Washington, DC: The National Academies Press. doi: 10.17226/6408.
×
Page 21
Suggested Citation:"1 Introduction." National Research Council. 1998. Peer Review in Environmental Technology Development Programs. Washington, DC: The National Academies Press. doi: 10.17226/6408.
×
Page 22
Suggested Citation:"1 Introduction." National Research Council. 1998. Peer Review in Environmental Technology Development Programs. Washington, DC: The National Academies Press. doi: 10.17226/6408.
×
Page 23
Suggested Citation:"1 Introduction." National Research Council. 1998. Peer Review in Environmental Technology Development Programs. Washington, DC: The National Academies Press. doi: 10.17226/6408.
×
Page 24
Suggested Citation:"1 Introduction." National Research Council. 1998. Peer Review in Environmental Technology Development Programs. Washington, DC: The National Academies Press. doi: 10.17226/6408.
×
Page 25
Suggested Citation:"1 Introduction." National Research Council. 1998. Peer Review in Environmental Technology Development Programs. Washington, DC: The National Academies Press. doi: 10.17226/6408.
×
Page 26
Next: 2 Elements of a Credible Peer Review Program »
Peer Review in Environmental Technology Development Programs Get This Book
×
Buy Paperback | $44.00
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF
  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!