Peer Review in Environmental Technology Development Programs (1998) / Chapter Skim
Currently Skimming:
Executive Summary
Executive Summary
Pages 1-20
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 1... ...
These projects are chosen for new and continued funding through a complex technology selection process, which uses the results from various types of reviews, including programmatic reviews, technical assessment reviews, and peer reviews. Several National Research Council (NRC)
|
From page 2... ...
In its interim report published in October 1997 (NRC, 1997b) , the committee conducted a preliminary assessment of OST's new peer review program.
|
From page 3... ...
. Improving Technical Quality The independence of peer reviewers makes them more effective than internal reviewers because experts who are newly exposed to a project often can recognize technical strengths and weaknesses, and can suggest ways to improve the project that may have been overlooked by those close to it (Bozeman, 1993~.
|
From page 4... ...
Because peer reviews are by definition technical in nature, both the objectives of the review and the review criteria should focus on technical considerations.
|
From page 5... ...
The process for selecting reviewers must consider the fundamental characteristics of peer review and the specific objectives and criteria for the particular review being organized, and should be conducted by a person or group independent of the group being reviewed (Cozzens, 1987; Koning, 1990~. Peer reviewers should be selected in accordance with formally established qualification criteria that include at a rnin~mum the following: relevant demonstrated experience, peer recognition, knowledge of the state of the art of the subject matter under review, absence of a real or perceived conflict of interest, and bias2 such that the panel as a whole is balanced.
|
From page 6... ...
In particular, OST has revised its review criteria to focus on technical issues, has developed a more systematic approach for selecting projects to be reviewed, and has modified its list of required documentation for the peer reviews. OST also has made a number of policy changes since this committee issued its interim report in October 1997 (see Table 1 and the main body of this report for more details on these policy changes)
|
From page 7... ...
The linkage between peer review results and OST's decision-making process also could be improved by explicitly identifying where and how the results of peer reviews will be used, before the review is conducted. Therefore, the committee recommends that as part of the documentation provided to peer review program management during the process of selecting projects for review, OST program managers be required to clearly identify the upcoming decision or milestone for which the results of the peer review will be used.
|
From page 8... ...
OST's approach addresses two primary issues identified by the committee In its interim report: the need to focus short-term peer review efforts on high-budget, late-stage projects that have never been peer reviewed, and the need to review all proposals for new technologies as they enter the project development cycle. Although OST's three selection criteria are reasonable and should help OST choose projects to be reviewed, however, they do not explicitly address two issues that have been emphasized recently by OST management: (a)
|
From page 9... ...
. The committee finds that these revised general criteria and the procedure for developing technology-specific criteria are a meaningful improvement over the original review criteria because they allow OST's peer review program staff to focus the reviews on important technical issues.
|
From page 10... ...
Documentation for Peer Reviews The documentation required for an OST peer review is listed in Chapter 4. This list identifies some of the documents required to evaluate the technical merit of a project and, if implemented, should improve He quality of background materials provided to peer review panels.
|
From page 11... ...
OST's current practice, in which nearly all peer reviews include formal presentations by the project team, followed by deliberations by the panel, and further questionand-answer sessions over the course of two to three days, places significant limits on the number of projects that can be peer reviewed. Even if the number of projects that were peer reviewed during a single review could be increased by improved efficiency, OST's backlog of technologies that have never been peer reviewed still would take years to address through the current process.
|
From page 12... ...
The prescreening evaluations should not be used as the sole means for providing technical input into decisions to fund high-budget, environmental remediation projects, however. Peer reviews involving presentations by the project team and question-andanswer sessions should be carried out for all projects involving significant capital investment by OST.
|
From page 13... ...
Although the committee discusses these issues at some length in this report, additional peer reviews should not be added to the large number of reviews currently used within OST without first evaluating the objectives and effectiveness of existing reviews. The committee recommends that OST carefully evaluate the objectives and roles of all of its existing reviews, and then determine the most effective use of peer reviews (of various types)
|
From page 14... ...
summaries of key performance metrics that demonstrate how peer reviews are used to meet the overall objectives of OST's program.
|
From page 18... ...
18 at .8 o c' _' m ¢ ~0 c L:4 U3 o o o Ct C)
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.