Peer Review in Environmental Technology Development Programs (1998) / Chapter Skim
Currently Skimming:
5 Analysis of OST's Peer Review Program
5 Analysis of OST's Peer Review Program
Pages 67-81
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 67... ...
The analysis is based on the committee's review of OST documents, presentations by DOE staff, observations of recent peer reviews and ASME Peer Review Committee meetings, and the committee's assessment of peer review reports. In each section of this chapter, the committee revisits the recommendations from the interim report and discuss specific changes that OST has implemented since the publication of its interim report to address these issues.
|
From page 68... ...
10~. In response to the committee's interim report, OST recently issued a policy restricting the term peer review to only those technical reviews conducted by independent, external experts (DOE, 1998b)
|
From page 69... ...
, and the total number of projects that met OST's selection criteria. Because of the relatively low rate of peer reviews conducted under this program relative to the large number of active projects, OST currently funds a large number of technologies that never have been peer reviewed.
|
From page 70... ...
Such a practice, however, does not constitute a sufficient and systematic approach for selecting and prioritizing the projects to be reviewed. In its interim report, the committee recommended that OST develop a rigorous process for selecting projects to be peer reviewed.
|
From page 71... ...
the need to reduce funding levels due to budget cuts. The committee believes that the procedure developed for selecting projects for peer review should ensure that OST would peer review any technology being considered for deployment if it has not already been peer reviewed.
|
From page 72... ...
A related problem arises when a technology is developed and deployed without OST involvement. Although application of the current policy of peer reviewing new starts and all projects at Gate 4 will ensure that all technologies developed by OST in the future are appropriately peer reviewed, technologies developed outside DOE and proposed for demonstration by OST would not have been subjected to an OST peer review during development.
|
From page 73... ...
Definition of Peer Review Objectives and Selection of Review Criteria Peer Review Objectives The committee noted in its interim report that a statement of the objective of the peer review should be made available to all participants in a peer review (i.e., PIs, the ASME Peer Review Committee, all reviewers, and observers) prior to the review so that they clearly understand its context.
|
From page 74... ...
13) requires FA/CC program managers, coordinating with the EM-53 Office Manager to submit to the Peer Review Coordinator the objective of the review, technology-specific review criteria, and a list of PIs who will be responsible for providing technical documentation and delivering the technical presentation.
|
From page 75... ...
The committee therefore recommended in its interim report that OST revise these broad criteria to focus on technical aspects of the issues mentioned above, or remove them from the list of review criteria. The committee also recommended that OST develop a well-def~ned general set of technical criteria for peer reviews, to be augmented by technology-specific criteria as needed for particular reviews.
|
From page 76... ...
Although the range of expertise on peer review panels observed by this committee has been acceptable, the committee believes that the databases may not have adequate scope to identify the broad range of reviewers likely to be necessary for some complex projects. In its interim report the committee noted that it would be appropriate for OST to gain access to databases Dom other professional societies or review organizations.
|
From page 77... ...
Planning and Conducting the Peer Review Organization and preparation of each peer review event are key to a successful peer review program. Well in advance of the review, peer reviewers should receive written documentation on the significance of the project and a focused charge that addresses technical review criteria.
|
From page 78... ...
Documentation The documentation required for an OST peer review is presented in Chapter 4. This list has been modified from OST's original list of required documentation in response to the committee's interim report, which pointed out that some of the required documents addressed nontechnical issues.
|
From page 79... ...
Two other improvements in peer review reports have been the addition of a section that summarizes the review criteria and the inclusion of short biographical sketches of the peer reviewers. In its interim report, the committee recommended that the peer review reports could be improved further 3If OST decides to implement a system of peer review to evaluate large numbers of research proposals, however, it may want to consider the pros and cons of anonymous versus open peer reviews for this purpose.
|
From page 80... ...
The committee noted in its interim report that the timeliness and quality of the formal written responses from the FA/CC program managers also have been uneven. None of the formal responses from the program managers for the FY97 reviews were transmitted to ASME within 30 days of the peer review, as required by OST policy.4 In addition, many of the written responses did not document how OST intended to follow through with the conclusions and recommendations of its peer review reports.
|
From page 81... ...
is a strong indication that peer review has not yet become a part of the "culture" of OST, whereby FA/CC program managers would embrace peer review as a valuable tool to help guide their decision process (see discussion in Chapter 79.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.