Strengthening Science at the U.S. Environmental Protection Agency Research-Management and Peer-Review Practices (2000) / Chapter Skim
Currently Skimming:
Peer-Review Practices at EPA
Peer-Review Practices at EPA
Pages 99-124
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 99... ...
When scientific and technical information is used as part of the basis for a public-policy decision, peer review can substantially enhance not only the quality but also the credibility of the scientific or technical basis for the decision. After-the-fact criticisms of the science are more difficult to sustain if it can be shown to have been properly and independently peer reviewed.
|
From page 100... ...
The value of the peer-review process in assessing and improving a scientific or technical work product depends on a strong commitment to conduct and apply the results of peer review appropriately in judging or improving the technical merit of the product. The benefits of peer review are diminished if the integrity of the peer-review process is compromised or if the criticisms and suggestions received from independent peer reviewers are to some degree ignored or taken lightly by decision-makers who may be more interested in meeting a deadline or producing a desired answer than in judging or enhancing technical merit.
|
From page 101... ...
Although peer review can be a valuable tool for improving a work product, it cannot be relied upon to ensure excellence in a product that is seriously lacking in technical merit when it enters peer review. Peer reviewers are human and therefore can occasionally be narrow, parochial, biased, over-committed, or mistaken.
|
From page 102... ...
However, in some other parts of EPA, scientific activities and the use of peer review have not traditionally been prominent, and over the years, EPA has been criticized many times for having a poor scientific basis for many of its regulatory decisions (Powell 1999~. In 1991, EPA Administrator William Reilly requested a pane]
|
From page 103... ...
He also noted that agency managers should maintain sufficient discretion to accommodate their program priorities and operating constraints, and he acknowledged a tension between peer review and the control of agency actions. He directed that major scientific and technical work products related to agency decisions should normally be peer reviewed, but that agency managers would continue to be accountable for decisions about when and how to utilize peer review.
|
From page 104... ...
to external peer reviews by such groups as the EPA SAB or FIFRA Science Advisory Panel. While stating that major scientific and technical work products related to agency decisions "normally should be peer-reviewed," the policy statement delegated to individual managers in the agency's headquarters offices, regions, laboratories, and field components the responsibility and accountability for deciding in individual circumstances whether to use peer review, and if so, deciding its "character, scope, and timing." It cautioned that formal peer review should not be conducted in a manner that caused EPA to miss statutory or court deadlines.
|
From page 105... ...
The GAO observed that EPA's peer-review oversight mechanisms essentially consisted of a two-part reporting scheme in which each office and region annually self-nom~nated products for peer review and updated the status of previously nominated products. The GAO argued that agency managers were being given too much leeway to avoid conducting peer reviews without adequate, documented justification.
|
From page 106... ...
to replace the standard operating procedures that had been developed by individual offices and regions in response to the administrator's June 1994 peer-review policy statement (Browner 1994~. The new handbook, developed in an agency-wide effort under the leadership of EPA's Science Policy Council, was designed to provide uniform implementation guidance to managers and staff in the agency's offices, regions, and laboratories for peer review of the 2,000 major scientific and technical
|
From page 107... ...
In questionand-answer format with flowcharts and checklists, the new handbook provides guidance on basic principles and definitions, including distinctions between peer review and peer input, public comment, and stakeholder involvement; planning and preparing for peer review, including the identification of "major scientific and technical" work products, appropriate peer-review mechanisms, and qualified experts; and conducting peer reviews, including materials required, recordkeeping, and the utilization of peer review comments. It specified three categories of annual reporting from each EPA office and region: (a)
|
From page 108... ...
The peer-review handbook provides detailed guidance on deciding what documents should be peer reviewed (Figure 3-~. It specifies that peer review should be conducted on scientific and technical work products that support a research agenda, regulatory program, policy position, or other agency position or action.
|
From page 109... ...
\/ YES '''~ l Y~8 Work Product 1e Not ~ Car~dida" tor f'.er R - lew T Pl8C. Wo~ Product on UstC FIGURE 3-1 Flow chart for pIanning a peer review.
|
From page 110... ...
In determining whether a scientific or technical work product warrants peer review, section 2.2.3 of the handbook allows case-by-case decisions to be made by agency officials but identifies several criteria for such judgments. These criteria include work products that significantly establish or depart from a precedent, model, or methodology; address novel, controversial, or emerging issues; have cross-agency or interagency implications; involve substantial resources; take innovative approaches; or satisfy statutory or other legal mandates for peer review.
|
From page 111... ...
The RSAC cautioned that decisions to review or not to review a product are not always being documented consistently, and it stressed the importance of transparency in EPA's process of deciding the subjects and mechanisms of peer review. The RSAC also recommended that the agency expand its peer-review practices beyond the "major scientific and technical work products" specified in the 1994 peer-review policy statement and defined in the peer-review handbook (EPASAB 1999~.
|
From page 112... ...
Nevertheless, the other EPA offices have often failed to submit planning documents to peer review, and this NRC committee believes that greater emphasis in the agency's peer-review handbook on the categories of documents identified by the SAB could help improve the scientific and technical basis for agency actions. FORMS AND MECHANISMS OF PEER REVIEW Although Administrator Browner's 1994 peer-review policy statement directed that major scientific and technical work products related to agency decisions "normally should be peer-reviewed," it delegated to individual managers in EPA headquarters offices, regions, laboratories, and field components the responsibility and accountability for deciding in individual cases whether to use peer review, and if so, de
|
From page 113... ...
The handbook emphasizes that the greatest credibility is provided when peer reviewers are external to the agency and the peer-review process is formal. However, it acknowledges that peer reviews at EPA might take many forms and allows substantial flexibility in determining the forms and mechanisms of peer reviews, depending on the importance and complexity of a work product; the relevant statutory and judicial deadlines and other requirements; the financial resources; and the office-specific policies and practices.
|
From page 114... ...
114 Sfrengfhening Science of ERA Work Product (Ready for Peer Review) l 1 Develop the Charge Select the Peer Review Mechanism | Set the l Tlmelines, including Deadlines Select Peer Reviewers , ~ , Send Materials to Peer Reviewers & Conduct Review 1 Complete Peer Review and obtain Comments of Peer Reviewers + Continue with Completing a Peer Review Add to Peer Review Record FIGURE 3-2 Flow chart for conducting a peer review.
|
From page 115... ...
For scientific and technical work products supporting major rules, including rules determined to be "significant" by the Office of Management and Budget uncler Executive Order 12866, the handbook emphasizes that external peer review is the procedure of first choice, and any decision to use internal peer review for such work products, although acceptable in some circumstances, should be the exception rather than the rule. While generally praising EPA's 1998 peer-review handbook and program, the RSAC of the SAB recently cautioned that the agency needs to ensure that the process does not become "inappropriately bureaucratic" (EPASAB 1999~.
|
From page 116... ...
The handbook also emphasizes that peer reviewers should be independent—not associated with the development of the work product, either through substantial contribution to its development or through significant consultation during its development. Section 1.4.S of the 1998 handbook specifies that peer reviewers are expected to perform their role with objectivity, as free as possible from institutional or ideological biases or financial conflicts of interest, although it notes that in many cases, some of these requirements might be impossible to meet or might not promote the best possible reviews.
|
From page 117... ...
The limits to this resource can be important. In its site visit to the ORD Extramural Center, members of our committee were informed that the review of STAR research grant applications alone requires the efforts of more than 1,000 peer reviewers per year, and EPA's Science Policy Council has estimated that over 2,000 major scientific and technical work products affecting agency actions currently require such review each year.
|
From page 118... ...
. —r Move the Work Product from lutist 8 to List A 1 Done FIGURE 3-3 Flow chart for completing a peer review.
|
From page 119... ...
The GAO argued that agency managers were being given too much leeway to avoid conducting peer reviews without having to justify the decisions. It recommended that managers be required to catalog all major scientific and technical work products, the plans for review of each, and the reasons why any of them were not to be reviewed.
|
From page 120... ...
The lists contain information on each work product, such as the responsible EPA office or region, peer-review leader, agency decision-maker, review mechanism, review dates or schedule, a summary of the review, and comments on the review process or a rationale for not conducting a peer review. The handbook specifies that the designated peer-review coordinator for each EPA office or region is responsible for organizing an annual review of Al peer review activities in that office or region and providing it to ORD according to annual guidance issued by the EPA deputy administrator.
|
From page 121... ...
The peer-review coordinator is the staff member responsible for monitoring and overseeing all peer-review activities within a given EPA office or region, coordinating peer-review training, mediating difficult issues, ensuring proper record-keeping on peer reviews, and functioning as the office or regional peer-review liaison with ORD and the Science Policy Council. For each peer review (i.e., the review of each work product)
|
From page 122... ...
What Decisio~/Rule/Regulation/Action Does this Work Product Support: 3) Determination of Major Scientific and Technical Work Products O Is the work product scientific or technical Byes _no?
|
From page 123... ...
The SAB plans to continue conducting an in-depth assessment of trends in EPA's uses of peer review, the impacts of the peer reviews, and additional opportunities for enhancing the benefits from peer review in the form of quality, credibility, relevance, and the agency's leadership position. The SAB expressed concern about potential conflict of interest on the part of designated peer-review leaders, noting that current agency policy, as stated in Section 1.4.4 of the peer-review handbook, allows an agency decision-maker on a particular work product to be the peerreview leader (EPASAB 1999~.
|
From page 124... ...
Although the decisionmaker should retain the authority to overrule provisionally any decisions or objections from a peer-review leader, with the final decision to be made by the EPA administrator, the independent decisions and any objections of a peer-review leader should be preserved and made a part of the agency decision package and public record for a work product. if such an independent assessment produces criticism of the adequacy or outcome of a peer review, EPA's policy should be to ensure that such criticism is clearly noted, divulged, and explained.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.