Peer Review in Environmental Technology Development Programs (1998)

In this chapter, the committee gives a general overview of the concept of ''peer review'' and highlights the basic elements of the peer review process. The committee presents a definition of peer review, discusses its potential benefits, and then summarizes the fundamental steps of the peer review process. The processes described in this chapter relate not only to OST's particular peer review program, but also to other peer review programs designed to evaluate the technical merit of technology development activities.

Peer review is used throughout the scientific and engineering communities to evaluate the technical merit of research proposals, projects, and programs. Although one might argue legitimately that peer review¹ is the name given to any evaluation of technical² merit by other experts working in or close to the field in question, the scientific and engineering communities generally use the term in a narrower sense. In its interim report, the committee adopted the following definition developed by the U.S. Nuclear Regulatory Commission that articulates these communities' sense of peer review:

A peer review is a documented, critical review performed by peers [defined in the USNRC report as "a person having technical expertise in the subject matter to be reviewed (or a subset of the subject matter to be reviewed) to a degree at least equivalent to that needed for the original work"] who are independent of the work being reviewed. The peer's independence from the work being reviewed means that the peer, a) was not involved as a participant, supervisor, technical reviewer, or advisor in the work being reviewed, and b) to the extent practical, has sufficient freedom from funding considerations to assure the work is impartially reviewed.

A peer review is an in-depth critique of assumptions, calculations, extrapolations, alternate interpretations, methodology, and acceptance criteria employed, and of conclusions drawn in the original work. Peer reviews confirm the adequacy of the work. (USNRC, 1988, p. 2)

In this definition, the term peer review has the following characteristics:

• expert (including national and international perspectives on the issue),
• independent,
• external, and
• technical.

Most importantly, peer reviews must be carried out by independent evaluators who are experts in the technical issues relevant to the projects under review. Such reviewers must be highly qualified³ and independent in order to evaluate credibly the scientific and engineering merit of the project (or subset of project components) with respect to current technologies, both domestic and international. In the report Allocating Federal Funds for Science and Technology (NRC, 1995a, p. 69), peers are defined as "established working scientists or engineers from diverse research institutions who are deeply knowledgeable about the field of study and who provide disinterested technical judgments as to the competence of the researchers, the scientific significance of the proposed work, the soundness of the research plan, and the likelihood of success." Note that such reviewers are not necessarily expert in, or familiar with,

the agency program or relevant contextual factors. These considerations are the proper province of agency management. It is important to note that internal reviews, although useful for program management, should not be confused with peer review. The independence of peer reviewers distinguishes them from internal reviewers; and thus, the term "internal peer review" is an oxymoron (Bozeman, 1993).

The USNRC's definition of "peer review" provides some guidance on the issue of potential conflict of interest by explicitly excluding potential reviewers who have been involved with the specific project being reviewed or who have financial interests in the outcome of the reviews. As Chubin and Hackett (1990) have pointed out, however, it is difficult, if not impossible, to fully separate persons with relevant expertise from those with potential conflicts of interest, because "experts" on a subject or technology almost necessarily have some interest in the outcome of the review. Dealing with such issues is a challenge to the management of any peer review program. The issue of conflict of interest is discussed more fully below in the section "Selection of Reviewers."

Peer review is used throughout the scientific and engineering communities to evaluate the technical merit of research proposals, projects, and programs. Peer review is recognized as an effective tool that R&D program managers can use to obtain high-quality input to decisions on allocating their resources (Committee on Economic Development, 1998; NRC 1995a). This is especially important in situations of constrained funding, where program managers are required to make decisions on the relative merit of projects within their program's research and development portfolio. If its results are used as a significant input into programmatic decision making, peer review can improve both the technical quality of projects in a research and development program and the credibility of the decision-making process. In the case of OST, such improvements may increase the likelihood of the program's producing technologies that prove effective in cleaning up contaminated sites throughout the nation's nuclear weapons complex. The 1995 NRC CEMT report recommended development and implementation of such a peer review program for the OST technology development program for just this reason (NRC, 1996).

The independence of peer reviewers makes them more effective than internal reviewers because experts who are newly exposed to a project often can recognize technical strengths and weaknesses, and can suggest ways to improve the project that may have been overlooked by those close to it (Bozeman, 1993). Peer review can improve the technical quality of projects in a research and development program in two ways: (1) by identifying projects that lack technical merit (or are technically inferior to other feasible alternatives) so that they can be discontinued early in the R&D cycle (before large investments of funds are made), and (2) by identifying specific ways to improve proposed or ongoing projects. As a result, a greater number of alternative projects can be supported in the early stages of the development cycle, thus increasing options and chances of ultimate success in meeting the program's objectives. In studies of corporate product development programs, Cooper (1993) has shown that such early decisions can result in a greater than 50 percent overall increase in productive efficiency.

When peer review results are used to improve the quality of a decision process (e.g., selection of proposals, prioritization of projects for funding), they also enhance the credibility of the decisions. External experts often can be more open, frank, and challenging to the status quo than internal reviewers, who may feel constrained by organizational concerns. Evaluation by external reviewers thus can enhance the credibility of the review process by avoiding both the reality and the appearance of conflict of interest (Kostoff, 1997a). In addition, peer reviews that are conducted publicly, using known reviewers and following an established process that provides immediate feedback in the reviewers' own words, can enhance credibility by increasing confidence in the review process (NRC, 1997b; Royal Society, 1995).

For all of these reasons, the use of peer review increases the likelihood that decisions are consistent with the best available scientific and technical information. Of course, peer reviews in and of themselves cannot ensure the success of a project or program. Effective peer review can increase the probability of project and program success, however. Realization of these benefits requires that the process of peer review be effective and credible and that its results be used as an important input in making decisions regarding future support for the reviewed project (Chubin and Hackett, 1990).

In the committee's interim report, the peer review process was defined by the following five general steps:

1. selection of proposals, projects, or programs to be reviewed;
2. definition of objectives of the peer review and selection of specific review criteria;
3. selection of the peer review panel;
4. preparing and conducting the peer review; and
5. use of peer review results in decision making.

In order for a peer review process to be credible and effective as a whole, each of these steps must be performed following well-defined procedures that are understood and accepted by everyone involved with the peer review process.

In many types of peer review programs, the selection process is straightforward (e.g., all proposals are reviewed before they are funded.). In some organizations that use peer review to evaluate the technical merit of ongoing projects (including DOE-OST), however, decisions must be made regarding which specific projects are to be peer reviewed and at what stage in their funding or development path. In the case of DOE-OST, for example, a large number of technology projects have been funded for years before the new peer review program was initiated in October 1996. As of May 1, 1998, 43 of 226 active projects had been peer reviewed. Specific issues related to this "backlog" of OST projects that have never been peer reviewed are addressed in Chapter 6 of this report. In the general case, peer review program managers must have a systematic and credible approach for selecting which projects (or programs) are reviewed by the peer review program. An effective selection process employs well-defined criteria to prioritize those activities to be peer reviewed.

The goals, or objectives, of the peer review must be spelled out clearly so that they are understood by all involved in the process (Chubin, 1994; Chubin and Hackett, 1990; Kostoff, 1997b). In addition, the specific review criteria (i.e., specific questions or issues that reviewers are asked to address in a particular review) should be defined prior to the selection of peer reviewers to ensure that the review panel as a whole has the appropriate mix of expertise required to address these issues. Because peer reviews are by definition technical in nature, both the objectives of the review and the review criteria should focus on technical considerations. Reviewers and presenters should be informed of the objectives and review criteria well in advance of the review.

For project reviews, the objectives and utility of peer review vary according to the stage of the technology development, adoption, and implementation processes. It is especially useful at the outset of a project—that is, at the proposal stage, when peer review can help to select which of several candidate technologies to develop. It must be emphasized, however, that peer review can play an important role even at later stages of development (e.g., at the point where a technology project moves from bench to field scale). Even at the implementation phase of technology development, peer review can be used to validate expectations of performance and to enable late-term modifications that would enhance a technology's utility. In early phases, peer review can help in the selection of technologies. In later phases, it can lead to late-stage refinements in technologies or to the validation of technologies already developed.

To be effective, review criteria should be consistent with the objectives of the review and be clearly understood by all involved in the review (i.e., reviewers, principal investigators [PIs],⁴ program managers). Many effective peer review programs employ a small number of general review criteria, and this is the standard operating mode for organizations that review a large number of proposals (e.g., NSF, NIH). For reviews of technology development projects at various stages of development, however, a flexible system that allows specific review criteria to be selected for individual projects can be effective in focusing reviewers on issues of particular interest for a given project or group of projects.

The selection of reviewers is a critically important step in the peer review process. The process for selecting reviewers must reflect the fundamental characteristics of peer review described earlier in this chapter (see section entitled "Definition of Peer Review") and the specific objectives and criteria for the particular review being organized. Reviewer selection should be conducted by a group independent⁵ of the one being reviewed (Cozzens, 1987; Koning, 1990).

Peer reviewers should be selected in accordance with formally established qualification criteria. In the committee's view, the minimum criteria for individual reviewers are relevant, demonstrated experience in the subject to be reviewed, including national and international perspectives on the issue (Bozeman, 1993; Porter and Rossini, 1985); and peer recognition (both nationally and internationally). In assessing an individual's qualifications for participation as a peer reviewer, all relevant career experience, published papers, patents, and participation in professional activities should be considered. The group of peer reviewers should be balanced by including individuals with an appropriate range of knowledge and experience to provide credible and effective peer review of the technology being judged (Porter and Rossini, 1985). It is also important to consider the individual's experience with peer review itself (Royal Society, 1995). In cases where a review panel is established, the chair should be internationally respected in the field under review and should be an experienced peer reviewer. Additionally, some representation of reviewers who have knowledge of competing or alternative technologies is desirable.

The reviewer selection process also must ensure that reviewers do not have real or potential conflicts of interest (e.g., not selected from OST or the OST contractor community; see "Conflict of Interest," Chapter 5) with the

activities under review or biases⁶ so that the panel as a whole is balanced (Abrams, 1991; Cole, 1991; Moxham and Anderson, 1992). In the ideal situation, peer reviewers, being fully independent and external, should have no conflicts of interest. That is, they should have no current or previous relationships with the PIs, their organization, their proposed project, or competing projects or technologies that would impair their ability to provide an objective review. However, for various real-world reasons (e.g., because contractors have many divisions and technical professionals often change jobs, or due to the inherent "interest" of any expert), it may not be practical (or possible) to avoid at least the appearance of conflicts. In some cases, it might be necessary to impose a "statute of limitations" on conflicts of interest whereby prior association with an investigator or organization that ended several years before the review does not necessarily preclude a potential reviewer from serving on a review panel.

Problems that might arise involving conflicts of interest can be mitigated by requiring all reviewers (and other persons involved) to sign a statement disclaiming or disclosing any real or apparent conflict of interest. If necessary, conflicted reviewers could be disqualified. For example, such statements are standard in the DOE (BES) national laboratories and are required by Public Law 102-564. All public officials must comply with this act and so disqualify themselves from engaging in any transaction or decision (e.g., appointment making, voting on an issue, entering into a contractual agreement, negotiating to affect decisions) that may materially affect their financial interest. Likewise, all reviewers in a peer review should affirm that they have no conflict of interest, whether financial, personal or intellectual. Some institutions—for example, the NRC—conduct a dialogue among review panel members, during which all panel members disclose and discuss their potential biases and conflicts of interest.

For a peer review to be objective and effective, peer reviewers should receive written documentation that describes the project and its significance (i.e., why the project is being conducted and what it proposes to contribute) and a focused charge that describes the purpose of the peer review and the review criteria. These materials should be provided to the reviewers well in advance of the review. In cases where a review panel is convened, the panel should be

provided with clear presentations by the project team, as well as adequate time to assess the project comprehensively so that the panel is able to write a report that effectively summarizes and supports its conclusions and recommendations.

One of the most challenging issues involved in the reviews of technology development projects (especially those involving industrial partners) is the handling of proprietary information necessary to evaluate the technical merit of a project. Reviewers may need to receive privileged information during the review process to evaluate the technical merit of a project. Complete respect for confidentiality is central to the successful operation of peer review (Royal Society, 1995) because technical information may not have been patented or copyrighted at the time of review and therefore requires protection as the intellectual property of the authors or proposers. Confidentiality of proprietary information during review can be dealt with by panel selection (i.e., avoiding reviewers with conflicts of interest) and by requiring panel members to formally agree not to use any such information without written permission from the author or proposer. If the PI does not disclose proprietary information, the project will receive a poor review because some technical bases for the evaluation will be missing. To avoid such situations, an investigator's agreement to disclose information critical to a meaningful peer review (under appropriate confidentiality agreements) should be a condition of the initial project award.

One consideration when planning and conducting peer reviews is whether the evaluations should be conducted anonymously or openly (i.e., using publicly known reviewers). Most peer reviews of proposals are conducted using anonymous reviewers (e.g., see Boxes 3.1, 3.2, and 3.3). The principal strength of anonymous reviews is that they may encourage reviewers to be more candid and frank in their evaluations, because they are shielded to some degree from potential reprisals from unsuccessful proposers. In practice, however, anonymity is often difficult to ensure due to characteristic points of view or writing styles, particularly in highly specialized fields (Chubin and Hackett, 1990). The shielding of reviewers from proponents of a project through anonymity also may enhance the perceived "fairness" of the review process because it requires that the reviewers base their evaluations solely on the written documentation

provided to them (which can be standardized in a relatively uniform format), rather than more "ad hoc" communications between the evaluators and proponents of different projects. This same lack of communication also can limit the effectiveness of anonymous peer reviews, however, because it severely limits the ability of reviewers to clarify questions that arise during the evaluation of written documentation.

Many peer reviews of programs and projects in progress are conducted using reviewers who are known to the proposers and other interested parties, and include open question-and-answer sessions, or "open reviews" (e.g., see Boxes 3.5, 3.6, and 3.7). It is important to note that the use of the term open review does not imply that all deliberative sessions are held in public. The ability of an evaluating body to frankly discuss the merits and weaknesses of a project and to reach consensus in a closed session is an important attribute of many open reviews. The principal strength of open reviews is that they allow for more detailed evaluations because reviewers are permitted to ask questions and request additional information to clarify issues that arise during the evaluation. The opportunity for interaction often allows evaluators to be more specific and constructive in their comments (e.g., by suggesting ways in which a project could be improved, rather than simply arriving at an overall ranking). Open reviews also can serve to increase the credibility of the review process because the process is more transparent to potential critics (e.g., government oversight bodies, advocates of unfunded projects), and because the reviewers are more accountable for their evaluations (Chubin and Hackett, 1990; Royal Society, 1995). The most significant potential weakness of open reviews is that reviewers may be less candid, especially in their evaluations of weak projects or proposals from well-known proponents, if they fear the possibility of reprisals.

A peer review program will be effective only if its results are an important factor in making program decisions, for example, regarding future support for the reviewed project and/or as input to improve the technical merit of the project (Bozeman, 1993; Cozzens, 1987). Peer review reports that clearly provide the rationale for their conclusions and recommendations are an essential first step in achieving these objectives. If a peer review has been planned for use in decision making, as recommended in this report, the use should be straightforward. In addition, there should be procedures to monitor how project personnel follow through on technical recommendations of the peer review panel. A well-established peer review program also should have specific metrics

to evaluate how well the peer review program achieves its objectives (see ''Metrics,'' Chapter 7).