"Triage" Approach for Reducing Project Backlogs
OST has participated in more than 800 environmental restoration technology development projects since 1989 (DOE, 1998b), 226 of which are currently funded technologies and most of which have not been peer reviewed. Peer review of these technologies would be invaluable in helping determine which are ready for deployment,1 which merit further development, and which should be canceled. The number of technologies is too large to be reviewed rapidly by the current OST peer review process, however, because only 28 peer reviews were completed in 1997 and only 38 reviews have been scheduled for FY98 (Table 4.1).2 The relatively small percentage of active projects that have been peer reviewed also raises questions about the effectiveness of the peer review process, because it does not allow a consistent application of peer review results in OST's decision-making process. For example, projects that have been peer reviewed may be more likely, or less likely, to be funded solely because they were reviewed, rather than because of their technical merit.
In Chapter 4, the committee recommends two procedural changes to improve both the rigor of the selection process and the efficiency of the review process. Although these changes would help deal with OST's backlog of projects, eliminating the problem would require more fundamental changes in its peer review process. In this chapter, the committee focuses specifically on how OST could reduce this large backlog of OST-supported projects that have never been peer reviewed.
OST's current practice in which nearly all peer reviews include formal presentations by the project team, followed by deliberations by the panel, and further question-and-answer sessions over the course of two to three days, places
significant limits on the number of projects that can be peer reviewed by a single panel.3 Even if the number of projects that were peer reviewed during a single Type I review could be increased by improved efficiency, OST's backlog of technologies that have never been peer reviewed still would take years to be eliminated through its current process. If OST is to fulfill its policy that "all projects are to be peer reviewed" in the short-term (i.e., the next year), it will have to make significant changes in how peer reviews are conducted.
The committee recommends that OST consider adopting a "triage" approach that would allow far greater numbers of technologies to be peer reviewed. This approach would involve a formal prescreening of projects by peer reviewers based exclusively on the written documentation on the project—in effect, a "mail review" of projects, to be followed by a formal meeting of the panel to discuss and rank the projects. During this prescreening review, panel members would be asked to rank all related technologies in a given area that are being considered for additional development or deployment.4 Rankings from the panel as a whole could then be used by OST program managers to determine those highly ranked, low-budget projects that should be considered for funding without additional peer review; those highly-ranked projects that should receive a more detailed evaluation (including presentations by the project team and question and answer sessions); and those technically weak projects that should not be considered for funding. This approach would provide OST program managers the basis for discontinuing funding for technically weak projects and might provide them with sufficient technical input (to be supplemented by input on nontechnical factors) to make a decision to fund a low-budget project. The prescreening evaluations should not be used as the sole means for providing technical input into decisions to fund high-budget, environmental remediation projects, however. Peer reviews involving presentations by the project team and question-and-answer sessions should be carried out for all projects involving significant capital investment by OST.
Because prescreening evaluations require only written documentation on the projects to be reviewed, the triage approach could be used to evaluate a large number of projects at a single review. For example, a single review could evaluate all projects developed to address a specific type of environmental management problem or an entire OST FA/CC program. The approach also could include related projects from OST's inventory of nearly 600 projects that
are not currently funded. This might allow OST to identify especially promising technologies within its inventory that should be funded for demonstration or deployment.
Because of the breadth of technical issues that might be evaluated during a review of a large number of technologies, the number of reviewers required for the prescreening evaluation could have to be significantly larger than the five to six members typically employed for an OST Type I review. If the prescreening review process results in a greatly reduced number of technologies requiring a formal, panel review (as intended), fewer reviewers might be needed for the panel review itself, however.
Increasing the efficiency of OST's review process through the triage approach outlined above could result in some trade-offs in terms of the quality of the peer reviews, however. The same level of detail and thoroughness that is possible through panel reviews involving lengthy interaction and questioning of the project team would not be possible in the prescreening evaluations. The difference in quality of the reviews could be minimized by ensuring that the written documentation provided to the panel is complete and of high quality. A detailed proposal or statement of work for every project, as recommended earlier in this report, would be especially important for these prescreening reviews. The committee believes that the potential decrease in quality of some peer reviews, however, would be more than offset by the increased effectiveness of a peer review system in which all projects funded by OST have been through a credible peer review.