Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
C H A P T E R 4 Recommendations ederal agencies that support research in science and engineering F have been challenged to find the most useful and effective ways to evaluate the performance and results of research programs that they support. Through a series of workshops and panel discussions involving industry, agency, and oversight personnel, COSEPUP has gathered information and formulated principles that can be helpful to agencies in evaluating their research programs and helpful to oversight bodies in considering the evaluations. COSEPUP offers six recommendations. COSEPUP intends to address mechanisms and guidelines for implementing the recommendations in meetings and workshops with representatives from agencies and congres- sional and other oversight bodies. Given the diverse portfolio of research conducted by federal agencies and the urgency of address- ing the questions of how research can be evaluated in the context of GPRA, the level of detail and specificity needed in designing procedures and guidelines for implementation was beyond the scope of this report. Recommendation 1: Because both applied research and basic research can be evaluated meaningfully on a regular basis and are vital to research and mission agencies, research programs should be described in strategic and performance plans and evaluated in performance reports. The performance of research is critical to the missions of many federal agencies. Therefore, a full description of an agencyâs 37
EVALUATING FEDERAL RESEARCH PROGRAMS goals and results, which is a principal objective of GPRA, must contain an evaluation of research activities and their relevance to the agencyâs mission. Recommendation 2: For applied research programs, agencies should measure progress toward practical outcomes. For basic research programs, agencies should measure quality, relevance, and leadership. In addition, agencies should conduct periodic reviews of the overall practical outcomes of an agencyâs overall past support of applied and basic research. The use of measurements needs to recognize what can and cannot be measured. Misuse of measurement can lead to strongly negative results; for example, measuring basic research on the basis of short-term relevance would be extremely destructive to quality work. Because the evaluation of applied research is directly connected to practical outcomes, whereas the evaluation of basic research is in terms of quality, relevance, and leadership, which ultimately lead to practical outcomes, there might be a tendency to bias an agencyâs overall research program toward applied research at the expense of basic research. This should be avoided, and a proper balance should be maintained. Recommendation 3: Federal agencies should use expert review to assess the quality of research they support, the relevance of that research to their mission, and the leadership of the research. Expert review must strive for balance between having the most knowledgeable and the most independent individuals serve as members. Each agency should develop clear, explicit guidance with regard to structuring and employing expert review processes. 38
Recommendations The most effective way to evaluate research programs is by expert review. The most commonly used form of expert review of quality is peer review. This operates on the premise that the people best qualified to judge the quality of research are experts in the field of research. This premise prevails across the research spectrum, from basic research to applied research. A second form of expert review is relevance review, in which potential users and experts in other fields or disciplines related to an agencyâs mission or to the potential application of the research evaluate the relevance of research to the agencyâs mission. A third form of expert review is benchmarking, in which an international panel of experts compares the level of leadership of a research program relative to research being performed worldwide. Recommendation 4: Both research and mission agencies should describe in their strategic and performance plans the goal of developing and maintaining adequate human resources in fields critical to their missions both at the national level and in their agencies. Human resources should become a part of the evaluation of a research program along with the programâs quality in terms of research advancement, relevance in terms of application development, and leadership in terms of the ability to take advantage of opportunities when they arise. In early drafts of strategic and performance plans, agencies have generally omitted discussions of education and training, which are fundamental to the ability of agencies to fulfill their missions. The goal of developing and maintaining adequate human resources in fields critical to their missions should be supported by plans that produce that outcome. The nation cannot benefit from advances in science and technology without a continuing supply of well- educated and well-trained scientists and engineers. In addition, in the absence of such a flow, the capability of an agency to fulfill its 39
EVALUATING FEDERAL RESEARCH PROGRAMS mission will be compromised and the knowledge learned and technology developed will be lost. Recommendation 5: Although GPRA is conducted agency-by-agency, a formal process should be established to identify and coordinate areas of research that are supported by multiple agencies. A lead agency should be identified for each field of research and that agency should be responsible for assuring that coordination occurs among the agencies. It is common and valuable for multiple agencies to ap- proach similar fields of research from different perspectives. Indeed, this pluralism is a major strength of the U.S. research enterprise. However, better communication among agencies would enhance opportunities for collaboration, help to keep important questions from being overlooked, and reduce instances of ineffi- cient duplication of effort. A single agency should be identified to serve as the focal point for each particular field of research so that all significant supported fields are covered. Information regarding support for that field should be provided to all the agencies in- volved in it so that they can adjust their efforts to ensure that the field is appropriately covered. Agencies should use benchmarking, which affords the opportunity to look across fields, in their efforts to understand the status of a particular field of research. Recommendation 6: The science and engineering community can and should play an important role in GPRA implementation. As a first step, they should become familiar with agency strategic and performance plans, which are available on the agenciesâ web sites. The researchers who work in agency, university, and industrial laboratories are the people who perform and best under- stand the research programs funded by the federal government. 40
Recommendations Many researchers contribute substantial time and effort to review- ing papers submitted for publication, grant applications, and program proposals, but few of them are aware of GPRA. Their greater involvement in implementing GPRA would be beneficial to the country. Increased contact with and advice from the broader scientific and engineering community regarding both the quality and the leadership position of agency research programs and the relevance of the research to agency missions can benefit the GPRA process. 41