Strengthening Peer Review in Federal Agencies That Support Education Research (2004)

Peer review is a method used to inform decision making by engaging experts in a critical evaluation of the merits of a product or proposal. Although it takes on many forms and serves a variety of purposes, it is most commonly known as a mechanism for judging the quality of proposals for research funding or manuscripts submitted for publication in academic journals.

The focus of this report is on peer review as it is applied to the evaluation of proposals for federal funding of education research projects. To set the proper context for our treatment of this topic, this chapter provides an overview of the nature and use of peer review in U.S. science policy; out-lines current policies and initiatives with implications for peer review in federal agencies; highlights the inherent tensions between political and scientific values in federal peer review systems (in education in particular), and describes our sources of evidence for, and approach to, setting forth our conclusions and recommendations.

Although there is no single definition of peer review across the many federal agencies that employ it (U.S. General Accounting Office, 1999), it is essentially a mechanism by which experts (“peers”) provide input to decision makers on the merits of proposals for research dollars. Peer review confers the imprimatur of high-quality to research proposals throughout

the federal government—judging quality (and potential quality) and lending credence to the allocation of scarce public resources.

The use of peer review in the United States can be traced back to the 19th century; for example, the Smithsonian Institution created an advisory committee for reviewing and recommending funding proposals in the 1840s (Guston, 2000). Since the 1940s and 1950s, peer review has evolved into a cornerstone of federal science policy, and today it is widely recognized as the preferred method for judging the merits of research proposals (Kostoff, 1994), for the ultimate purpose of improving government decision making. More specifically (Guston, 2000, pp. 4-31):

As a tool of science policy, peer review is embedded in the American system of government. Since it is used to steer the investment of federal funding in research, peer review is necessarily and appropriately influenced by political values and accountable to elected officials for the fairness of its processes and the success of its outcomes. Accountability to the public suggests, among other things, that the process be transparent and legitimate to a broad range of stakeholders.

As a tool for researchers to decide the merits of new research projects and directions, peer review is also part of the fabric of scientific communities.¹ Scientific judgment is formalized through the peer review process, as groups of researchers familiar with an area of scientific inquiry reflect on the likelihood of proposed ideas to advance what is known in productive ways. As such, peer review is a tool through which researchers develop, sustain, and communicate their professional culture, suggesting the need for a buffer from political influences.

Peer review in federal agencies, then, reflects both the principles of democratic accountability and the principles of scientific merit. In practice, upholding both sets of principles simultaneously can be difficult. The ways in which agencies go about promoting both take on many different forms,

and negotiating the proper roles and boundaries of each is an inescapable and constant task, as the description of the current policy context below makes clear.

No single policy sets standards for, or provides oversight of the use of, peer review in the federal government. In fact, peer review is largely a practice shaped by culture and experience. However, there are a host of statutes, regulations, and other policy issuances that influence (or could influence) the structure and use of peer review across a broad swath of agencies, including those that fund education research. Some of these policies invoke peer review as a way to promote particular objectives, including enhancing the technical quality and credibility of information disseminated by federal agencies (e.g., U.S. Office of Management and Budget, 2003, 2004), supporting research that is rigorous and relevant to national problems (e.g., Education Sciences Reform Act of 2002, H.R. 3801), and using scientific advisory panels to inform government decisions of many kinds (and thus is subject to the Federal Advisory Committee Act, P.L. 92-463, Sec. 1, Oct. 1972). In addition, there are policies not about peer review per se that nonetheless have potentially significant implications for its management and use. For example, the President’s Management Agenda, a recent directive aimed at managing and investing federal dollars more efficiently, has as-yet unknown implications for federal personnel, including those who oversee the peer review infrastructure in scientific agencies.

In the past few years, federal officials have set forth many of these new initiatives and proposed a range of clarifications and modifications to other elements of this policy apparatus, inviting a spirited debate and illustrating the tensions that arise as peer review serves both political and scientific ends. We provide an overview of a few of them here to characterize the current landscape of which our consideration of peer review of education research proposals is a part, to demonstrate the high stakes associated with the topic, and to illustrate and foreshadow some of the complexities of peer review in a range of fields, disciplines, and applications.

For example, the U.S. Office of Management and Budget (OMB)—the agency in the Executive Office of the President charged with developing and overseeing the implementation of the federal budget—recently issued a bulletin to provide guidance for defining government-wide standards for the peer review of “influential” scientific information and assessment

(including social science studies). The intent of the guidance is to improve the quality of the information the government uses to make policy, for example, in developing federal regulations that govern the nation’s safety, health, and environmental policies (U.S. Office of Management and Budget, 2003, 2004). Although the purview of the bulletin explicitly excludes the peer review of research proposals (our specific topic), the input of the scientific communities critiquing an early draft of the bulletin raised important questions and issues about the roles and purposes of peer review in general, many of which we address in this report. Later in this chapter, we highlight some of these commonalities as prologue to our consideration of peer review of education research proposals in federal agencies.

Another recent example of policy reforms with implications for peer review stems from the competitive sourcing and consolidation directives contained in the President’s Management Agenda, the Federal Activities Inventory Reform Act of 1998, P.L. 105-270, and the OMB Circular Number A-76. Overall, these initiatives are intended to promote efficiency in government operations. Agency implementation of these wide-ranging initiatives features, among other requirements, a multiyear effort to catalogue and to assess whether the functions performed by federal personnel—including those who support the peer review infrastructure in research agencies—are “inherently governmental functions” and whether they could be more efficiently performed by private sources. While the scope of this effort and its implications for federal jobs are still in flux, it is possible that organizations like the National Institutes of Health (NIH) Center for Scientific Review (which oversees the peer review of grant proposals to the agency) could be downsized. Because peer review is so central to NIH and other research agencies, many researchers have expressed concern about the potential negative consequences of this initiative for the peer review infrastructure in the federal government (Kaiser, 2003).

The centrality of peer review for funding decisions in scientific agencies and for the use of scientific information throughout the government is also reflected in recent agency planning documents prepared to comply with the Government Performance and Results Act of 1993, P.L. 103-62. The Department of Interior’s most recent strategic plan, for example, states: “The world-wide hallmark of good science is the collegial, cooperative, peer review of study plans and experimental results.” Similarly, the U.S. Department of Education’s (ED) strategic plan cites the need to “articulate clear standards” for peer review as a strategy for raising the quality of research funded or conducted by the department.

The concerns raised by scientists in response to the OMB peer review bulletin and the competitive sourcing efforts, coupled with the prominence of peer review in strategic planning documents in federal agencies, suggest that there is strong consensus about the important role of peer review among researchers and policy makers alike. But because it sits at the nexus of science and government, peer review is not without its controversies in practice. Recent events in the halls of Congress provide another case in point. During debate on the fiscal year 2004 labor, health and human services, and education appropriations bill, an amendment was introduced to eliminate funding for a handful of projects that had passed through the NIH peer review system because the subject of inquiry—human sexuality—was objectionable to some members. A debate ensued about the appropriate role of Congress vis-à-vis the merit-based peer review process. In the end, the amendment was defeated and the projects maintained funding, but the controversy continues.

Recent federal initiatives in education research and the critical role of peer review in it also illustrate the often controversial negotiations and high-stakes debates that characterize peer review in the federal government, providing further context for our consideration of peer review with respect to education research funding in the federal government. Most notable in this regard are a set of federal education policies aimed at applying “scientifically based research” to improve policy and practice. In many recent reauthorizations of the K-12 education laws that govern the federal role in elementary and secondary education, there is explicit reference to the use of research to inform reform efforts. Such language appears over 100 times in the No Child Left Behind Act of 2001 alone. A component of the detailed definition of scientifically based research in the act—which is essentially the standard that federal grantees must meet in providing evidence to support their program choices—is that it “has been accepted by a peer-reviewed journal or approved by a panel of independent experts through a comparably rigorous, objective, and scientific review” (P.L. 107-110, the No Child Left Behind Act). Similar provisions that include reference to peer review appear in the Education Sciences Reform Act, H.R. 3801, legislation passed in 2002 that replaced the Office of Educational Research and Improvement (OERI) with the Institute of Education Sciences (IES) as the primary research arm of the ED.

In crafting these provisions, members of Congress and their staffs aimed to upgrade the quality of research and to promote its widespread use (see, e.g., Sweet, 2002); here again is a reflection of policy makers’ faith in

peer review. Among many education researchers, however, the insertion of these provisions into federal law has provoked protests about what they view as an encroachment on their profession to define legislatively (politically) what “counts” as rigorous, scientific education research (Feuer, Towne, and Shavelson, 2002; Erickson and Gutierrez, 2002). Arguably, the profile of education research is at a high point, but so too are the controversies in and around the field of education researchers.

In this environment, IES officials have recently made changes to their peer review system. The leadership of the agency contracted out the management of their peer review system, a responsibility formerly held by agency staff. IES director Grover (Russ) Whitehurst explained this decision at the workshop, saying that it was made to reduce the likelihood of bias or the appearance of bias in the agency’s peer review system by separating agency staff who develop research program announcements from the review process. In the current system, IES staff generates lists of potential reviewers. In conjunction with the chair of each review panel, the independent contractor then vets and appoints the reviewers, with Whitehurst approving the final list. Concern was raised by participants at the workshop and elsewhere that this arrangement could result in the exclusion of relevant perspectives from the important considerations that take place at the peer review table.

We do not intend to adjudicate the current or future consequences—positive or negative—that may come of any of the initiatives we highlight in this brief sketch of the policy landscape. In addition to providing context for our consideration of peer review of education research proposals, this snapshot of recent events illustrates the complexities inherent in peer review in general and relevant activities in education research specifically, laying the foundation for our discussion of the use of peer review in federal agencies that support education research.

Taken together, the conversations that have swirled around this array of activity go to the core issues involved in developing effective peer review systems and the proper role of researchers, elected officials, and other stakeholders in it. Indeed, many of the same issues are raised regardless of the type of review or the scholarly field in which the debate is situated. For example, a November 2003 National Research Council (NRC) workshop

convened by the Science, Technology, and Law Program on Peer Review Standards for Regulatory Science and Technical Information convened stakeholders to discuss the initial draft of the OMB peer review bulletin. In offering a conceptual overview of issues in this context, Sheila Jasanoff, a leading expert in science policy and author of the influential 1990 book The Fifth Branch, noted that peer review serves a number of different purposes in government and that its processes vary substantially across contexts, citing the “endless diversity” of existing models and insisting on the need for flexibility in agencies to develop systems that meet their needs. John Graham, the OMB official leading the development and implementation of the bulletin, stressed the need to bring top experts to the table, but concerns were raised about who to involve and how to deal with potential conflicts of interest and biases. Donald Kennedy, editor-in-chief of Science magazine, discussed the importance of, and complexities involved in, developing a process that is sufficiently transparent and inclusive to promote its legitimacy among a range of stakeholder groups, including practitioners and consumers.

These issues were central to the discussion of education research peer review at our own workshop and are prominent themes in this report. Echoing Jasanoff’s observation of the multiple purposes that peer review is asked to fulfill and the variation in models across and within agencies that fund education research, we argue for flexibility for agencies coupled with a clear articulation of the objectives each system is designed to serve. We tackle the tough question of what counts as expertise in the diverse and applied field of education research. Specific issues of whether and how to involve stakeholders in the process and how to identify and deal with potential conflicts of interest and bias among candidates for participating in peer review panels are also addressed, picking up on the issues Graham and members of the scientific community raised about assembling panelists. And finally, a theme that runs throughout the report echoes Kennedy’s concern of ensuring a system that is widely viewed as legitimate by the many diverse stakeholders in science, including scientific education research.

This report is one in a series designed to highlight and to promote improvements in the quality of scientific research in education. Our consideration of peer review as one important leverage point for promoting

quality is informed by three main sources of evidence: the workshop the committee held on the topic in February 2003, a select review of relevant literature, and our own experience on peer review panels.

We designed the workshop to promote a broad-based discussion of the purposes, models, and results of peer review systems used across a range of federal agencies that support education research in light of the recent changes and issues in the policy landscape we have described. To help frame the event, we commissioned Edward Hackett and Daryl Chubin, authors of the highly influential book Peerless Science: Peer Review and U.S. Science Policy (Chubin and Hackett, 1990), to write a paper on the institutional and social contexts of peer review of education research proposals and to present major themes at the start of the workshop. We organized the rest of the workshop sessions to address topics and to feature speakers that provided maximum coverage of issues and representation of viewpoints. Workshop speakers included federal officials from the ED; the NIH; the Office of Naval Research (ONR); and the National Science Foundation (NSF). In addition, speakers included investigators from a range of fields in education research and social science disciplines as well as from epidemiology and nuclear physics.

Over the course of the day and a half long dialogue, this diverse set of experts and policy makers provided their perspectives and experiences and, in limited cases, the results of their research, on such issues as the purposes of reviews, selection and training of reviewers, review criteria and scoring systems, roles of staff and stakeholders in the process, and timelines and other management issues. The agenda for the meeting, a transcript of the presentations and discussion sessions, and the Hackett and Chubin paper can be found at http://www7.nationalacademies.org/core/Peer%20Review.html. The wide-ranging and probing dialogue that resulted from the paper and the event are the main sources of support for the conclusions and recommendations in this report.

A secondary source of evidence for our conclusions and recommendations is a select review of the published literature on peer review. This empirical research base on peer review of research proposals is surprisingly limited; while there are some important exceptions (several of which we cite in this report, and some of which we individually authored), most publications on the topic are based on theoretical arguments and personal experience. We draw most heavily from a few seminal publications that provide a survey treatment of issues in peer review and that focus on education research specifically, including Chubin and Hackett (1990), the U.S.

General Accounting Office (1999), August and Muraskin (1998), and several publications of the NRC. The NRC reports we consulted related to peer review and science policy include Peer Review in Environmental Technology Development Programs (National Research Council, 1999), Evaluating Federal Research Programs: Research and the Government Performance and Results Act (Committee on Science, Engineering, and Public Policy, 1999), and Assessing the Need for Independent Project Reviews in the Department of Energy (National Research Council, 1998). We also relied on NRC reports related to education research, including Research and Education Reform: Roles for the Office of Educational Research and Improvement (National Research Council, 1992) and Scientific Research in Education (National Research Council, 2002).

Finally, we draw on our own personal experience serving on panels to supplement the evidence gathered during the workshop and in our limited review of published literature. Our collective experience on panels is substantial, including service for such federal agencies as the IES and its predecessor agencies, several directorates of the NSF, a number of study sections in the NIH, the Veteran’s Administration, the Department of Defense, and the Department of Energy; state agencies, such as the Massachusetts Department of Health; participation in panel reviews of research proposals to be funded by other national governments, such as those in the United Kingdom, Canada, Japan, Australia, and Israel; research reviews for philanthropic foundations and professional associations, such as the American Educational Research Association, the Association of Teachers of Preventative Medicine, the March of Dimes, the Spencer Foundation, the Markle Foundation, the Russell Sage Foundation, the Sloan Foundation, the All Kinds of Minds Foundation, and the McDonnell Foundation; and in peer review deliberations across a range of disciplines and fields, including sociology, history, psychology, cognition, epidemiology, statistics, cultural anthropology, nuclear physics, and an array of subfields in education research.

Our main objective in issuing this report is to inform decision makers charged with developing or maintaining peer review systems for education research proposals in a rapidly changing policy context. To meet this objective, we set a number of parameters to frame how we would approach designing the workshop and developing this report. We articulate these parameters to further orient the remaining discussion.

First, our conclusions and recommendations apply to peer review of education research proposals in any federal agency that funds research relevant to education. Our primary sponsor for the event—the National Educational Research Policy and Priorities Board—was disbanded with the passage of the Education Sciences Reform Act, H.R. 3801, in late 2002. That act created the IES, replacing the OERI and calling for the formation of a new policy board that will work with IES leaders on issues related to peer review in the agency. (As of this writing, the president has nominated several individuals to be members of the board; they now await confirmation by the Senate, as required by law.) In addition to informing this new board’s work, we also hope to provide direction for other government agencies that fund education research. Thus, we designed the workshop and wrote this report to apply to the range of federal agencies that use peer review (or will in the future) to aid education research funding decisions—including, but not isolated to, the IES.

Furthermore, although the committee did hear about an evaluation of OERI’s peer review process in the 1990s, the workshop and this report are not evaluations of OERI or any other agency’s peer review system. Rather, we treated the information we learned about agency systems at our workshop as examples of peer review in practice. We describe aspects of these systems, highlighting their strengths and weaknesses in meeting particular objectives and drawing on the experiences of the agency representatives to ground our deliberations and recommendations in the reality of the current federal policy environment.

Finally, our treatment of peer review in this report focuses only on federal systems designed to handle proposals for education research projects. The use of peers to judge proposals or products is common in other pursuits both inside and outside government. In the ED, for example, peer review has been used to judge state accountability plans for implementing certain provisions in Title I of the No Child Left Behind Act, as well as to provide input on state applications for funding under the discretionary programs authorized by the law (e.g., Reading First). In nonprofit and private institutions with research programs (e.g., foundations), peer review is also frequently used. We expect that much of our discussion of the issues and our recommendations will be relevant to these other uses of peer review, but we have not analyzed their generalizability to other such areas explicitly.

One other part of the education research infrastructure in which peer review plays an important role is in manuscript submissions to journals.

We did not take up this kind of peer review at our workshop in any detail and thus do not treat it in this report; we addressed this issue more directly in a later workshop on journals and their role in advancing knowledge in education (the workshop agenda and transcript from this event can be viewed at http://www7.nationalacademies.org/core/Journal%20Standards. html).

To develop a concise report for policy audiences, we treat a complex topic with over 100 years of history and scholarship thematically and selectively. We organize the remainder of this report into two chapters. Chapter 2 provides an analysis of some of the key features of and debates around the peer review of education research proposals. Building on that discussion, Chapter 3 contains our proposals for strengthening and improving peer review systems used to aid decision making about the federal funding of education research.