Strengthening Peer Review in Federal Agencies That Support Education Research (2004) / Chapter Skim
Currently Skimming:
3 Strengthening the System
3 Strengthening the System
Pages 50-77
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 50... ...
We chose this perspective on the basis of an assessment of the current policy landscape and the near-term needs of two key parts of the federal government that support education research: the Institute of Education Sciences (IES) and the Office of Special Education Programs (OSEP)
|
From page 51... ...
3. Agencies that fund education research should develop clear statements of the intended outcomes of their peer review systems and establish organizational routines for ongoing, systematic evaluation of the peer review process.
|
From page 52... ...
In addition, peer review panels should be composed so as to minimize conflicts of interest and balance biases, promote the participation of people from a range of scholarly perspectives and traditionally underrepresented groups, and provide opportunities for professional development.
|
From page 53... ...
Recommendation 2: Agencies that fund education research should explicitly focus on two key objectives for peer review: (a) to iden tify and support high-quality education research and (b)
|
From page 54... ...
, a starting point for these discussions in peer review panels with respect to technical or intellectual merit criteria could be the principles of scientific education research outlined in Scientific Research in Education (National Research Council, 2002, pp.
|
From page 55... ...
Typically the main product of the peer review process is a slate, or list of proposals, which rankorders the proposals according to how peers rated them on the evaluation criteria. With variations on the theme, it is common practice for authorized agency staff (typically the director of the division or directorate)
|
From page 56... ...
When review criteria include explicit reference to innovation or originality, it is important for applicants to describe and to justify the ways in which the proposed work departs from, and could advance understanding in, current knowledge. However, since the peer review process can be conservative with respect to risk, additional attention to innovation is likely to be needed.
|
From page 57... ...
In Chapter 2, we analyzed several facets of peer review that can be leveraged to promote these professional development opportunities, including panel membership, standing panels, and feedback mechanisms. We elaborate on several specific strategies in recommendations 4, 8, and 10.
|
From page 58... ...
Using the two objectives we recommend to guide peer review systems in federal agencies that support education research, we offer a few examples of potential data collection and analysis efforts. Evaluating the extent to which the peer review process results in the identification and support of high-quality research could include such strategies as retrospective assessments of research portfolios by researchers and
|
From page 59... ...
We think they ought to use it to critically examine their own practices and to set an example for other organizations inside and outside government. Recommendation 4: Agencies that fund education research should build strong infrastructures to support their peer review processes.
|
From page 60... ...
While the specifics vary, common tasks include preparing grant announcements, identifying and recruiting reviewers, developing and managing reviewer training, handling logistics of the review process, summarizing the comments of reviewers, participating in review meetings, communicating with those submitting proposals, and (in fewer cases) using their own judgment in making final funding decisions.
|
From page 61... ...
suggesting that staff eliminate applications for funding activities that do not involve research. In determining the optimal size of review panels, staff must balance cost and efficiency criteria with the need for multiple perspectives and backgrounds.
|
From page 62... ...
For peer review systems to meet the overall goals of supporting highquality research and providing professional development for the field, procedures must be developed to facilitate communication about the proposals, among the group of reviewers and among the reviewers, applicants, and staff. The amount and type of feedback given on proposals varies across the
|
From page 63... ...
at NIH exemplifies these possibilities. Agencies that rely solely on ad hoc review panels miss valuable opportunities to develop the human resources of education research.
|
From page 64... ...
Recommendation 5: Effective peer review systems require plan ning and organization in advance of a review. In order to schedule, agencies that fund education research need relatively predictable levels and timing of funding.
|
From page 65... ...
If the meeting requires complicated travel, poor accommodations, or other sources of administrative burden, reviewers will be tired, demoralized, and less inclined to participate in future panels. Recommendation 6: Agencies that fund education research should uphold basic principles of peer review but retain flexibility in de signing peer review systems to meet their individual needs.
|
From page 66... ...
For example, as the President's Commission on Excellence in Special Education recently noted in its report (2002) , legislative requirements about the composition of review panels are especially difficult for an agency and can have deleterious effects on the ability of the peer review system to identify and support high-quality education research.
|
From page 67... ...
Recommendation 8: As a group, peer review panels should have the research experience and expertise to judge the theoretical and technical merits of the proposals they review. In addition, peer review panels should be composed so as to minimize conflicts of interest and balance biases, promote the participation of people from a range of scholarly perspectives and traditionally under
|
From page 68... ...
. Thus, peer reviewers of research proposals should be chosen first and foremost for their experience and expertise in an area of investigation under review.
|
From page 69... ...
In assembling peer review panels, attention to the diversity of potential reviewers with respect to disciplinary orientation as well as social background characteristics is important for a number of reasons. Diverse membership promotes the legitimacy of the process among a broad range of scholars and stakeholders.
|
From page 70... ...
Attending to promising scholars at early stages of their careers can also target professional development opportunities for up-and-coming researchers who have solid credentials but less experience reviewing. The testaments of many workshop participants citing early experiences serving on NIH (standing)
|
From page 71... ...
The first and most controversial practice used in some agencies involves the inclusion of practitioners and community members on peer review panels alongside researchers. Since this approach is a significant topic of interest generally and among the workshop participants specifically, we analyze the underlying issues as they pertain to the review of education research proposals in federal agencies in some detail and outline the conditions under which such an approach could be beneficial to all involved.
|
From page 72... ...
We also see the potential benefits of including practitioners and community members on panels evaluating education research funding applications to help identify high-quality proposals and to contribute to professional development opportunities for researchers, practitioners, and community members alike. With respect to quality, practitioners and community members are well suited to provide insights about the relevance and significance of research proposals -- an important evaluation criterion across all agencies represented at the workshop.
|
From page 73... ...
All peer reviewers -- whether they are researchers, teacher trainers, dissemination specialists, administrators, parent trainers, policy makers or others -- should be deeply knowledgeable about the area under investigation and screened for potential conflicts of interest and biases. Finally, to engage practitioners and community members on peer review panels successfully, it is critical that agencies provide thorough training to all reviewers so they understand the expertise they are expected to bring to bear to the review and can participate in the process effectively.
|
From page 74... ...
Practitioners and community members who represent diverse viewpoints bring important perspectives to education research. Their participation in the work of federal agencies that support education research should be ensured in ways that capitalize on their strengths: assessing the relevance and societal significance of the research.
|
From page 75... ...
While most scholarly proposals do contain similar elements -- the importance of the research question, how the study is to be conducted, what one expects to find, and what importance it has to the field -- proposals for external funding require a particular level of clarity and specificity that is not typical in other areas. It is not just the level of detail that distinguishes proposals for external funding: such proposals also have to be uniquely conceived and written according to guidelines that can change within and across the various agencies that support education research.
|
From page 76... ...
Based on workshop discussions, we see this as an area in major need of improvement in most agencies. The National Institute on Drug Abuse has a program to train investigators selected for review panels, which was described at the workshop by Teresa Levitin.
|
From page 77... ...
STRENGTHENING THE SYSTEM 77 this brief treatment to help education research policy makers approach the task of improving peer review in this era of evidence-based education. It is our view that the current emphasis on peer review is welcome -- provided that those charged with overseeing the process understand the strengths and weaknesses of various approaches and implement them with clarity of purpose.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.