Skip to main content

Currently Skimming:

8 Application and Review Process
Pages 171-190

The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.


From page 171...
... OVERVIEW OF THE APPLICATION AND REVIEW PROCESS Each year, NCER and NCSER oversee multiple grant competitions. In 2021, NCER and NCSER awarded more than 147 research grants to universities, research firms, developers, and other organizations.
From page 172...
... 172 FIGURE 8-1 Annual awards for NCER and NCSER, 2002–2020, for grants categorized as Exploration, Development & Innovation, Efficacy, Replication/Effectiveness, or Measurement. SOURCE: Klager & Tipton, 2021 [Commissioned Paper]
From page 173...
... . Applicants to the main competitions receive scores from the review panels about a week after the panel meetings end, and summary statements (narrative reviews and discussion summaries)
From page 174...
... . While some speakers urged the committee to recommend that IES adopt shorter proposal lengths, committee members were concerned that this might limit the level of detail in IES proposals that reviewers need to judge the proposals, especially in light of the committee's call for basing the significance of the research in the needs of the field as well as in disciplinary knowledge.
From page 175...
... Among other things, they "determine the number and type of review panels needed, select and recruit peer reviewers, assign grant applications to the appropriate review panels, [and] assign primary reviewers to each application" (IES, 2021a)
From page 176...
... . Panel chairs are provided with the materials described above, as well as with a Panel Chair Supplement to the IES Guide for Grant Peer Review Panel Members.
From page 177...
... ISSUES WITH THE CURRENT APPLICATION AND REVIEW PROCESS As noted above, the annual process has served IES well in that it is predictable, investigators have ample information to write their proposals, and the procedures to score proposals and award funding provide all stakeholders with a common framework for assessing a study's potential for funding. Despite these strengths, the committee's assessment of the current application and review process revealed three issues that if addressed, may allow IES to build on its current strengths toward funding even stronger and more useful research: (1)
From page 178...
... by project type, and, within project type, by institution type.2 These tables are inclusive of all NCER and NCSER grantmaking, including research centers, training, and research grants, but exclude funding for Small Business Innovation Research grants. These data indicate that overall, approximately 7 percent of NCER and 8 percent of NCSER grants have been held by Minority-Serving Institutions (MSIs)
From page 179...
... There is much to learn about the role that multiple perspectives in the review process can play in supporting high-quality research, as the current literature on diversity in review panels4 has come to suggest.5 For example, Langfeldt and colleagues (2020) found that review panels with scholars from multiple disciplinary backgrounds and approaches more frequently supported diverse forms of research by extending definitions of quality beyond disciplinary norms.
From page 180...
... For this reason, the RPP column only includes grants with the RPP project type specifically identified, and therefore does not include the entire suite of partnership investments.
From page 181...
... TABLE 8-2  Average Proportion of NCSER Funding by Project Type and Institution, 2002–2020 SOURCE: Klager & Tipton, 2021 [Commissioned Paper]
From page 182...
... . From their personal experiences, committee members noted IES review panels often include a range of disciplines, with panels typically including those in both the NCER and NCSER communities, researchers in multiple disciplines that pertain to the panel, and experts in methods and measurement.
From page 183...
... Reviewers are instructed to carefully read the RFA and evaluate applications based on the stipulations of the most current RFA text. Additional materials are provided to panel chairs who meet with the Office of Science prior to the panel meeting; however, it is the experience of members of this committee that chairs of review panels are left to their own discretion to lead and facilitate the conversation around individual applications.
From page 184...
... In the absence of clear and meaningful anchors for judgment, reviewers in different panels may be harsher or more lenient than others and, over the review panel meeting period, there may be drift in these scores. Furthermore, the committee notes that while the scoring scale is continuous, it is understood by committee members who have participated in this process that a review score below 2.0 is typically considered "fundable" and a score above 2.0 is not, as noted earlier in this chapter.
From page 185...
... The committee recognizes that this requirement represents an attempt to ensure that funded research ultimately makes its way into the hands of "end users." However, we have identified a set of ways in which the current dissemination requirement does not actually function to ensure that funded research will be useful to education stakeholders. As with the Significance section described above, the RFAs include relatively open-ended instructions with minimal guidance for the required dissemination plan, and no clear direction for how reviewers should evaluate the dissemination requirement.
From page 186...
... Given the proximity of these professionals to the work of education, it is possible that the review process is missing a unique opportunity to ensure the application and review process yields useful research. The issues highlighted above, taken together, point to a process wherein reviewers lack a clear north star by which to make calibrated judgments about what proposed research will be useful to stakeholders in the field, which can result in funded research that does not sufficiently meet the needs of education stakeholders and decision makers.
From page 187...
... and co-PIs, composition of review panels, and study samples. Specific to the second issue noted in Chapter 8 -- timely and responsive application cycles -- the committee found evidence that the current structure of a single annual review panel is not functional for the research community in education, and a September deadline for proposals is particularly problematic given the timing of the school year.
From page 188...
... Importantly, the goal of this work is for IES to define a role for these communities that is both distinct and meaningful, such that these already burdened professionals can maximize their valuable time and effort. RECOMMENDATION 8.4: IES should engage a working group representing the practitioner and policymaker communities along with members of the research community to develop realistic mechanisms for incorporating practitioner and policy-maker perspectives in the review process systematically across multiple panels.
From page 189...
... Some thought that the program officer could take on this role; a problem, however, is that this changes the role of program officers, opening them up to have undue influence on funding decisions. Another idea was for the panel chair to take on this role; here the concern was that this would increase reviewer burden.
From page 190...
... . Education Research Grants Program Request for Applications.


This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More information on Chapter Skim is available.