8

Application and Review Process

While previous chapters focus on the content of grants funded by the National Center for Education Research (NCER) and the National Center for Special Education Research (NCSER), in this chapter we focus on the application and review process through which these grants are awarded. Understanding and making recommendations related to this process responds to the second element of the committee’s charge.

OVERVIEW OF THE APPLICATION AND REVIEW PROCESS

Each year, NCER and NCSER oversee multiple grant competitions. In 2021, NCER and NCSER awarded more than 147 research grants to universities, research firms, developers, and other organizations. This total included grants focused on each of the five project types (Chapter 4) and a myriad of topics (Chapter 5), as well as those focused on research methodology (Chapter 6) and training (Chapter 7). The overall funding for FY2021 was roughly on par with that of 2020, although less than 2010. The total planned funding commitment for grants initially awarded in FY2021 was $226,469,425 in NCER and $79,314,071 in NCSER. Figure 8-1 indicates the total funding for NCER and NCSER for grants that were categorized as Exploration, Development & Innovation, Efficacy, Replication/Effectiveness, or Measurement from 2002 to 2020.

The annual grant process¹ for the main NCER and NCSER Education and Special Education Research competitions begins with a Notice Inviting Application (NIA) published each year in the Federal Register, along with an accompanying request for applications (RFA) published on the Institute of Education Sciences (IES) website. The NIAs and RFAs for many—but not all—of the research and research training competitions are typically released, advertised, and promoted in spring (~ April–June) of each year, leading to grant submissions in late summer/early fall (~ August–September). Additional reviewers for relevant panels are recruited beginning in the summer, applications are released to reviewers for initial conflict of interest identification in November, and applications are assigned to primary reviewers for initial reviews in December. IES maintains some standing review panels with some principal members who continue on each year. The Office of Science also recruits new principal and rotating members for those panels. The Office of Science also recruits reviewers for single-session panels that are newly constituted panels for one-off or irregularly run competitions.

Panels meet in mid-winter (~ February) and final decisions regarding funding are made in mid-spring of the following year (~ April–May). Applicants to the main competitions receive scores from the review panels about a week after the panel meetings end, and summary statements (narrative reviews and discussion summaries) about a month after the panel meetings. If selected for funding, a first disbursement usually occurs in late summer at the earliest, over a full year after the grant application was submitted. There are also smaller competitions that are run at different times and with different time frames that have much shorter turn-around times.

This research grant cycle thus includes three major activities for IES staff: the generation of the NIA and RFA (primarily the responsibility of NCER and NCSER, with input from the IES Director and the Office of Science); the application receipt, processing, and peer-review process (primarily the responsibility of the Office of Science with input from NCER and NCSER); and the funding decisions and obligation of new awards (primarily the responsibility of NCER and NCSER with IES Grants Administration staff).

In this chapter, we discuss the role and function of each purpose of the review process. We then turn to a discussion of three areas where the committee believes the current structure and organization of the review process

___________________

¹ The committee notes that this grant competition schedule represents a typical schedule for the main Education Research and Special Education Research grants. NCER and NCSER regularly run several other competitions that are competed on different time schedules. In FY2021, the Research Centers ran a total of eight grant competitions.

presents challenges to meeting NCER and NCSER’s overarching goals, offering insight into how IES might consider each of these issues.

Elements and Functions of the Application and Review Process

The research community interprets IES’s mission and values primarily through the application and review process. Indeed, the application and review process is the primary way that IES is able to convey its understanding of what research the field needs to improve outcomes for students and educators. By identifying a set of topic areas and requirements for what high-quality research looks like, the application and review process is a codex for a field looking to understand IES’s priorities for research. Within those priorities, it is up to the applicants to determine the specific focus of their research. In this section, we discuss the purpose and functions of the three elements of the application and review process at IES: the request for applications (the RFA), panels and reviewers, and the review and scoring process.

RFAs

In FY2022, RFAs allowed grant proposals for NCER’s Education Research Grants competition to include up to 22 pages of a narrative that included four required sections: Significance, Research Plan, Personnel, and Resources. The committee heard testimony that this length exceeds those allowed by other agencies, including the National Science Foundation (NSF) (maximum of 15 pages) and National Institutes of Health (NIH) (maximum of 12 pages). While some speakers urged the committee to recommend that IES adopt shorter proposal lengths, committee members were concerned that this might limit the level of detail in IES proposals that reviewers need to judge the proposals, especially in light of the committee’s call for basing the significance of the research in the needs of the field as well as in disciplinary knowledge. In addition to this narrative, applications can include several required and/or optional appendixes on topics such as dissemination history (required), responses to reviewers (required for resubmissions), charts and figures, letters of agreement, and budget. In total, a grant proposal can thus include nearly 100 pages of material.

According to the committee’s review, requirements found in the RFAs are more explicit than those provided by other funders, with clear directions for each section, as well as suggestions for the kinds of content that have been included in past successful applications. The committee carefully reviewed a range of RFAs from both NCER and NCSER over time: in addition to hearing testimony from IES staff and other speakers, the committee reviewed multiple iterations of the document itself, searching for

places where the document was either unclear or redundant. Ultimately, the committee found that the explicit nature of the RFA’s directions is one of the strengths of the IES grant review system, even if it precludes a shorter proposal. The requirements articulated throughout the document scaffold a complex process even for first-time applicants who might be working in institutions without strong, centralized support for grant submission. The committee was particularly impressed by the document’s attention to detailed recommendations for strong proposals. For example, suggestions for Initial Efficacy studies include clear and explicit guidance regarding what should be reported regarding statistical power analyses (e.g., effect size selected).

Panels and Reviewers

Following the completion and submission of an application, each grant application that is responsive to and compliant with the requirements of the RFA to which the application was submitted is assigned to a specific review panel. The Standards and Review staff within the Office of Science at IES manages the entirety of the scientific peer-review process for NCER and NCSER’s grant competitions. In order to ensure the integrity of the review process, and allow the program officers to provide intensive technical assistance to applicants, the Standards and Review team is completely independent from NCER and NCSER. A contractor provides support to IES and coordinates many aspects of the logistics of the review process, as well as maintains and enhances the online peer-review system. Standards and Review staff are responsible for all of the substantive activities related to peer review. Among other things, they “determine the number and type of review panels needed, select and recruit peer reviewers, assign grant applications to the appropriate review panels, [and] assign primary reviewers to each application” (IES, 2021a). Thus, at the same time that NCER and NCSER staff are working to develop RFAs, encourage applications, and provide technical assistance to applicants, the Standards and Review staff is working to complete recruitment of reviewers. The majority of these are standing panels that currently include a commitment of 5 years from the principal members of the panels (although the panels also include rotating members who serve for a particular session, and ad hoc reviewers who provide specialized expertise and review a small number of applications). Additional single-session panels also occur when necessary. Depending upon the number of applications received, each standing panel might need one or more sections, each with approximately 15 members, including both experts in the topic area(s) itself as well as experts in measurement and methods (research design, data analysis, cost analysis) in education research. For reviewers, this commitment includes serving as the primary

reviewer on up to eight proposals, as well as reading and discussing all of the proposals that are forwarded after the triage process to the full panel for discussion and final scoring for the panel (between 10 and 20 typically).

Review Process and Scoring

The Office of Science oversees the review process. Prior to beginning the process, reviewers are provided with a variety of instructional materials to guide them through the premeeting, meeting, and postmeeting review process. Currently, the reviewer materials include an IES Guide for Grant Peer Review Panel Members, and a set of Review Notes with information specific to each panel or group of panels. In addition, the Office of Science now provides a set of three videos that explain what happens to an application after it is submitted, what the responsibilities are of an IES peer reviewer, and what panel meetings are like (including a mock panel meeting). Panel chairs are provided with the materials described above, as well as with a Panel Chair Supplement to the IES Guide for Grant Peer Review Panel Members. Before the panels meet, reviewers provide detailed feedback and scores (1–7, with 7 = Excellent) related to each of the review criteria specified in the relevant RFA, as well as an overall rating (1.0–5.0, with 1.0–1.5 = Outstanding). Based upon these initial primary reviews, the Standards and Review team “conduct[s] discrepancy analyses of initial rating scores, [and] conduct[s] the triage of applications to be considered by the full panel” (IES, 2021a). Applications above a given cut-score are then discussed by the full panel. For each application considered by the full panel, this includes a brief presentation by the primary reviewers (usually two to four reviewers per proposal), followed by a discussion by the full panel, panel discussion summary, reconsideration of initial scores by the primary reviewers, and final scoring by each panel member on both individual criterion (1–7, with 7 = Excellent) and overall (1.0–5.0, with 1.0–1.5 = Outstanding) scores. Importantly, each application is required to be reviewed on its own merits, relative to the expectations in the RFA, not in relation to other applications discussed. Given available funds, applications in the Outstanding and Excellent range, which generally corresponds to an average overall score of 2.0 or better, are considered for funding.

As noted above, this RFA and review process ensures that research funded through NCER and NCSER serves to advance the mission of promoting the development and evaluation of interventions to improve educational outcomes for students. Evidence of IES’s success in using the RFA and review process toward these ends can be seen in a few ways. First, IES has iteratively improved the quality of causal studies funded by shifting its RFA requirements such that successful proposals reflect contemporary understandings around rigorous design. For example, requirements regard-

ing assumptions and sensitivity testing for quasi-experiments, as well as sample size and statistical power requirements for randomized experiments, were not originally included in IES’s first round of RFAs, but were added in later in order to incentivize higher-quality studies. Similarly, the requirements addressing concerns regarding the ultimate usefulness of research to practice were added over time, including requirements for addressing issues of generalizability and sample recruitment, data sharing, and most recently, inclusion of a dissemination plan. The committee thought that this use of the RFA for promoting best practices was a strength of NCER and NCSER.

Finally, throughout this process, IES has established procedures to ensure that the system is fair and objective. This can be seen in the explicit criteria in the RFAs, the separation of proposals and review by the SRO, the inclusion of a thorough conflict-of-interest process, and the focus on review conducted entirely by a panel of experts. Akin to NIH but unlike NSF, IES program officers have no role in the review process, other than to encourage applicants and provide guidance on the RFAs. Thus, the determination for funding arises only in relation to the final proposal score and the cut-score for that particular year. The committee found that these steps to ensure the independence of the enterprise are a considerable strength of the current system.

ISSUES WITH THE CURRENT APPLICATION AND REVIEW PROCESS

As noted above, the annual process has served IES well in that it is predictable, investigators have ample information to write their proposals, and the procedures to score proposals and award funding provide all stakeholders with a common framework for assessing a study’s potential for funding. Despite these strengths, the committee’s assessment of the current application and review process revealed three issues that if addressed, may allow IES to build on its current strengths toward funding even stronger and more useful research: (1) IES does not consistently share demographic information on its applicants, reviewers, and grantees with the public, making it impossible to track whether the application and review process is resulting in an equitable distribution of awards and, if not, where in the process disparities are introduced; (2) the current procedures undermine IES’s ability to be timely and responsive to the needs of the education research community; and (3) the current procedures do not allow for sufficient understanding of how well-proposed research addresses the needs of the field. We review these challenges in the section below, describing how current regulations or procedures may inadvertently create barriers to funding the best possible research proposals.

Data on Applicants, Reviewers, and Grantees

As with all aspects of its charge, the committee formulated its considerations around how well the current application and review process functions in the context of the crosscutting themes identified at the beginning of this report (see Chapter 1). In light of these themes, one of the first questions the committee asked was how equitable the review process is in terms of those who applied for and were ultimately funded. This issue is particularly important to the committee given President Biden’s Executive Order, which asks agencies to assess “potential barriers that underserved communities and individuals may face to enrollment in and access to benefits and services in Federal programs” (Executive Order 13985, 2021). In order to better understand the implications of this order for funders, the committee heard testimony from IES staff, as well as from representatives from NSF and NIH.

The committee was surprised to find that in comparison to both NSF and NIH, IES reports very little data on equity. The most common source of data available is on institutions that receive IES grants. Tables 8-1 (NCER) and 8-2 (NCSER) provide overall funding (across years 2002–2020) by project type, and, within project type, by institution type.² These tables are inclusive of all NCER and NCSER grantmaking, including research centers, training, and research grants, but exclude funding for Small Business Innovation Research grants. These data indicate that overall, approximately 7 percent of NCER and 8 percent of NCSER grants have been held by Minority-Serving Institutions (MSIs); relative to other project types, MSIs were more likely to hold Exploration grants than any other type. By and large, most grants have been held by Carnegie-classified Research 1³ universities (68% NCER, 72% NCSER).

IES collects and reports considerably less information on applicants. A recent IES blog post reported voluntarily submitted demographic information for the principal investigators (PIs) on applications submitted to the FY2021 competitions (IES, 2021b). Across NCER and NCSER, 59 percent of PIs who received funding were female (compared to 62% of applicants; 82% response rate). Only 13 percent of awardees were non-White or multi-racial (compared to 22% of applicants; response rate 75%). Similarly, 3 percent of awardees were Hispanic (compared to 5% of applicants;

___________________

² Although NCSER was not founded until 2006, Table 8-2 includes nine grants that were initially awarded at the Office of Special Education Programs but ultimately inherited by NCSER at its inception. The trends in these data do not qualitatively change when these nine grants are excluded from analyses. Given that NCSER includes these data in their list of funded research, the committee elects to include these grants as part of NCSER’s portfolio.

³ Research 1 universities may also be Minority-Serving Institutions, and so may be counted in both groups cited here.

response rate 72%). Finally, 4 percent of awardees identified as having a disability (compared to 3% of applicants; response rate 70%). As written in the blog post, “These data underscore the need for IES to continue to broaden and diversify the education research pipeline, including institutions and researchers, and better support the needs of underrepresented researchers in the education community” (IES, 2021b). Moreover, these data only represent a single year in the life of IES, leaving the committee unable to assess whether the state of information above is typical, or if the situation is improving or declining.

Finally it is important to highlight that while there are very limited data on applicants and awardees, to date there is zero publicly available information regarding the demographic background of members of review panels.

While there is very little information available regarding equity in the Application and Review process, the available data surface significant challenges. Clearly, both non-White and Hispanic researchers are less likely to submit applications (22% and 5%, respectively). Even when they do submit applications, they are less likely to receive funding (13% and 4%, respectively).

Review Panels

Available research suggests that that there are reasons to attend to the composition of review panels that extend above and beyond the rationales for attending to equity noted in the section above. There is much to learn about the role that multiple perspectives in the review process can play in supporting high-quality research, as the current literature on diversity in review panels⁴ has come to suggest.⁵ For example, Langfeldt and colleagues (2020) found that review panels with scholars from multiple disciplinary backgrounds and approaches more frequently supported diverse forms of research by extending definitions of quality beyond disciplinary norms. In contrast, Huutoniemi (2012) found that panels of researchers from similar backgrounds competed to establish their expertise and authority using narrow criteria to advance specific fields. Diverse groups, in terms of race, ethnicity, and research background, are less likely to fall prey to “groupthink,” encouraging debate to counteract preformed preferences and biases (Esarey, 2017; Laudel, 2006; Antonio et al., 2004). Considering more diverse criteria of evaluation has been advocated to support innovative and risk-taking research (Azoulay and Li, 2020; Hofstra et al., 2020; Valantine

___________________

⁴ This section draws on findings synthesized for the committee by Zilberstein (2021).

⁵ The committee recognizes that attending to racial and disciplinary diversity in review panels in and of itself does not guarantee an equitable review process. Given the evidence about the importance of racial and disciplinary diversity in supporting high-quality research, we argue that this particular dimension of equity is of critical import.

TABLE 8-1 Average Proportion of NCER Funding by Project Type and Institution Type, 2002–2020¹

Table 4. Proportion of funding by grant category and institution type-NCER

SOURCE: Klager & Tipton, 2021 [Commissioned Paper]. Data from https://ies.ed.gov/funding/grantsearch/.

¹ As discussed in Chapter 4 and in the Klager and Tipton (2021) paper, the categories identified here are delineated by project type and not grantmaking program. For this reason, the RPP column only includes grants with the RPP project type specifically identified, and therefore does not include the entire suite of partnership investments.

TABLE 8-2 Average Proportion of NCSER Funding by Project Type and Institution, 2002–2020

Table 5. Proportion of funding by grant category and institution type-NCSER

SOURCE: Klager & Tipton, 2021 [Commissioned Paper]. Data from https://ies.ed.gov/funding/grantsearch/.

and Collins, 2015; Dezsö and Ross, 2012). Also notably, a lack of racially diverse reviewers perpetuates in-group bias and favoritism for the status quo, continually disadvantaging researchers from underrepresented groups whose research commonly lies outside of reviewers’ areas of expertise (Hayden, 2015).

From their personal experiences, committee members noted IES review panels often include a range of disciplines, with panels typically including those in both the NCER and NCSER communities, researchers in multiple disciplines that pertain to the panel, and experts in methods and measurement. At the same time—in the committee members’ experiences—most of the review panels were composed of researchers who had at some point been funded by IES. When considering this observation in concert with IES’s reported data that the majority of awardees are White, it stands to reason that current review panels may not be able to access the benefits associated with racially diverse groups.

Given the role that both racial and disciplinary diversity on review panels can play in supporting high-quality research, the committee notes the importance of ensuring that review panels are, in fact, representative of multiple perspectives. In this case, a lack of consistently reported information has undermined the committee’s ability to assess the degree to which IES has attended to these issues in its application and review process.

Timely and Responsive Application Cycles

The NCER and NCSER application and review processes takes, on average, 8–10 months from the time that a grant application is submitted until it is ultimately funded. Committee member experience (as reviewers and applicants) suggests that most grant proposals are not funded in their first submission but may take two or three submissions before ultimately being funded, resulting in a total process of as much as 3 years. While this timeline offers benefits in terms of both feasibility (for IES) and refinement of the proposal and research plan, it can impede the ability of researchers to be responsive to on-the-ground concerns of practitioners and decision makers in schools. Programs and interventions tend to move quickly within school districts, and it is likely that many programs that were ripe for research have been understudied due to the lack of federal funding at the crucial moment in time.

This timeline impacts proposals in that it makes it difficult to develop and maintain true partnerships with schools and districts. Currently, applications require letters of support from school district personnel indicating a commitment to take part in the study. However, school district superintendents and school leaders often move schools and school districts, as do teachers. From the researcher standpoint, the lengthy timeline means that

the schools recruited for the first application may ultimately not be available for the second application, resulting in them investing less heavily in the partnership than they may otherwise. From the school district position, this means that most researchers who contact them are unlikely to lead to productive partnerships in a timeframe that matters to them or, even worse, valuable time invested into partnership building may be wasted. Finally, in the committee’s experience, letters of support do not necessarily articulate a warrant for conducting research on a given topic in a given location, as support for conducting research is not always equivalent to identifying a rationale for why something is important. It is the committee’s judgment that the current letters of support mechanism is not ideally suited toward guaranteeing participation or identifying the significance of proposed research.

Coherence with the Needs of the Field

The committee reviewed the current application and review process with an eye toward whether or not the process resulted in research that is ultimately useful to the field. In considering these questions, the committee noted a set of critical junctures wherein the current procedures do not allow for sufficient information to assess the significance of individual proposals and the extent to which proposals, if funded, are likely to serve the needs of the education community. In this section, we delineate several places in the application and review process where we see this problem emerge.

Reviewer Preparation and Scoring

Reviewers are encouraged to engage with a series of preparatory materials in advance of their review process. Reviewers are instructed to carefully read the RFA and evaluate applications based on the stipulations of the most current RFA text. Additional materials are provided to panel chairs who meet with the Office of Science prior to the panel meeting; however, it is the experience of members of this committee that chairs of review panels are left to their own discretion to lead and facilitate the conversation around individual applications. In addition, Office of Science staff attend and monitor the panel meetings to address questions or issues that arise, and to ensure that review criteria are appropriately applied. Reviewers are asked to draw upon their own expertise when evaluating how well applications respond to each aspect of the RFA.

Although the committee does not dispute the substantial expertise that each reviewer brings to the process, we note the absence of any kind of directive or orienting material that allows reviewers to gauge the significance of a proposal against expressed research priorities or notable needs in the field. Further, reviewers are also explicitly advised against attempting to

build a complementary group of studies among those that they review and are asked instead to consider each study on its own individual merits. As a result, it is challenging for review panels to track whether a set of funded proposals coherently maps onto the needs of the field. This question, important as it is, is simply not structured into the review process.

Specific to scoring, the committee notes that after a proposal is discussed by the panel, the proposal’s original reviewers are able to change their holistic scores for the proposal and then every panel member submits a score. As noted earlier in this chapter, these scores are between 1 and 7 to the tenth of a point. The committee notes that, in our experience, there is no real anchor for this scoring and that different reviewers may conceive of the meaning of scores differently; for example, the difference between a 1.9 and 2.1 is likely measurement error, not a precise difference. In the absence of clear and meaningful anchors for judgment, reviewers in different panels may be harsher or more lenient than others and, over the review panel meeting period, there may be drift in these scores.

Furthermore, the committee notes that while the scoring scale is continuous, it is understood by committee members who have participated in this process that a review score below 2.0 is typically considered “fundable” and a score above 2.0 is not, as noted earlier in this chapter. As a result, a repeated concern is that it is likely that reviewers are not simply providing a scale score of “merit” when providing an overall summary, but also a “vote” regarding whether they think the grant should be funded. It is thus possible to bias the merit review process by providing slightly lower scores (just below 2.0) for grants that reviewers prefer, or slightly higher scores (just above 2.0) for grants they do not, thus making it possible for reviewers to “game” the system in ways that may result in bias and inequities. The committee discussed these concerns at length, but observed that a comprehensive understanding of potential problems in this arena would require deeper analyses of data on applicants.

RFA: Significance

Applications submitted to NCER and NCSER typically include four parts: Significance, Research Plan, Personnel, and Resources. In the FY2022 Education Research Grants RFA (IES RFA, 2022), guidance for strong applications indicates that the Significance section should include a description of “how the factors you propose to study are under the control of education agencies” [Exploration], why the intervention would “be an improvement over what already exists” [Development], a description of the “population of learners and educators intended to benefit from this intervention” [Development], and “the learners who should benefit … from this

intervention” [Initial Efficacy]. The full RFA document contains additional relevant guidance intended to support strong applications.

While each of these suggestions encourages researchers to consider how their particular intervention or study might connect to improving practice, it does not ask them to provide rationale that the problem the intervention is attempting to address is one in need of additional research. That is, it is possible that there are problems and opportunities that education decision makers face that need research (that would clearly be “significant”) and yet there are no studies conducted in this area (see Chapter 5 for our discussion about how current constraints impede the study of certain topics). At the same time, there may be many studies (each significant in a more narrow sense) on a single topic or intervention. Although the suggestions included in the RFA are intended to assist applicants in considering the current research landscape around a particular problem, they ultimately serve to direct applicants away from locating the value of their work inside the existing needs of educators and education stakeholders. Across many proposals and studies, the result of this framing is that it puts the interests of researchers above the needs of the field.

RFA: Dissemination

The Education Research Grant RFA includes a requirement that researchers identify a plan for how they will share the results of their study upon completion. The committee recognizes that this requirement represents an attempt to ensure that funded research ultimately makes its way into the hands of “end users.” However, we have identified a set of ways in which the current dissemination requirement does not actually function to ensure that funded research will be useful to education stakeholders.

As with the Significance section described above, the RFAs include relatively open-ended instructions with minimal guidance for the required dissemination plan, and no clear direction for how reviewers should evaluate the dissemination requirement. In the absence of such guidance, research teams and review panels may be applying idiosyncratic judgments of the kind of dissemination that is appropriate and effective. Committee members note from their own experience on reviews that panels vary greatly in how they approach this portion of the application.

Additionally, the committee notes that current framing of the Dissemination section suggests a largely unidirectional kind of engagement around research results: that is, researchers tell stakeholders about their findings, and then stakeholders use those findings. However, as described in Chapter 4, contemporary scholarship around knowledge mobilization problematizes this unidirectional assumption. Stakeholders need to engage with the research at multiple stages of the process to interpret, adapt, and apply it

in practice. Some projects with active partnerships may be well positioned for stakeholder engagement before, during, and after conducting the research. However, relying on a post-project dissemination plan alone may perpetuate inequities in access to relevant research. Further, the committee believes it is important to consider how dissemination and engagement extend beyond those who are immediately involved in the development and production of the research.

Role of Practitioners

Finally, the committee notes that the current application and review process does not have a consistent plan or procedure for engaging the education practice community. While some educators or policy makers may participate in the review process, the voices of practice stakeholders are not regularly integrated into review. Given the proximity of these professionals to the work of education, it is possible that the review process is missing a unique opportunity to ensure the application and review process yields useful research.

The issues highlighted above, taken together, point to a process wherein reviewers lack a clear north star by which to make calibrated judgments about what proposed research will be useful to stakeholders in the field, which can result in funded research that does not sufficiently meet the needs of education stakeholders and decision makers. We conclude this chapter with a set of recommendations for how IES might address this challenge.

RECOMMENDATIONS

In this chapter, we describe the elements and functions of each component of the application and review process. Given the role of the RFA as the primary mechanism through which NCER and NCSER signal their priorities to the field, the committee was particularly concerned with how to organize the review process. As noted in this chapter, the committee concluded that the RFA is well organized and purposeful, it is intentionally oriented toward providing applicants with an equitable experience, and its directions are clear and understandable.

Despite these successes, the committee did observe a few areas in which the current organization of the application and review process sets up a series of challenges: (1) IES does not publicly share information on its applicants, reviewers, and grantees, making it impossible to track whether the application and review process is resulting an equitable distribution of awards, and if not where in the process disparities are introduced; (2) the current procedures undermine IES’s ability to be timely and responsive to the needs of the educational research community; and (3) the current

procedures do not allow for sufficient understanding of how well-proposed research addresses the needs of the field. In this chapter, we described how these issues may inadvertently create barriers for NCER and NCSER in funding research that meets their stated goals. It is our sincere belief that with some modification to the process, IES will be even more successful in funding research that meets the needs of the field.

In regard to the first challenge noted above (i.e., lack of consistently reported data), the committee determined that given the centrality of equity issues to the mission and purpose of IES, it is critical that IES provide the field with transparent data on not only who is funded, but also who applies for funding and who is selected to review applications. Where this demographic data reveal inequitable inputs and outcomes into the review process, IES will want to craft immediate responses, but it is impossible to know what these problems are in the absence of a regular data report. For this reason, the committee recommends that IES takes immediate action related to the reporting of data.

RECOMMENDATION 8.1:

IES should regularly collect and publish information on the racial, ethnic, gender, disciplinary, and institutional backgrounds of applicants and funded principal investigators (PIs) and co-PIs, composition of review panels, and study samples.

Specific to the second issue noted in Chapter 8—timely and responsive application cycles—the committee found evidence that the current structure of a single annual review panel is not functional for the research community in education, and a September deadline for proposals is particularly problematic given the timing of the school year.

RECOMMENDATION 8.2:

IES should review and fund grants more quickly and re-introduce two application cycles per year.

The committee agrees that attending to the third challenge described in Chapter 8—ensuring that funded research is useful to the field—will require longer and more concerted effort. For a variety of reasons described in the chapter, reviewers in the current system do not have a way to calibrate their review of application materials toward any kind of shared understanding of what the field needs. It is therefore difficult to ensure that the work is relevant to policy and practice decision makers, which leads to funded research that meets the requirements of the RFA, but is not always aligned with the needs of education more broadly.

In general, the committee thinks that attending to the larger structural issues facing NCER and NCSER (see Recommendations 4.1 and 5.1–5.5) will serve to help ensure that funded research is ultimately positioned to be useful for the practitioners and policy makers. However, the effects of implementing these recommendations may take several years to emerge, and the committee notes that the field needs useful research as soon as possible. For this reason, we offer two recommendations that may help ameliorate some of the challenges related to usefulness that the committee laid out. First, in response to the current letter of support mechanism at work in the RFA, we considered how adjusting expectations around collaboration might better serve both researchers and involved communities. Below, we recommend an alternative approach to the letter of support that we believe will better map onto the current grantmaking timeframe, and also help better ensure that funded research is warranted in the community in which it is proposed.

RECOMMENDATION 8.3:

For proposals that include collaborating with local and state education agencies, the request for applications should require that applicants explain the rationale and preliminary plan for the collaboration in lieu of the current requirement for a letter of support. Upon notification of a successful award, grantees must then provide a comprehensive partnership engagement plan and letter(s) of support in order to receive funding.

The committee also noted the current lack of a consistent plan for engaging practitioner and policy-maker perspectives in the application and review process. The committee discussed multiple ways that IES might want to leverage these communities, ranging from consistent participation on panels to separate working groups, but notes that practitioner and policymaker communities should be involved in determining the mechanism that works best for IES. Ultimately, the committee agreed that each approach has trade-offs to consider regarding the burden placed on policy makers and practitioners, as well as the logistics of working with school schedules. Importantly, the goal of this work is for IES to define a role for these communities that is both distinct and meaningful, such that these already burdened professionals can maximize their valuable time and effort.

RECOMMENDATION 8.4:

IES should engage a working group representing the practitioner and policymaker communities along with members of the research community to develop realistic mechanisms for incorporating practitioner and policy-maker perspectives in the review process systematically across multiple panels.

The committee also discussed possible approaches to changing the review process to take on issues with rating described in this chapter. One idea was for IES to identify a person or entity to oversee and audit panel decision making. This person could, for example, review triaged proposals—ultimately pulling proposals out of triage when there appeared to be discrepancies or errors. They might also examine the final panel scores around the cut-point (2.0) and make substantive recommendations regarding priorities for funding. The committee, however, had difficulty determining who the right person for this position might be. Some thought that the program officer could take on this role; a problem, however, is that this changes the role of program officers, opening them up to have undue influence on funding decisions. Another idea was for the panel chair to take on this role; here the concern was that this would increase reviewer burden.

The committee also spoke about the problem of the cut-point at length. Some highlighted that another solution altogether was to shift from a known cut-point to a funding percentage instead, with such funding percentage cut-offs varying across panels. A benefit of this approach, some felt, was that it took into account differences in scoring across panels and did not allow for such clear gaming. Others worried, however, that there may be real differences across panels and that some panels may have stronger proposals than others, and that such a relative score would not be fair. Overall, while the committee declined to make a recommendation on the best approach to addressing these concerns, we agree that these problems require careful consideration in the future.

REFERENCES

Huutoniemi, K. (2012). Communicating and compromising on disciplinary expertise in the peer review of research proposals. Social Studies of Science, 42(6), 897–921. https://doi.org/10.1177/0306312712458478.

Institute of Education Sciences. (IES) (2022). Education Research Grants Program Request for Applications.

——— (2021a). Peer Review of Grant Applications. https://ies.ed.gov/director/sro/peer_review/application_review.asp.

——— (2021b). Updates on Research Center Efforts to Increase Diversity, Equity, Inclusion, and Accessibility. IES Blog, September 16. https://ies.ed.gov/blogs/research/post/updates-on-research-center-efforts-to-increase-diversity-equity-inclusion-and-accessibility.

Langfeldt, L., Nedeva, M., Sörlin, S., and Thomas, D.A. (2020). Co-existing notions of research quality: A framework to study context-specific understandings of good research. Minerva, 58, 115–137. https://doi.org/10.1007/s11024-019-09385-2.

Laudel, G. (2006). Conclave in the Tower of Babel: How peers review interdisciplinary research proposals. Research Evaluation 15 (1), 57–68.

Valantine, H.A., and Collins, F.S. (2015). National Institutes of Health addresses the science of diversity. Proceedings of the National Academy of Sciences of the United States of America, 112(40), 12240–12242. https://doi.org/10.1073/pnas.1515612112.

Zilberstein, S. (2021). National Academies of Science and Medicine: Diversity, equity, and inclusion in peer review. [Commissioned Paper].