4

Project Types for NCER/NCSER Grants

Grants funded by the National Center for Education Research (NCER) and National Center for Special Education Research (NCSER) use a structure of goals or project types to divide the studies “into stages for both theoretical and practical purposes” (IES RFA, 2018). Since 2002, five such project types have been funded: (1) Exploration, (2) Development and Innovation, (3) Initial Efficacy and Follow-up, (4) Scale-up/Effectiveness/Systematic Replication,¹ and (5) Measurement.² In this chapter, we focus on the first four of the project types; we address Measurement (along with Statistical and Research Methodology projects) in Chapter 6.

We begin our response to the question of new problems and issues that warrant Institute of Education Sciences (IES) research grant funding with a focus on project types for two reasons. First, these project types play an administrative role in IES, as different types of projects result in different request for applications (RFA) requirements and different budgets. Project types thus set the stage for the types of studies that IES would like to see conducted, including the purpose of each study. Second, these project types have from the outset played a normative role in education research, reflecting assumptions about the process through which interventions—programs, policies, and practices—ought to be developed and evaluated. For

___________________

¹ Note that in FY2020, 1–3 and 5 have been funded under the Education Research grants competition, whereas 4 is funded under a separate RFA.

² Whereas grants submitted to the main research funding competitions of NCER and NCSER enter with a specific project type, applications for Research Networks and Research & Development Centers typically encompass multiple project types.

example, these project types emphasize that randomized controlled trials offer the highest form of evidence regarding the effect of an intervention. Importantly, this normative role is what is often perceived by education researchers as being the core identity of IES.

Based on testimony from numerous speakers and our own analysis of grant patterns, the committee identified a fundamental mismatch between the presumed structure of scientific practice as expressed in the IES project structure and what is required to meet the needs of children, schools, and society. This is not to say that a scientific structure is not needed, but that such a structure should be based upon the realities and contexts found in education from early childhood to adulthood. Based upon this analysis, we articulate a new system of science that is distinctly aligned and attuned to education science. Corresponding to this system, we propose a new project type structure, and recommend its adoption by IES.

PROGRESSION ACROSS PROJECT TYPES

Prior to 2020, what are now referred to as “project types” were called “goals.”³ The numbering of these goals gave the appearance of a linear process, with a possible intervention moving from an idea (Goal 1) to a scalable intervention (Goal 4) that could then reach and impact student outcomes in schools across the nation. While no longer called “goals,” this same logic can be found in the descriptions of the project types. In the current system, Exploration projects focus on the identification of relationships between learner, educator, school, and policy-level characteristics and student outcomes; in particular, the focus is on identifying characteristics that can be changed via new interventions. Projects that might be funded in this type include small experiments testing if it is possible to change an observed factor, and the identification of associations between possible malleable factors and outcomes using both primary and secondary data. In Development and Innovation projects, an intervention is “developed,” resulting in a logic model, intervention components, and a pilot study in a handful of schools (or sites). This intervention is then evaluated in an Initial Efficacy project. This is an explanatory, proof-of-concept study, focused on establishing that the intervention can produce an effect under “ideal” conditions (i.e., when

___________________

³ While Exploration studies and Development and Innovation studies have remained roughly the same over time, project types (3) and (4) have been continually changed. Until 2018, studies of type (3) were called “Efficacy and Replication”; from 2019 onward these were renamed “Initial Efficacy and Follow-up.” Until 2012, studies of type (4) were called “Scale-up Evaluations,” then 2013 to 2018, they were called “Effectiveness,” in 2019, “Replication: Efficacy and Effectiveness,” and since 2020, this competition has been removed. In its place, “Systematic Replication” studies are now funded through a separate competition. See Brock and McLaughlin (2018) for more information.

implemented well). Finally, if an efficacy study suggests that an intervention has a positive impact, a Replication study may be conducted. In a replication study, the focus is on systematically changing one or more features of the intervention or context, to see if the previous efficacy findings are robust to this change. This replication study can itself be an efficacy study or an effectiveness study. In the latter case, the intervention is evaluated under routine conditions by an independent evaluator, with less researcher control and, likely, more variable implementation.

Notably, this last project type is where most changes have occurred in the past two decades. In the beginning, these fourth project types were referred to as “Scale-up” studies, with a focus on studying the intervention in a larger, broader, and more representative sample of schools. Later, these studies were renamed “Effectiveness” studies, with a focus on “typical” implementation and independent evaluation. This shift from “scale-up” to “effectiveness” on the one hand offered cost savings to IES (since “scale-up” studies were more expensive⁴), while on the other hand they deemphasized the need for interventions to be studied in more heterogeneous settings. At the same time, these changes to the fourth project type led to changes to the third. Initially these were referred to as “Efficacy” studies—which could include replications of previous efficacy studies. When the fourth project type shifted to “Replication,” this third type was thus repositioned as “Initial Efficacy” studies instead.

To better understand these project types, Klager and Tipton (2021) analyzed data made available on the IES website about funded grants. These data include grants funded since 2002 and categorize them by project types, as well as program, center, topic area, year, principal investigator (PI), and institution. Since the purpose of this analysis was to understand project types, these analyses included all grants, regardless of funding mechanism. For further details on data coding this analysis, please see Appendix D of this report.

Tables 4-1 and 4-2 provide the total number of all grants and the total dollars spent on grants by NCER (Table 4-1) and NCSER (Table 4-2). The top rows of these tables show the total number of grants awarded in each 5-year time period (with the exception of the last time period, 2017–2020, which only covers 4 years) and overall. The second row shows the total funding awarded in millions of dollars. The next three rows indicate the number of grants, funding, and proportion of the total funding that fall into the Exploration, Development & Innovation, Efficacy, and Replication/Effectiveness categories. The columns depict the proportion of funds distributed in

___________________

⁴ Many researchers now turn to Investing in Innovation (i3, first established with American Recovery and Reinvestment Act funds in 2009, and now called Education Innovation and Research, or EIR) for scaling studies instead.

TABLE 4-1 Proportion of Funding by Project Type and Year—NCER

SOURCE: Klager & Tipton, 2021 [Commissioned Paper]. Data from https://ies.ed.gov/funding/grantsearch/.

NOTE: Total grants includes all grant types, including Research Networks and R&D Centers. Project grants include those awarded in specific goals or project types.

TABLE 4-2 Proportion of Funding by Project Type and Year—NCSER

SOURCE: Klager & Tipton, 2021 [Commissioned Paper]. Data from https://ies.ed.gov/funding/grantsearch/.

NOTE: This analysis includes all grant types, including Research Networks and R&D Centers.

each time period to each grant category. Going across a row for each grant category shows how the proportion of funding awarded in each category has changed over time.

These tables indicate that over time, IES has focused an increasing proportion of its funding on these four project types—increasing from 57 percent to 71 percent of the total spending in NCER and from 48 percent to 85 percent in NCSER. Much of this increase can be attributed to the implementation of a more standardized goal structure over time. In the first time period (2002–2006), a large portion of the NCER/NCSER spending

went to studies that were “other goals,” “no goals,” or some combination of goals (e.g., “development and measurement”); for NCSER this included grants already encumbered by the Office of Special Education Programs. Over time the portion of these funds provided to each of the four project types has shifted. For example, both Exploration (5%–23% NCER; 1%–12% NCSER) and Efficacy (44%–54% NCER; 49%–56% NCSER) projects have increased in share over time, while Development (35%–18% NCER; 37%–31% NCSER) and Replication/Effectiveness (from 16%–6% NCER; 14%–2% NCSER) have decreased.

Examining Progression of Projects

The committee began by examining whether and how projects progress through and among the project types. To study this, Klager and Tipton (2021) examined the reporting of “related grants” in IES grant abstracts. For each grant, they examined whether there were later grants (of any type) identified as related that were funded. These results are shown in Table 4-3. Importantly, because these data do not indicate whether the “related” studies are of the exact same intervention or are only loosely related, these analyses may overestimate the amount of progression across projects.^5,6

The analyses presented in Table 4-3 indicate that within IES-funded studies, interventions are not moving across the project types in a connected way from Exploration to Replication/Effectiveness very often. This lack of connection is most prominent for Exploration grants, of which only 16 percent are connected to at least one later IES-funded study. Given the nature of exploratory work, it might be expected that a smaller percentage of these types of studies would progress. In comparison, 30 percent of Development and Innovation grants are later connected to other grants, including 20 percent associated with Efficacy grants and 4 percent with

___________________

⁵ There are no public data available that clearly identify progressions across project types by intervention. To approximate this, this table uses public data on “related grants” as a proxy. Parsing “initial efficacy” versus “replication” studies is also not definitive in these data, and instead the latter are identified by use of the word “replication” in the title or abstract. This results in 47 “Initial Efficacy” studies that are “related to” later “Initial Efficacy” studies, but that do not use the word “replication” in the title or abstract. A cursory read of these studies suggests that many are “related” in that they have the same PI or team members.

⁶ While it not possible to tell from these data how studies funded by other agencies (e.g., National Institutes of Health [NIH], National Science Foundation [NSF]) might precede or follow IES-funded studies, it is possible to make some inferences regarding how they might connect. For example, we know that NSF EHR also funds development studies, but that they less often fund efficacy or effectiveness studies. Similarly, we know that NIH (and the National Institute of Child Health and Human Development specifically) funds both the development of interventions and efficacy and effectiveness studies; however, these are focused on a small subset of the education space. Thus, it is more likely that development studies funded by these other agencies funnel into IES Efficacy studies than the reverse.

TABLE 4-3 Grants Related to Future Grants by Goal (2002–2017)

Table 7. Grants funded from 2002-2017 that are related to future grants by category

Note. Grants included in the “Grants from...” rows were funded bewteen 2002 and 2017. Grants included in the “Related to a future grant in...” were funded between 2002 and 2020.

SOURCE: Klager & Tipton, 2021 [Commissioned Paper]. Data from https://ies.ed.gov/funding/grantsearch/.

NOTE: Relationship determined from language in abstracts of grants. If the originating study resulted in multiple later studies of the same project type, it is only counted once in this table. However, if the originating study resulted in later studies of different project types, it is counted in both; thus the rows do not sum to the “total.” The bolded diagonal numbers indicate multiple grants of the same type related to one another. Those above the bolded diagonal numbers indicate goals that progressed forward (e.g., D&I to Efficacy), while those below the diagonal progressed backward (e.g., Efficacy to D&I).

Replication/Effectiveness grants. (Since a grant many be associated with more than one subsequent grant, some of these may be duplicates.) Perhaps surprisingly, only 9 percent of Initial Efficacy studies are associated with later Replication/Effectiveness grants, while 6 percent are associated with additional Efficacy grants, and another 6 percent with new Development and Innovation grants. Notably, most grants are not associated with any future grants at all.

As early as 2013, in collaboration with NSF, IES noted few interventions were moving across these goals in a direct path. As the IES-NSF Common Evidence Guidelines (p. 10) state, “Knowledge development is not linear. The current of understanding does not flow only in one direction (that is, from basic research to studies of effectiveness). Rather, research generates important feedback loops, with each type of research potentially contributing to an evidence base that can inform and provide justification for other types of research.”

Later analyses by Albro and Buckley (2015) supported the highly nonlinear and iterative process through which research moved across and within the pipeline. This observation contributed to the decision to change the names from “goals” to “project types” and remove the numbering. The data in Table 4-3 speak to the iterative, nonlinear nature of the development process; for example, many Development and Innovation grants are followed up with new Development and Innovation grants, and Efficacy

studies are sometimes also followed up with Development and Innovation grants.

Understanding Project Progression

The committee considered several potential reasons for the lack of consistent progression across project types (Farley-Ripple et al., 2018; Farrell & Coburn, 2017; Greenhalgh et al., 2004). The first is the lack of connections between different researchers or research teams. Nearly 84 percent of Exploration grants and 70 percent of Development and Innovation grants are not associated with any later grants. Furthermore, only 24 percent of the Development and Innovation grants were associated with later Efficacy, Effectiveness, or Replication grants. While one interpretation of this could be that the interventions explored and developed simply did not achieve desired ends, another possible explanation is a hand-off problem between project types that leaves promising interventions slipping through the cracks.

This hand-off problem is not hard to imagine, given that a given researcher may be more likely to possess skills and interests that fit into one (or maybe two) project types. For example, relative to those at research firms, university researchers are far more likely to be involved in Development and Innovation studies than Efficacy studies (Klager & Tipton, 2021). This makes sense in many regards, given the complexities of running randomized trials in large, sometimes geographically dispersed sets of schools. But unlike other fields, such as the pharmaceutical industry, researchers who undertake Development and those who undertake Efficacy studies are found in disparate organizations, with limited opportunities for natural connection across skills and interests. Thus, it is possible that one reason some interventions do or do not move from Development and Innovation to Efficacy has less to do with promise and more to do with connections researchers do or do not have across skill sets and organizations.

A second potential reason for lack of progression is related to implementation concerns. How an intervention works in a local context is directly related to how well it can be implemented given the culture, constraints, and resources found in its classrooms, teachers, schools, and districts (Century & Cassata, 2016; Bauer et al., 2015). As a result, most interventions are ultimately adapted to local environments, and some of these adaptations are likely better than others. Highly scripted, packaged programs provide a means to control implementation—which is ideal for teasing apart causality—but these can lead to an entire intervention being discarded when it does not fit well into the school environment. This creates an inherent tension between implementation and usefulness. The interventions most implementable with fidelity are heavily scripted and require

specific supports, yet these requirements may not be feasible or desirable in many school environments (Coburn, 2003).

As a result, implementation plays out differently in Development and Innovation grants versus Efficacy and beyond studies. In Development and Innovation studies, implementation can be more tightly controlled: through the selection of schools and teachers into the study, through the small sample size, and through close monitoring by researchers. This degree of selection and monitoring simply cannot continue, however, as sample sizes grow larger in Efficacy studies. Thus, even though Efficacy studies often seek to focus on “high implementation” conditions, these goals are not always achieved.

A third possible reason for lack of progression relates to heterogeneity. The fact that students are inherently nested in classrooms, schools, and school systems is important since classrooms, teachers, and schools vary considerably in a myriad of ways. These factors include student backgrounds and baseline knowledge, classroom composition, teacher characteristics and behaviors, and school policies, practices, and resources (Weiss, Bloom, & Brock, 2014). Indeed, numerous studies in education research show that teachers’ experiences of teaching and students’ experiences of learning vary considerably across contexts (Nasir et al., 2016).

Furthermore, studies show that treatment effects can vary across these school and contextual factors (e.g., see Weiss et al., 2017). One of the most obvious sources of variation in effects, both across and within studies, has to do with what practice would have been absent the intervention. That is, the comparison condition (“business as usual”) in causal studies is rarely doing nothing; rather, these studies are often teaching the same subjects and skills but using different curricula or approaches. It is easy to see, then, that these business-as-usual practices might vary, and as a result, so do impacts of an intervention.

The committee recognizes that NCER and NCSER have been aware of and actively sought to address many of these concerns. Efforts to respond to these challenges are reflected in additional recommendations included in the RFAs. For example, the RFAs include a list of possibilities that could be included for a “strong proposal” in addition to the primary focus on estimation of the average treatment effect. The 2022 RFA for Education Research Grants, for example, recommends that researchers proposing Efficacy and Replication studies “describe the setting and implementation conditions,” assess “fidelity of implementation and comparison group practice,” collect implementation outcomes, and describe a plan for examining fidelity of implementation.

Similarly, the RFA requires proposals to describe their sample and recommends that they define a target population. Proposals are encouraged to develop a plan to represent this target population during recruitment and

to address concerns with the generalizability of their sample. “[A]lthough not required,” the RFA suggests “the analysis of factors that influence the relationship between the intervention and learner outcomes (mediators and moderators)” and notes that when these are included, power analyses for related hypothesis tests should be included.

Regarding the concerns about connecting points between different project types, however, there has been considerably less work. The committee understood that in the original conception of the goal structure, the same researcher or team might develop an intervention over a sequence of studies from Exploration to Effectiveness.⁷ In this conception, no connecting points were needed since the same team carried the idea through to completion. Over time, however, it became clear that this model was rarely followed in practice, and that the research process was iterative. But removing the numbering and changing the names did nothing to address this connecting point problem—if different researchers conduct different types of studies, how should interventions be moved along in the system?

Perhaps the closest IES has come to addressing this concern is again through the “dissemination plan” found in the RFA. For example, for Development and Innovation grants, proposals must include dissemination plans that focus on “letting others know about the availability of the new intervention for more rigorous evaluation and further adaptation.” This includes activities like journal publications, presentations, and engagement with research networks. Importantly, these activities are focused on individual researchers and their own networks and interests since no repository or database of preliminary findings exists within IES.

CONNECTING RESEARCH AND PRACTICE

The model of education improvement that IES research is built upon assumes that interventions are developed, tested, refined, and tested some more and then ultimately the successful interventions are adopted by school districts, schools, and teachers—thus ultimately improving student learning and reducing disparities at a national scale. Information about successful interventions is made accessible to decision makers through a variety of dissemination activities and especially through the What Works Clearinghouse (WWC), which is housed in the National Center for Educational Evaluation and Regional Assistance. To this end, the WWC developed and maintains a database of intervention effects that can be accessed online, as well as practice guides. The WWC also develops and maintains a Standards Handbook that provides rules regarding different designations that studies

___________________

⁷ This sentence was modified after release of the report to IES to indicate that this statement is part of the committee’s judgment.

can receive. As a result, most Efficacy, Effectiveness, and Replication grants at NCER and NCSER strive to meet these standards so that their results can ultimately contribute to this knowledge base and make it to schools and decision makers.

A continued question at IES is whether the research produced by IES grants is used in schools and educational contexts for decision making. To date, however, it has been difficult to answer this question since there are little data available to understand the outputs of the IES research system. For example, simple online searches of interventions studied via IES grants often result in project or intervention websites, but without any clear indicator of how often the program is adopted. Research on knowledge mobilization suggests that only 17 percent of school and district leaders report accessing research from the WWC “often” (13%) or “all the time” (4%) (Penuel et al., 2017). Yet at the same time, some individual researchers have made considerable headway in getting their curricula and/or interventions into classrooms.

As noted in Chapter 1, however, there is now a deeper understanding of how educators and education decision makers access and use research evidence to inform practice and policy. These insights suggest that traditional models of dissemination are insufficient for connecting the education policy and practice communities with the evidence produced by research. The complexity of decision making in education raises several possible explanations for why a given intervention may not be identified or adopted by education decision makers even when it is accessible through the WWC.

First, decision makers may be facing problems other than those being addressed by IES-funded educational interventions. Adoption decisions are one of many decisions that educational leaders make, and some research suggests that these decisions are relatively rare (Penuel et al., 2018; Coburn, Toure, & Yamashita, 2009). Put another way, researcher foci and school and district needs may be out of sync. This disconnect could occur because the current project structure does not prioritize understanding and connecting with decision makers regarding their needs and current practice, or because IES’s preferred study types and methods are not well suited to investigating the issues that schools and districts are facing. There may be whole classes of interventions or approaches that are not being studied by IES-funded research but that are of interest to education organizations. These might include issues that are difficult to study using randomized controlled trials (e.g., student assignment algorithms, school funding approaches, teacher hiring practices, course de-tracking policies, or meal subsidy policies) or that are not directly focused on student achievement (e.g., approaches to school discipline, social-emotional learning, or school-community collaborations related to health, safety, and wellness).

A second possible explanation is that even if an intervention or approach addresses a “real” problem faced by teachers, schools, and school districts and is potentially implementable, it may not be available for schools and school districts to use. That is, the intervention may not be marketed and distributed in the same way as commercial curricula, leaving most schools and districts unaware that the intervention even exists. Or it may be that even if marketed and available, it is not packaged and supported in the same way that other curricula and programs are. This suggests that there may be a need for approaches that bring together and incentivize partnerships between researchers, communities, education technology companies, publishers, and nonprofits that focus on selling curricula and professional development to schools. Through these partnerships, researchers can take advantage of the scale and reach of these organizations, thus getting “best practices” out to schools more efficiently.

Third, interventions developed by researchers may not be readily adaptable, implementable, and sustainable in schools and districts: that is, the results are not useful because they are difficult to use. For example, a brief, 9-week science program may be effective and yet, if compared to a full-year science program, may be difficult to implement (since curricula for the remaining weeks of the year then need to be selected, too). Other barriers may arise around training personnel, freeing up staff time from competing demands, or aligning programs to related initiatives.

Furthermore, in many instances, schools and districts may not be interested in packaged programs as much as developing their own, locale-specific programs based upon best practices found in research. This suggests a need also for research evaluating approaches developed by practitioners, strategies for developing locale-specific interventions (e.g., for districts), and identification of core components of interventions.

Again, the concerns the committee raised above are not unfamiliar to IES. Indeed, over time IES has responded to concerns of this type via various revisions to both the WWC and to RFAs and requirements for grants. Perhaps one of the most concerted efforts can be found in the requirements for dissemination plans for Efficacy and Replication grant proposals. The RFA notes that “IES considers all types of findings from these projects be potentially useful to researchers, policymakers, and practitioners…” and that researchers who create interventions “are expected to make these products available for research purposes or … for general use.” In practice, these efforts often include workshops or trainings for the districts and schools involved in the study, for other districts or schools, or the development of a website for the intervention. The success of these plans, however, is difficult to monitor since they occur in the last year of the grant funding for a study.

More broadly, though, research shows that dissemination by itself is

not sufficient for enacting research in practice (Rabin & Brownson, 2012; Greenhalgh et al., 2005). Decision making occurs through relational dynamics within larger systems (Best & Holmes, 2010; Boswell & Smith, 2007), where policy makers engage with multiple actors around multiple forms of knowledge (Farrell & Coburn, 2017; Greenhalgh et al., 2004). Furthermore, packaging, scaling, and marketing interventions are far beyond the skill set of most academic researchers. To have substantive impact, dissemination and engagement activities require time and resources that go beyond what can be conducted in the last year of a grant, as educators also need continued support beyond the adoption decision to train, implement, and adapt new practices within their local contexts (Dearing & Kee, 2012). While the challenges are clear, much remains to be learned about robust strategies for ensuring that findings from education research reverberate in the decisions of educational leaders and practitioners (Conaway, 2021).

This speaks to a need to better understand knowledge mobilization, including how schools and decision makers identify problems and develop solutions; which interventions, curricula, and programs are currently used in schools; how to get promising evidence into their hands; how education leaders harness that evidence to guide action; and what conditions support education leaders to use research more centrally and substantively in their decision making (Farley-Ripple et al., 2018). Improving understanding of the processes of knowledge mobilization would help develop better mechanisms for determining what research would be useful for education policy makers and practitioners, as well as identifying strategies for supporting them in using that research when it is available (Jackson, 2021).

A REVISED SCIENTIFIC STRUCTURE IS NEEDED

As the committee showed in the previous section, the existing IES project structure has encountered and addressed a variety of problems over the previous 20 years. These include problems moving interventions from Exploration to Efficacy and from Efficacy to scale and practice. In the face of each of these concerns, IES has reflected on and acknowledged these shortcomings, each time attempting to address concerns with new additions, including new names, new requirements, and new trainings. But at the core, this project structure has remained the same.

This project structure was developed around assumptions that seemed reasonable 20 years ago: that the challenges facing schools could be addressed by developing and testing interventions that could be easily packaged (and thus randomized) and that would uniformly increase student achievement. Twenty years into this science, however, it is clear that this model does not map onto the reality of education science and U.S. schools,

that changing practice is harder than simply providing evidence, and that changing school environments and reducing inequity is difficult work.

For these reasons, the committee argues that now is the time for IES to rejuvenate and revise its project type structure. Unlike the previous project structure, which followed a pattern that is familiar from other scientific fields such as biomedical research, the new structure is grounded in the specific challenges of education today, and thus is uniquely designed to support a robust and cumulative science of education. To develop this new project structure, we begin with the charge, often repeated in RFAs and reports that the goal of research supported by IES is to determine “what works, for whom, and under what conditions.” This framing puts the users of evidence at the center: the school districts, schools, teachers, and students. This charge is echoed in the mission statement for IES:

As we articulated in previous chapters, an overarching goal of this science of education should be to reduce inequities in schooling and society. From the No Child Left Behind Act, to the Every Student Succeeds Act, to President Biden’s Executive Order on Racial Equity, a bipartisan sequence of Presidents and other policy makers has placed equity at the heart of the national goals for education. It is time for the research enterprise to do the same (Jackson, 2021; Farrell et al., 2021).

Drawing on both the five themes introduced in Chapter 1, and on the previous analysis of how and why interventions may or may not be progressing through the existing project types, the committee identified a set of framing principles for education research that inform the revised project structure.

1. One of the major purposes of education research is to identify and intervene on inequities in schools and society. This purpose pushes beyond understanding what works simply for the sake of science toward identifying the most promising ways to improve schools. It targets the nation’s greatest educational challenge: to eliminate pervasive and persisting disparities among groups such as those defined by race, ethnicity, gender, income, disability, and language minority status, as called for in repeated enactments of the Elementary and Secondary Education Act, the Higher Education Act, the Education Sciences Reform Act, and President Biden’s Executive Order.
2. The effects of interventions will vary given the complexities of school contexts, cultures, resources, learners, and existing practices (Bryan, Tipton,

2. & Yeager, 2021; Joyce & Cartwright, 2010). Contextual conditions, such as the social, economic, political, and resource structures in which education operates, shape the needs of actors in the education system, the feasibility of implementation, and the effects of interventions. Greater attention to contextual differences is also essential to make progress toward advancing equity through education research. Decision makers are rarely interested in the average impact of an intervention; instead, they want to understand the projected effect in their local context, often for a specific student population. This suggests that the primary focus on “the effect” of an intervention—at any stage of research—is likely inappropriate.
3. Interventions will be adapted differently in different environments, thus contributing to the heterogeneity of effects. This implies that it is important to both develop and evaluate interventions in the realistic conditions found in schools and school systems. Given concerns with implementation, adaptation is an inherent part of the adoption of new interventions in schools. For this reason, decision makers need information regarding which adaptations are responsive versus unresponsive to local contexts, which barriers and facilitators may affect implementation, and which supports are needed (McLeod et al., 2017; Abry, Hulleman, & Rimm-Kaufman, 2015; Nilsen, 2015; Powell et al., 2015; Waltz et al., 2014; Michie, van Stralen, & West, 2011; Damschroder et al., 2009).
4. Decision makers obtain information on educational interventions from a variety of sources. Decision makers are inundated with potential interventions and professional development services, in addition to frequently adapting and creating their own, and would benefit from guidance on how to efficiently surface and weigh evidence to compare different options. Altogether, this speaks to a need to better understand knowledge mobilization, including how schools and decision makers identify problems and develop solutions; which interventions, curricula, and programs are currently used in schools; how to get promising evidence into their hands; how educational leaders harness that evidence to guide action; and what conditions support educational leaders to use research more centrally and substantively in their decision making. Improving understanding of the processes of knowledge mobilization would help develop better mechanisms for determining what research would be useful for education policy makers and practitioners, as well as identifying strategies for supporting them in using that research when it is available.
5. The most promising interventions will not necessarily find their way through the research structure and into educational settings. Infrastructure is needed to both support research (e.g., to disseminate knowledge across project types, to surface promising interventions, to encourage evaluations of these interventions) and to connect researchers with users (e.g., to develop networks, identify knowledge brokers). There are many potential

1. forms for this infrastructure, but at the core, they need to be about building systems to integrate research with practice.

Beginning with these principles makes clear that issues of equity, implementation, heterogeneity, and usefulness need to be addressed from the very beginning of the research and development process, not at the end. The research process needs to begin in the field—in schools and other educational settings—and should involve exploring what current constraints, resources, and needs teachers, schools, and school systems face; the range of practices and policies they have already been developing and exploring; and the variety of contexts found in schools nationwide. The development of new interventions, including policies, practices, supports, and organizational approaches, needs to, from the beginning, account for issues of adaptation, implementation, and heterogeneity that arise in this diversity of contexts, when researchers are not nearby. Studies evaluating these interventions need to focus not only on estimating the average effect, but also on understanding variation in effects, and helping to guide decision makers where, under what conditions, and for whom such an intervention may be promising.

Finally, infrastructure is needed that continually synthesizes and updates what is known—for each project type—and uses this infrastructure to connect with and direct research in other project types. Importantly, this means systematic reviews of not only efficacy studies and those that meet WWC standards intended for decision makers, but also of exploration and development studies intended for researchers. This interstitial work might surface, for example, promising interventions that have been developed and that need to be evaluated. This could be the work of IES directly or commissioned to be carried out by others. These syntheses—and the ability to understand gaps—are essential to creating the feedback loops necessary to move the field forward.

In more practical terms, this means revising the underlying project structure. Like the current structure, we envision four project types, each of which can be crossed with a topic area. However, these project types would differ in focus and content from their current versions. Importantly, these changes should not be seen as in opposition to the current structure so much as an outgrowth and evolution of this structure—and of the knowledge we, as a field, have accumulated over the past two decades. The committee notes that these proposed project types would encompass research that currently exists in the IES grant system but would expand beyond and address some of the limitations, thus making space for new research. (A fifth project type, for measurement studies, is discussed in Chapter 6.) These project types pertain to both NCER and NCSER.

1. Discovery and Needs Assessment

Current: In the original goal structure, the intervention pipeline was assumed to begin in the research “lab.” This meant early studies would focus on identifying “malleable factors” associated with educational outcomes (IES RFA, 2019). Many current Exploration studies continue to focus on establishing the relationships between pre-determined “malleable” factors and pre-determined “outcomes.” In this way, Exploration studies are often less “exploratory” and more “confirmatory” in nature, focused on determining if theories developed in the laboratory can be confirmed to hold in schools.

Over time, Exploration studies have expanded to include a broader range of study types. As Table 4-4 indicates, more than one-third (35% NCER; 13% NCSER) of these studies have been focused on questions of causality, with some addressed via strong quasi-experimental methods (11%; e.g., regression discontinuity designs) and others using small experiments (22%). Here is it notable that studies focused on causal questions—answered with quasi-experimental methods and secondary data—are considered “exploratory.” Calling these exploratory indicates that even findings from high-quality quasi-experiments are not to be taken as serious evidence.

Finally, to date 6 percent (4% NCER; 7% NCSER) of these Exploration studies have been systematic reviews and meta-analyses. In examining the abstracts of these reviews, nearly all of them focus on synthesizing the results of randomized trials and high-quality quasi-experiments. Yet in the current structure, while they summarize a broad base of causal research, the research itself is considered “exploratory.”

New: If, as a field, we are to develop and refine interventions that can successfully improve educational outcomes for students, then it is imperative that these interventions consider the diversity of the educational

TABLE 4-4 Current Exploration Study Types

Note. When grants included multiple data sources and/or studies, primary data collection supercedes secondary data collection. Likewise, meta-analysis supercedes experiments which supercede quasi-experiments.

SOURCE: Klager & Tipton, 2021 [Commissioned Paper]. Data from https://ies.ed.gov/funding/grantsearch/.

contexts found in this nation. For this reason, the committee proposes that schools, districts, and out-of-school learning spaces should actively be the breeding ground of scientific theories themselves. To highlight this anchoring in educational context, we call this new project type Discovery and Needs Assessment. These studies would begin in authentic learning environments, with a focus on observing, measuring, and understanding the varieties of practices and processes on the ground and determining gaps between “what is” and “what could be.”

By emphasizing the need for situating work in authentic school environments, we are also highlighting the need for a broad range of descriptive work that involves primary data collection. Qualitative data might include deep descriptions of processes and problems, identifying the ways in which these processes and problems might contribute to persistent inequality and surfacing potential barriers, facilitators, and implementation strategies. Quantitative data might include descriptive or correlational analysis of surveys of current practices—including curricula used and time allocations—as well as the problems faced by teachers, schools, and school systems. Of course, this is also a place in which new sources of data, such as big data and administrative data, could be examined to better describe educational contexts and trajectories.

The language of Needs Assessment makes clear that the factors under study would shift from a primary focus on students to also consider classrooms, teachers, schools, and systems. This means shifting toward landscape analyses and diagnostic work, with the goal of understanding the social, economic, political, and resource structures in which schools operate; current business-as-usual practices; considerations for implementation of new practices; and possibilities and levers available for intervening. Importantly, this also means soliciting and understanding the problems education organizations care about and are searching to solve. This is not to say that this research needs to only respond to the immediate, stated concerns of decision makers; certainly science can have a longer and broader vision of what is possible than what is immediate. But without this information—without understanding the needs of the actors involved in the system—moving toward this broader vision is not possible.

This project type would allow IES to continue its current stance to “encourage, support, and prioritize collaboration between researchers and practitioners, but without specifying how that cooperation should be structured” (Schneider, 2020). However, the research literature suggests that research-practice partnerships, which are increasingly found in large school districts across the country as well as many states and regional

collaborations,⁸ would provide an especially hospitable context for discovery and needs assessment research. As Turley and Stevens (2015) explained, by jointly developing a research agenda, researchers are more likely to ask questions whose answers matter to educational decision makers. The partnership also enables researchers to understand the context more deeply, and to interpret their findings in light of local conditions. Meanwhile, educators benefit from the chance to have their questions addressed in the most pertinent context of all, their own district, state, or region. Both needs assessment and discovery of responses to those needs may be enhanced when undertaken in the context of a sustained partnership that embodies trust, a diverse range of expertise, and opportunities for many actors to have a voice in the questions pursued and the interpretation of findings (Farrell et al., 2021).

This new framing of Discovery and Needs Assessment studies is responsive to the general mission of IES as laid out in the Education Sciences Reform Act (ESRA), Section 111(b)(1): “to provide national leadership in expanding fundamental knowledge and understanding of education from early childhood through postsecondary study, in order to provide parents, educators, students, researchers, policymakers, and the general public with reliable information” about the condition of education; practices that support access, learning, and achievement; and program effectiveness. By beginning in the field instead of in the laboratory, IES-sponsored research will be better positioned to uncover new knowledge and understanding that responds to the need to improve outcomes and advance equity in U.S. education.

2. Development and Adaptation

Current: In the current project structure, Development and Innovation grants focus on iteratively developing or refining new interventions for use in schools. These studies are encouraged to identify how their innovation differs from current practice, how much such an intervention would cost, and how the sample of schools in which it is piloted represent a (narrow) target population that might use the intervention (IES RFA, 2022). Here we focus on two issues.

Pilot studies often include only a small sample of schools (e.g., less than 10) and involve a high degree of researcher control. This means that the intervention is developed in an optimal condition: that is, one where implementation is highly monitored so that high fidelity is achieved. In the short run, this focus on optimal implementation is ideal, as it allows for

___________________

⁸ The National Network of Education Research–Practice Partnerships now includes 57 members across the United States; see https://nnerpp.rice.edu/.

the causal effect to be isolated in the best-case scenario. In the long run, this can be far from optimal, as it means that problems of implementation and adaptation are not discovered until later studies with larger samples (Farley-Ripple et al., 2018; Finnigan & Daly, 2014).

The current project structure also has not sufficiently addressed heterogeneity. Development studies are encouraged to identify a target population that is “narrow,” and in practice—given the smaller resources found in these grants relative to Efficacy studies—this can mean a population that is local to the researcher and somewhat homogenous. Again, in the short run, this may be optimal, as reduced variation can improve statistical precision. But this pushes questions of heterogeneity to later studies. To see why, note that “narrow” target populations may not represent well schools elsewhere in terms of resources, practices, organizational conditions, students, or business-as-usual curricula and teaching practices. As a result, the intervention has only been developed and refined in one, particular population, delaying questions of contextual variation and the feasibility and fit elsewhere until later studies.

New: In contrast, we propose that researchers need to—from the outset—consider the types of heterogeneous environments that an intervention may be implemented within and focus on determining barriers and facilitators (Tabak et al., 2012) and effective implementation strategies. We include the word Adaptation in the title to clarify that adaptations to local contexts will occur and that it is incumbent on researchers to develop their interventions with this in mind. The language of adaptation recognizes that by adapting to local conditions, an intervention may be more likely to be adopted, supported, and sustained, thus improving educational outcomes. As Joyce and Cartwright (2020, pp. 1048–1049) explained, research on local conditions that support or impede program success requires information that goes well beyond impact assessment.

The kinds of research that can produce the requisite information, locally or more generally, often in coproduction, require a mix of methods well beyond those listed in current evidence hierarchies. The standard reasons for mixing methods in evidence-based education are to aid implementation (Gorard, See, & Siddiqui, 2017) and to make general effectiveness claims more reliable (Connolly et al., 2017; Bryk, 2015). We, by contrast, encourage mixed methods because reliable and useful effectiveness predictions require a variety of different kinds of information relevant to determining how an intervention will perform in a specific setting that different kinds of research help uncover. These different modes of research allow the development of interventions that not only work in one, narrow population, but that are robust and potentially effective in a broad range of school contexts.

Importantly, while planning for adaptation does require greater heterogeneity in the samples included in pilot studies, it does not necessarily mean

that larger samples are required. For example, it is possible to include a small sample that is even more heterogeneous than a population simply by carefully and purposively including schools that differ in a myriad of ways during the design process (see Tipton, 2021). Collecting both quantitative and qualitative data would allow information on supports and derailers to be studied, and for the intervention itself to be robustly developed quickly.

By positioning adaptation and heterogeneity as central to the development of interventions, we also highlight the potential for new approaches to intervention development entirely. Often, development is a researcher-led activity, one in which a person from the outside brings in new ideas and approaches. But if the goal is for an intervention to be implementable and adaptable, it may mean that starting with existing practices and programs and refining these to be more evidence based may ultimately be more scalable. This framing, too, allows for consideration of who designs, for what purposes, and how design will take place (Philip et al., 2018; Bang & Vossoughi, 2016).

Finally, taken together, this call for focusing on adaptation and heterogeneity in design has important consequences for the goals and purpose of the pilot study in Development and Adaptation studies. Certainly, this increased variation will make it more difficult to estimate a statistically significant average treatment effect in a pilot study. But many scholars in both medicine and education research have already argued that the focus of pilot studies should not be on estimation of the average treatment effect or on testing null hypotheses (e.g., see Westlund & Stuart, 2017). Instead, these studies should be focused on the preparations needed to ensure success in later efficacy studies—and being able to anticipate adaptations across a wide range of contexts is essential to this work. This means that requirements for later efficacy, effectiveness, and replication studies would not be focused primarily on an estimated effect size or hypothesis test, but should consider them in tandem with the logic model for the intervention, proximal measures, and the ability to implement and adapt to a range of contexts.

This new conception of Development and Adaptation studies is no less responsive than current practice to ESRA’s call for “scientifically valid research activities, including basic and applied research, statistics activities, scientifically valid education evaluation, development, and widespread dissemination” (Section 112(1)). Indeed, if it leads to greater identification of programs and practices that, because they are responsive to real needs and attend to implementation challenges, can actually be implemented and are in fact implemented, the new conception will meet the law’s requirements with even greater force than the present system.

3. Impact and Heterogeneity

Current: In the current goal structure, interventions are tested first in “some” population (often considered “ideal,” often “narrow”) via an “initial” efficacy study, later in another context (or version of the intervention) via an efficacy “replication” study, and, rarely, in a broad population, via an “effectiveness” replication study. In the best-case scenario, this results in over a decade of evaluation before the intervention is considered ready for marketing to schools. By this time, both current practices in schools and the intervention itself may have shifted, making the direct evidence from these studies out of date. And in the interim, the average effect from each of these studies may still be considered evidence for school decision making (e.g., via the WWC), albeit based on evidence from a small fraction of school environments that may not at all represent the schools that might benefit from the intervention.

In the current framework, the focus is on ensuring high internal validity—the ability to detect cause-and-effect relationships—at the expense of external validity. This can be seen in the fact that considerations of heterogeneity and implementation are pushed later and later in the process. Initial efficacy trials are not required to focus on either, leaving these for replication grants (efficacy or effectiveness), which are rarely conducted. Studies of existing research practices indicate that the samples included in efficacy studies have not historically represented either national or state populations of schools well and are vastly less heterogeneous than these populations (Tipton, 2021; Tipton, et al., 2021; Stuart et al., 2017). Thus, understanding heterogeneity is saved for replication studies, which are to systematically vary “at least one aspect” of a prior study, in order to determine “the conditions under which [interventions] work and for whom” (IES RFA, 2021). Importantly, these aspects could include the version of the intervention itself (a “conceptual” replication), thus not necessarily addressing the heterogeneity found in business-as-usual conditions across education contexts.

This prioritization of internal validity can also be seen in the predominance of randomized trials in both efficacy and effectiveness studies.⁹ The fact that randomization to treatment provides clear identification of a cause-and-effect relationship is not to be disputed. But not all interventions can feasibly be randomized, particularly those involving school and system interventions, both because of the large commitment required by schools and budget limitations. Even when randomization is possible, attrition before outcomes are measured, particularly in long-term interventions, can

___________________

⁹ This section has been modified after release of the report to IES to clarify research design types permitted for Efficacy, Effectiveness, and Replication projects, as well as for Exploration projects.

undermine the benefits of randomization. Thus, it is not surprising that the interventions studied via Initial Efficacy and Replication studies largely focus on student-level interventions, while teacher, school, and system interventions are largely found in quasi-experiments.

Finally, while IES focuses strongly on internal validity, new research suggests that policy makers appear mainly concerned with external validity when accessing and using research. This point is driven home by a recent experimental study that found that while policy makers do not exhibit a preference for experimental over observational findings, they are more likely to be drawn to evidence from larger studies and from contexts like their own than to smaller studies and contexts that differ from their own settings (Nakajima, 2021). Likewise, a recent national survey of education leaders found that when asked to identify specific examples of research that informed their practice, respondents most commonly pointed to books covering broad topics, and they rarely identified research studies that would meet the top tier of evidence in the Every Student Succeeds Act (Farrell, Penuel, & Davidson, 2022). These studies point to a disconnect between the priorities of researchers and decision makers when considering what makes evidence useful for practice.

New: In contrast, we call for combining all studies focused on determining causal effects of interventions into a single project type referred to as Impact and Heterogeneity studies. This includes quasi-experimental studies (currently most commonly funded via Exploration studies), as well as all types of efficacy and effectiveness studies. Combining these type of studies under a single project highlights that the question and purpose is the same in each—to estimate causal effects—while allowing variation in the approaches used depending upon the population, context, and intervention. Furthermore, a single project type for causal questions, which includes both efficacy and effectiveness studies and addresses heterogeneity as well as the average impact, will elevate matters of external validity to be considered on par with the matters of internal validity.

The inclusion of quasi-experiments in this category is of particular importance, as some interventions may simply not be able to be studied using randomized trials. For example, we know that it is difficult to recruit schools into trials, in general, and particularly in trials with intensive interventions. Interventions focused on changing school policies, leadership, and structures may be particularly difficult to randomize. This means that focusing only on the inclusion of randomized trials as “evidence” of a causal impact severely narrows the types of interventions that can be studied and evaluated. Here, we are calling to elevate the ability to use high-quality quasi-experimental designs when conducting a randomized trial would be infeasible. High-quality quasi-experimental designs might include regression discontinuity, instrumental variables, (comparative) difference-in-difference,

and propensity score methods. (Importantly, studies of this type would not be possible without the methodological developments for quasi-experiments funded by IES Statistical and Research Methods grants over the past two decades.) By elevating these methodologies, the focus becomes clearly on determining the best evidence for the interventions that schools need, instead of on finding interventions that fit the best methods of evidence.

Within randomized trials, combining efficacy and effectiveness studies into a single project type also removes what can be arbitrary distinctions between the two. In medicine, it has long been noted, for example, that very few studies are fully either efficacy or effectiveness trials, with most operating on a continuum between these (e.g., the PRECIS-2 tool; see Loudon et al., 2017). Our assessment as a committee is that in education, it is hard if not impossible to fully “control” the school environment in ways that are typical in efficacy trials in medicine. For example, in education, the comparison condition is very often a business-as-usual condition (effectiveness language) instead of a researcher-determined comparison (efficacy language). Similarly, even when the intervention is highly scripted, it is often very difficult to highly control how it is both delivered and how well its implementation matches what is intended (efficacy language); instead, very often the intervention is adapted to local conditions (effectiveness language). Similarly, the line between a “conceptual replication” study—in which an intervention that exists is changed systematically in some way—and a new “efficacy” study—based on a “new” intervention—can also be an arbitrary distinction. The point here is that this is not an either-or, that most studies fall on the spectrum of efficacy-effectiveness, and that the current language reifies a distinction that is often false.

Furthermore, combining Initial Efficacy and Replication studies (of both types) into a single project type makes clear that heterogeneity and implementation should be front and center in any impact analysis (Bryan, Tipton, & Yeager, 2021). For some interventions, this may mean condensing what would have been several studies—Initial efficacy, Efficacy replication, another Efficacy replication, and perhaps others—into a single large study focused on testing theories about the mechanism of the intervention (via moderator and mediation analyses), questions of impact variation, and questions about the local conditions under which the intervention may be effective. Certainly, a study of this type would be more expensive (and require more schools) than a single efficacy trial. Yet, it may be less expensive (and require fewer schools) when compared to the multiple trials necessary to produce this evidence in the current approach.

For other interventions, this may mean that a series of smaller studies, closer to the prototypical “efficacy” study, may be necessary. For example, this may be the case for a high-dosage, focused intervention funded by NCSER. The distinction is that while the studies may be conducted sequentially,

the planning of the studies would need to consider the broader set of studies and what the contribution of a particular study is to answer theoretical questions about mechanism and heterogeneity. Instead of a wait-and-see approach, researchers would design a series of studies (what may currently be considered replication studies) in advance, to develop what might be called a “prospective meta-analysis” and to argue clearly for how these studies in combination will answer the questions posed.

Taken together, this combination of experimental and quasi-experimental, efficacy and effectiveness studies into a single project type means that decisions regarding the methods, scale, and purpose of the study would need to be aligned clearly with the intervention proposed, the population in need, and the state of knowledge in the field. With these considerations in mind, a study would need to articulate why this research design is the best possible design for the intervention studied. It may be, for example, that for a structural intervention affecting school district organization—in a literature in which there are no previous causal studies—a well-done quasi-experimental design would provide the best evidence possible at this time. At the same time, arguing for this design in a study focused on a student-level intervention in which there are several previous studies using randomized designs may be much harder. Again, this framing allows researchers to center the needs of schools and gaps in the research base, instead of the choice of study design.

Finally, we note that this new conception of Impact and Heterogeneity is grounded in IES’s longstanding commitment to assessing causal impact using designs that warrant such inferences. As stated in ESRA Section 102(19)(D), a “scientifically valid education evaluation” is one that “employs experimental designs using random assignment, when feasible, and other research methodologies that allow for the strongest possible causal inferences when random assignment is not feasible [emphasis added].” The revision we are recommending fulfills ESRA’s promise to ensure that education research meets the standards of science, but with sufficient nuance to be implemented in real schools in a timely way.

4. Knowledge Mobilization

Current: The current project structure is built upon the assumption that decision makers will act upon evidence once it is available. In fact, very few school and district leaders regularly consult the WWC as a way to learn about research (Penuel et al., 2016). More generally, education leaders do not regularly incorporate examination of research findings in an instrumental way when making decisions about programs or policies (Finnigan & Daly, 2014; Coburn, Honig, & Stein, 2009). In recent years, IES has increased requirements for dissemination from grants. These dissemina-

tion plans focus on increasing and diversifying the number of outputs that researchers produce; for example, this might include publishing in both scholarly journals and practitioner journals. These dissemination plans also encourage researchers to make their intervention available (e.g., via websites) and to provide findings to the schools involved in the research. Based on available research, however, these specific dissemination plans, however well intended, are unlikely to greatly influence education decision makers at the early childhood, K–12, and postsecondary levels, because reporting research evidence, even when it is timely, relevant, and accessible, does not necessarily lead to the use of evidence (Finnigan & Daly, 2014).

While IES’s operating assumption is that educational leaders need high-quality evidence about interventions, research shows that educational leaders and practitioners in fact face a glut of information of varying quality (e.g., Tseng, 2012; Honig & Coburn, 2008). It is not clear how decision makers weigh the evidence produced by IES-funded research versus other sources, including research that is less rigorous. This suggests that the problem is not simply one of providing evidence to those in need. Instead, it is about understanding how school and district leaders are making decisions about improving student outcomes, how research makes its way into these deliberations, and the conditions and supports that enable them to use this research evidence in productive ways. Conceiving of the problem of evidence use in this way makes it clear that the same sorts of questions that IES fosters about evidence production also warrant asking about evidence use.

Further, even if education decision makers consult research to adopt interventions, implementation requires practitioners to integrate and adapt the interventions in a new setting (Joyce & Cartwright, 2020; Nilsen, 2015; Dearing & Kee, 2012; Rabin et al., 2012; Greenhalgh et al., 2005). That adaptation process may then result in practices that no longer align to the original evidence (e.g., Cohen, 1990). Conversely, studying interventions developed in practice may enable more systematic spread of success (LeMahieu et al., 2017; Spreitzer & Sonenshein, 2004). For these reasons, we propose that strategies to mobilize knowledge be studied directly, not merely as another stage of the project to implement the findings in practice, but rather as a research enterprise in itself.

New: We propose a new project type focused on Knowledge Mobilization. We propose the term “knowledge mobilization” rather than “knowledge utilization” or “research evidence use” because we incorporate into this project type the organization and synthesis of bodies of evidence as well as improvement of the use of research evidence in real-world settings. This project type would encompass a range of activities and would, in many ways, serve as the central engine of the research infrastructure.

These projects would focus on studies of the conditions that foster research use in a range of contexts from early childhood to postsecond-

ary, synthesizing bodies of evidence to arrive at generalizable conclusions (about “what works, for whom, and under what conditions”), developing and testing robust strategies to foster the use of research in varied contexts, and studies to support the development of robust measures of research use. As Conaway (2021) argued,

Consistent with the view that evidence use demands the same type of attention as that given to evidence production, we propose that projects of this type might include descriptive, synthesis, intervention, or measurement focus, which we describe below. Given this range of possible study foci, the committee deliberated on whether it would be more appropriate for Knowledge Mobilization to be a topic or a project type. We ultimately decided that by positioning Knowledge Mobilization as a project type, we emphasize that the entire field of education research needs to develop and study the success of these strategies to integrate research with practice. Establishing Knowledge Mobilization as a project type that cuts across multiple topics, rather than as a standalone topic, also recognizes that due to heterogeneity in populations, interventions, implementation, adaptation, and contexts, successful mobilization strategies likely differ by topic. They vary not just by domain (e.g., language and literacy, math, socioemotional learning, or technology use), but also by sector (e.g., early childhood, postsecondary education, or special education), given different structures, accountability requirements, staffing pipelines, family partnership roles, and cultural norms. Thus, by designating it as a project type, the responsibility for understanding how to mobilize knowledge lies within existing topic areas (e.g., literacy), not as a separate body of research that can be ignored.

4a. Descriptive studies on Knowledge Mobilization would include research on the ways that system leaders draw on (or do not use) research evidence in their ongoing decision making, including but not limited to the factors influencing development, adoption, and adaptation of policies and practices. Other studies might examine the organizational, social, and political conditions that enhance or inhibit research engagement in school

___________________

¹⁰ For a discussion of these and other terms for evidence use, see Nelson and Campbell (2019).

systems, including how existing power structures maintain and reproduce inequities in knowledge use. Yet others might explore how publishers update materials when new research emerges, how education technology companies incorporate current research, how the media portray and report on research, and the networks via which educational decision makers share knowledge. Some studies of this type might focus on the broader research enterprise, including whose knowledge is valued, who gets to decide on the implications of knowledge, and who is benefited or harmed by the production or use of that knowledge.

Another critical set of questions could explore circumstances leading to inequitable or harmful consequences of knowledge mobilization, particularly research which devalues lived experience or perpetuates deficit narratives (Chicago Beyond, 2019; Doucet, 2019; Kirkland, 2019; Tuck & Yang, 2014). Importantly, these studies would need to identify ways in which the system could be changed or intervened upon to increase productive and equitable research use. Finally, because studies of the social, political, and organizational conditions that foster decision making have largely been done in the context of K–12 education and, more specifically in K–12 general education, descriptive studies are needed that focus on early childhood settings, postsecondary, and special education at every level for the birth–grade 16 system.

4b. Synthesis studies would take stock of both current practices in schools as well as interventions studied to date, indicating across interventions the types of program features that are effective and which are not. At the same time, and just as importantly, these syntheses would identify gaps in the evidence base—places where decision makers need evidence and where such evidence does not yet exist. In this way, these syntheses would both provide evidence (ultimately to be “mobilized”) for schools and decision makers and for the research community regarding priorities for the future.

This information for the research community could also include interstitial work between project areas, helping summarize and disseminate important research regarding current practices and contexts in schools (to those that develop interventions), and locating and elevating promising new interventions (to those that conduct impact studies). Notably, this would mean moving meta-analyses from Exploration studies (current) to Knowledge Mobilization (new).

4c. Intervention studies in Knowledge Mobilization would focus on the development and evaluation of strategies for mobilizing knowledge; developing and investigating tools to support incorporating evidence in decision making; partnerships between intervention developers and vendors; and partnerships between researchers and practitioners. Indeed, more research on the effectiveness of research-practice partnerships is needed to

contribute to improved understanding about whether and how these structures for connecting research and practice not only provide a context for the emergence of trusting relationships, but also affect decisions made by educational leaders and outcomes for students (Penuel et al., 2020; Schneider, 2020). However, interventions should not only target the instrumental use that IES has long prized: that is, situations where research is applied to inform a specific decision, usually after weighing the relative costs and benefits of various options (Weiss & Bucuvalas, 1980). In light of research that suggests the power and importance of conceptual use in educational decision making at the K–12 and postsecondary level (Penuel et al., 2017; Finnigan & Daly, 2014; Farley-Ripple, 2012)—that is, situations where individuals change how they view a problem or possible solutions via engagement with research, often outside of a specific decision—interventions designed to foster such use should also be a priority. Finally, intervention research should be especially attentive to fostering knowledge mobilization strategies that are most likely to address structural and systemic inequality, as there is a long and unfortunate history of research reinforcing rather than interrupting inequality (Kirkland, 2019; Saini, 2019).

Intervention studies related to knowledge mobilization should be rooted in existing research on the nature of decision making in specific contexts, such as those discussed above. Specific knowledge mobilization strategies may differ in the way they affect the production as well as the use of evidence. For example, community-engaged scholarship may strengthen the relevance and rigor of the research produced, through refining the questions and methods to better fit the local context (Balazs & Morello-Frosch, 2013). In contrast, intermediaries and networks may be especially valuable for facilitating conceptual research use, through increasing connections to research knowledge and enabling dialogue with trusted colleagues about implications (Penuel et al., 2020; Neal et al., 2015; Finnigan & Daly, 2014). Understanding these distinct mechanisms and outcomes could help elucidate how best to mobilize knowledge across research and practice.

4d. Measurement studies would also be necessary to develop valid and reliable measures of knowledge mobilization. Gitomer and Crouse (2019) provide a reference guide to such measurement work, and Penuel and colleagues (2016) provide an example from IES-funded research. In developing these measures, it will be important to attend to the variety of ways in which research evidence may be used. Measuring conceptual use of research is notoriously difficult and thus should be a priority for measurement studies. Likewise, developing reliable measures of research use and the varied contributions of multiple stakeholders to the generation, use, and impact of high-quality research will help the field build a common understanding of success.

Finally, we note that ESRA specifically charges IES with disseminating its work “in forms that are understandable, easily accessible, and usable, or adaptable for use in the improvement of educational practice by teachers, administrators, librarians, other practitioners, researchers, parents, policymakers, and the public…” (Section 102(10)). ESRA also asks, as part of the mission of the Research Center, for IES to “support the synthesis, and as appropriate, the integration of education research” (Section 131(b)(1) (D)) and to “synthesize and disseminate, through the National Center for Education Evaluation and Regional Assistance, the findings and results of education research conducted or supported by the Research Center” (Section 133(a)(7)). A project type for Knowledge Mobilization would be newly responsive to these elements of the IES mandate.

CONCLUSION

Throughout this chapter, we have argued that it is time for a structure of science that embraces and builds upon the past 20 years of knowledge generated by IES. We have focused here on project types both since they structure the types of studies that can be funded by the agency and since they serve a normative role, identifying clearly to the field what “IES research” is about. We have argued that this new system needs to be focused around the end goal—to improve student outcomes and reduce disparities—and around the decision makers who ultimately mediate this process. This new system needs to be built upon five principles: equity, heterogeneity, implementation, usefulness, and infrastructure. These principles result in four new project types that are uniquely suited to the science of education: Discovery and Needs Assessment, Development and Adaptation, Impact and Heterogeneity, and Knowledge Mobilization. In sum, the committee recommends:

RECOMMENDATION 4.1:

IES should adopt new categories for types of research that will be more responsive to the needs, structures, resources, and constraints found in education. The revised types of research should include

• Discovery and Needs Assessment
• Development and Adaptation
• Impact and Heterogeneity
• Knowledge Mobilization
• Measurement

The committee envisions a model of research that would have multiple parts working simultaneously for educational change. Since this new structure assumes that different researchers will play different roles, focusing

on different types of studies, this system cannot depend upon individual researchers to move interventions along the research-to-practice process. For this reason, Knowledge Mobilization sits at the center, serving as the engine of this structure. These studies fit at the interstitial spaces, connecting research of one type with researchers focused on another, researchers with practitioners, and synthesizing and integrating knowledge (see Figure 4-1). Each of the other three project types interface both with one another and with this central engine.

The committee expects that this revised project structure will facilitate research that will be more useful and better used in practice. First, shining an equity lens across the entire research portfolio demands a broader examination of systems and practices, as well as a deeper analysis of the mechanisms by which inequities emerge and persist. Elevating the use of descriptive and quasi-experimental methods enables unraveling the many contextual factors and systemic processes that perpetuate or disrupt inequitable opportunities. Second, the need to anticipate and examine heterogeneity requires prioritizing these important questions immediately, facilitating faster discovery of and response to the many differences which exist across populations and contexts. Third, planning for implementation from the beginning requires researchers to ensure that their proposed strategies and interventions are usable, emphasizing the need to study phenomena in real-life settings, not just in the laboratory, and to study adaptations throughout the process. Finally, more fluid movement and more rapid iteration across project types accelerates the production of useful and actionable research, not just theoretically interesting findings that await further study before yielding relevant implications. With these changes, research will be better

positioned to address urgent questions for policy and practice through providing more useful knowledge. We provide an overview of this new structure in Table 4-5, describing the possible questions that might emerge when examining crosscutting themes (heterogeneity, implementation, and equity) within each of our proposed project types.

Connecting all of these parts is the new project type of Knowledge Mobilization, which highlights the need to systematically study and improve both research usefulness and research use. Such inquiry may unearth processes of knowledge exchange that would enable researchers to develop a richer understanding of the kinds of research that would be useful for educational practice. Further, knowledge mobilization explicitly studies the conditions and processes that promote more systematic, sustained, and reliable use of research. Positioning knowledge mobilization in the center of the engine creates greater opportunities for sharing and applying knowledge across all stages of the research enterprise. Embedding these revised expectations within IES’s RFAs would direct the education research community to prioritize these needs in how they conceptualize and conduct research.

Reorganizing project types in the way we have described will allow IES to fund research that more closely addresses the needs of the field. In the following chapter, we turn to a discussion of topic areas, and offer insight into how IES might continue this reorganization toward better meeting its stated goals.

TABLE 4-5 Proposed Project Type Structure and Crosscutting Themes

REFERENCES

Conaway, C. (2021). Funding Research That Is Useful and Used. Blog post. https://www.educationnext.org/funding-research-that-is-useful-and-used-20-years-institute-education-sciences/.

Connolly, P., Biggart, A., Miller, S., O’Hare, L., and Thurston, A. (2017). Using Randomised Controlled Trials in Education. Thousand Oaks, CA: SAGE Publications.

Damschroder, L.J., Aron, D.C., Keith, R.E., Kirsh, S.R., Alexander, J.A., and Lowery, J.C. (2009). Fostering implementation of health services research findings into practice: A consolidated framework for advancing implementation science. Implementation Science, 4, 50. https://doi.org/10.1186/1748-5908-4-50.

Dearing, J.W., and Kee, K.F. (2012). Historical roots of dissemination and implementation science. In R.W. Brownson, G.A. Colditz, and E.K. Proctor (Eds.), Dissemination and Implementation in Health: Translating Science to Practice (pp. 47–62). New York: Oxford University Press.

Doucet, F. (2019). Centering the Margins: (Re)defining Useful Research Evidence Through Critical Perspectives. New York: William T. Grant Foundation.

Farley-Ripple, E.N. (2012). Research use in school district central office decision making: A case study. Educational Management Administration & Leadership, 40(6), 786–806. https://doi.org/10.1177/1741143212456912.

Farley-Ripple, E., May, H., Karpyn, A., Tilley, K., and McDonough, K. (2018). Rethinking connections between research and practice in education: A conceptual framework. Educational Researcher, 47(4), 235–245.

Farrell, C.C., and Coburn, C.E. (2017). Absorptive capacity: A conceptual framework for understanding district central office learning. Journal of Educational Change, 18, 135–159.

Farrell, C.C., Penuel, W.R., and Davidson, K. (2022). “What Counts” as research? Comparing policy guidelines to the evidence education leaders report as useful. AERA Open, 8(1), 1–17.

Farrell, C.C., Penuel, W.R., Coburn, C., Daniel, J., and Steup, L. (2021). Research-Practice Partnerships in Education: The State of the Field. New York: William T. Grant Foundation. http://wtgrantfoundation.org/library/uploads/2021/07/RPP_State-of-the-Field_2021.pdf.

Finnigan, K.S., and Daly, A.J. (Eds.) (2014). Using Research Evidence in Education: From the Schoolhouse Door to Capitol Hill. Cham, Switzerland: Springer.

Gartlehner, G., Hansen, R.A., Nissman, D., et al. (2006). Criteria for Distinguishing Effectiveness from Efficacy Trials in Systematic Reviews. Technical Reviews, No. 12. Rockville, MD: Agency for Healthcare Research and Quality. https://www.ncbi.nlm.nih.gov/books/NBK44024/.

Gitomer, D.H., and Crouse, K. (2019). Studying the Use of Research Evidence: A Review of Methods. New York: William T. Grant Foundation.

Gorard, S., See, B.H., and Siddiqui, N. (2017). The Trials of Evidence-Based Education: The Promises, Opportunities, and Problems of Trials in Education. New York: Routledge.

Greenhalgh, T., Robert, G., Bate, P., Kyriakidou, O., Macfarlane, F., and Peacock, R. (2005). Diffusion of Innovations in Health Service Organisations: A Systematic Literature Review. Oxford, UK: Blackwell.

Greenhalgh, T., Robert, G., Macfarlane, F., Bate, P., and Kyriakidou, O. (2004). Diffusion of innovations in service organizations: Systematic review and recommendations. The Milbank Quarterly, 82, 581–629. https://doi.org/10.1111/j.0887-378X.2004.00325.x.

Honig, M.I., and Coburn, C. (2008). Evidence-based decision making in school district central offices toward a policy and research agenda. Educational Policy, 22, 578–608. https://doi.org/10.1177/0895904807307067.

Institute of Education Sciences (IES) Request for Applications (RFA). (2018). Education Research Grants Program Request for Applications.

——— (2022). Education Research Grants Program Request for Applications.

——— (2021). Education Research Grants Program Request for Applications.

——— (2019). Education Research Grants Program Request for Applications.

Institute of Education Sciences and National Science Foundation (2013). Common Guidelines for Education Research and Development. https://ies.ed.gov/pdf/CommonGuidelines.pdf.

Jackson, C. (2021). Democratizing the development of evidence. Educational Researcher, 49.

Joyce, K.E., and Cartwright, N. (2020). Bridging the gap between research and practice: Predicting what will work locally. American Educational Research Journal, 57, 1045–1082.

Kirkland, D.E. (2019). No Small Matters: Reimagining the Use of Research Evidence from a Racial Justice Perspective. New York: William T. Grant Foundation.

Klager, C.R., and Tipton, E.L. (2021). Commissioned Paper on the Summary of IES Funded Topics. Paper prepared for the National Academies of Sciences, Engineering, and Medicine, Committee on the Future of Education Research at the Institute of Education Sciences in the U.S. Department of Education. https://nap.nationalacademies.org/resource/26428/READY-KlagerTipton_IES_Topic_Analysis_Jan2022v4.pdf.

LeMahieu, P.G., Grunow, A., Baker, L., Nordstrum, L.E., and Gomez, L.M. (2017). Networked improvement communities: The discipline of improvement science meets the power of networks. Quality Assurance in Education, 25(1), 5–25.

Loudon, K., Treweek, S., Sullivan, F., Donnan, P., Thorpe, K.E., and Zwarenstein, M. (2015). The PRECIS-2 tool: Designing trials that are fit for purpose. BMJ, 350, h2147. https://doi.org/10.1136/bmj.h2147.

McLeod, S., Verdon, S., and International Expert Panel on Multilingual Children’s Speech. (2017). Tutorial: Speech assessment for multilingual children who do not speak the same language(s) as the speech-language pathologist. American Journal of Speech-Language Pathology, 26(3), 691–708.

Michie, S., van Stralen, M.M., and West, R. (2011). The behaviour change wheel: A new method for characterising and designing behaviour change interventions. Implementation Science, 6(42). https://doi.org/10.1186/1748-5908-6-42.

Nakajima, N. (2021). Evidence-Based Decisions and Education Policymakers. Unpublished paper. Cambridge, MA: Harvard University.

Nasir, N.S., Scott, J., Trujillo, T., and Hernandez, L.E. (2016). The sociopolitical context of teaching. In D. Gitomer and C. Bell (Eds.), The Handbook of Research on Teaching (pp. 349–390). Washington, DC: American Educational Research Association.

Neal, J.W., Neal, Z.P., Kornbluh, M., Mills, K.J., and Lawlor, J.A. (2015). Brokering the research-practice gap: A typology. American Journal of Community Psychology, 56, 422–435. https://doi.org/10.1007/s10464-015-9745-8.

Nelson, J., and Campbell, C. (2019). Using evidence in education. In A. Boaz, H. Davies, A. Fraser, and S. Nutley (Eds.), What Works Now? Evidence-Informed Policy and Practice (pp. 131–149). Bristol, UK: Policy Press.

Nilsen, P. (2015). Making sense of implementation theories, models and frameworks. Implementation Science, 10, 53. https://doi.org/10.1186/s13012-015-0242-0.

Penuel, W.R., Farrell, C.C., Anderson, E.R., Coburn, C.E., Allen, A.-R., Bohannon, A.X., Hopkins, M., and Brown, S. (2020). A Comparative, Descriptive Study of Three Research-Practice Partnerships: Goals, Activities, and Influence on District Policy, Practice, and Decision Making. Technical Report 4. Boulder, CO: National Center for Research in Policy and Practice.

Penuel, W.R., Farrell, C.C., Allen, A.R., Toyama, Y., and Coburn, C.E. (2018). What research district leaders find useful. Educational Policy, 32(4), 540–568. https://doi.org/10.1177/0895904816673580.

Penuel, W.R., Briggs, D.C., Davidson, K.L., Herlihy, C., Sherer, D., Hill, H.C., Farrell, C., and Allen, A.-R. (2017). How school and district leaders access, perceive, and use research. AERA Open. https://doi.org/10.1177/2332858417705370.

Penuel, W.R., Briggs, D.C., Davidson, K., Herlihy, C., Sherer, D., Hill, H.C., Farrell, C., and Allen, A.-R. (2016). Findings from a National Study of Research Use among School and District Leaders. Technical Report No. 1. Boulder, CO: National Center for Research in Policy and Practice.

Philip, T.M., Bang, M., and Jackson, K. (2018). Articulating the “How,” the “For What,” the “For Whom,” and the “With Whom” in Concert: A Call to Broaden the Benchmarks of our Scholarship. Cognition and Instruction, 36:2, 83–88.

Powell, B.J., Waltz, T.J., Chinman, M.J., Damschroder, L.J., Smith, J.L., Matthieu, M.M., Procter, E.K., and Kirchner, J.E. (2015). A refined compilation of implementation strategies: Results from the Expert Recommendations for Implementing Change (ERIC) project. Implementation Science, 10, 21. https://doi.org/10.1186/s13012-015-0209-1.

Rabin, B., and Brownson, R. (2012). Developing the terminology for dissemination and implementation research. In R.C. Brownson, G.A. Colditz, and E.K. Proctor (Eds.), Dissemination and Implementation Research in Health: Translating Science to Practice (Chapter 2). New York: Oxford University Press.

Rabin, B.A., Purcell, P., Naveed, S., Moser, R.P., Henton, M.D., Proctor, E.K., Brownson, R.C., and Glasgow, R.E. (2012). Advancing the application, quality and harmonization of implementation science measures. Implementation Science, 7, 119. https://doi.org/10.1186/1748-5908-7-119.

Schneider, M. (2020). Research-Practice Partnerships, Redux [Blog post]. https://ies.ed.gov/director/remarks/2-4-2020.asp.

Saini, A. (2019). Superior: The Return of Race Science. Boston, MA: Beacon Press.

Spreitzer, G.M., and Sonenshein, S. (2004). Toward the construct definition of positive deviance. American Behavioral Scientist, 47(6), 828–847. https://doi.org/10.1177/0002764203260212.

Stuart, E.A., Bell, S.H., Ebnesajjad, C., Olsen, R.B., and Orr, L.L. (2017). Characteristics of school districts that participate in rigorous national educational evaluations. Journal of Research on Educational Effectiveness, 10, 168–206.

Tabak, R.G., Khoong, E.C., Chambers, D.A., and Brownson, R.C. (2012). Bridging research and practice: Models for dissemination and implementation research. American Journal of Preventive Medicine, 43(3), 337–350. https://doi.org/10.1016/j.amepre.2012.05.024.

Tipton, E. (2021). Beyond generalization of the ATE: Designing randomized trials to understand treatment effect heterogeneity. Journal of the Royal Statistical Society: Series A (Statistics in Society), 184(2), 504–521.

Tipton, E., Spybrook, J., Fitzgerald, K.G., Wang, Q., and Davidson, C. (2021). Toward a system of evidence for all: Current practices and future opportunities in 37 randomized trials. Educational Researcher, 50(3), 145–156. https://doi.org/10.3102/0013189X20960686.

Tseng, V. (2012). The uses of research in policy and practice and commentaries. Social Policy Report, 26, 1–24. https://doi.org/10.1002/j.2379-3988.2012.tb00071.x.

Tuck, E., and Yang, K.W. (2014). R-words: Refusing research. In D. Paris and M.T. Winn (Eds.), Humanizing Research: Decolonizing Qualitative Inquiry with Youth and Communities (pp. 223–247). Thousand Oaks, CA: SAGE Publications.

Turley, R.N.L., and Stevens, C. (2015). Lessons from a school district–university research partnership: The Houston Education Research Consortium. Educational Evaluation and Policy Analysis, 37, 6S–15S.

Waltz, T.J., Powell, B.J., Chinman, M.J., Smith, J.L., Matthieu, M.M., Proctor, E.K., Damschroder, L.J., and Kirchner, J.E. (2014). Expert recommendations for implementing change (ERIC): Protocol for a mixed methods study. Implementation Science, 9, 39. https://doi.org/10.1186/1748-5908-9-39.

Weiss, M.J., Bloom, H.S., and Brock, T. (2014). A conceptual framework for studying the sources of variation in program effects. Journal of Policy Analysis and Management, 33, 778–808.

Weiss, C.H., and Bucuvalas, M.J. (1980). Social Science Research and Decision Making. New York: Columbia University Press.

Weiss, M.J., Bloom, H.S., Verbitsky-Savitz, N., Gupta, H., Vigil, A.E., and Cullinan, D.N. (2017). How much do the effects of education and training programs vary across sites? Evidence from past multisite randomized trials. Journal of Research on Educational Effectiveness, 10(4), 843–876. https://doi.org/10.1080/19345747.2017.1300719.

Westlund, E., and Stuart, E.A. (2017). The nonuse, misuse, and proper use of pilot studies in experimental evaluation research. American Journal of Evaluation, 38(2), 246–261. https://doi.org/10.1177/1098214016651489.

This page intentionally left blank.