Skip to main content

Currently Skimming:


Pages 54-85

The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.


From page 54...
... 54 THE FUTURE OF EDUCATION RESEARCH AT IES of the activities of the institute; and (4) advising and providing recommendations to the IES director in a number of areas related to enhancing the scope and impact of IES-funded activities and enhancing the overall effectiveness of the institute.
From page 55...
... In FY2010, NCSER received more than $71 million, but this amount was cut by Congress by more than $20 million annually in subsequent years. NCSER's current funding is still $27.1 million short of the buying power of its FY2010 funding level after factoring in inflation, an issue that has yielded serious consternation and instability within the special education research community.
From page 56...
... Known as the Standards for Excellence in Education Research (or the SEER principles) , the principles are comprised of a set of "key domains and core questions" aimed at identifying quality in research proposals and supporting the production of high-quality research (see Box 3-1)
From page 57...
... . Given the identification of usefulness in education research as a crosscutting theme described in Chapter 1, the committee notes the existence of multiple bodies of research that provide evidence related to the utility and function of research-practice partnerships.
From page 58...
... . Though RPPs are no longer a separate topic, NCER continues to fund research that involves partnerships between researchers and practitioners, including awards made under the FY2020 and FY2021 Using Longitudinal Data to Support State Education Policymaking competitions.
From page 59...
... Educational Researcher, 45(1)
From page 60...
... . Special Education Research and Development Centers.
From page 61...
... Project types thus set the stage for the types of studies that IES would like to see conducted, including the purpose of each study. Second, these project types have from the outset played a normative role in education research, reflecting assumptions about the process through which interventions -- programs, policies, and practices -- ought to be developed and evaluated.
From page 62...
... This is an explanatory, proof-of-concept study, focused on establishing that the intervention can produce an effect under "ideal" conditions (i.e., when 3 While Exploration studies and Development and Innovation studies have remained roughly the same over time, project types (3)
From page 63...
... When the fourth project type shifted to "Replication," this third type was thus repositioned as "Initial Efficacy" studies instead. To better understand these project types, Klager and Tipton (2021)
From page 64...
... NOTE: Total grants includes all grant types, including Research Networks and R&D Centers. Project grants include those awarded in specific goals or project types.
From page 65...
... Given the nature of exploratory work, it might be expected that a smaller percentage of these types of studies would progress. In comparison, 30 percent of Development and Innovation grants are later connected to other grants, including 20 percent associated with Efficacy grants and 4 percent with 5 There are no public data available that clearly identify progressions across project types by intervention.
From page 66...
... However, if the originating study resulted in later studies of different project types, it is counted in both; thus the rows do not sum to the "total." The bolded diagonal numbers indicate multiple grants of the same type related to one another. Those above the bolded diagonal numbers indicate goals that progressed forward (e.g., D&I to Efficacy)
From page 67...
... project types. For example, relative to those at research firms, university researchers are far more likely to be involved in Development and Innovation studies than Efficacy studies (Klager & Tipton, 2021)
From page 68...
... The 2022 RFA for Education Research Grants, for example, recommends that researchers proposing Efficacy and Replication studies "describe the setting and implementation conditions," assess "fidelity of implementation and comparison group practice," collect implementation outcomes, and describe a plan for examining fidelity of implementation. Similarly, the RFA requires proposals to describe their sample and recommends that they define a target population.
From page 69...
... " and notes that when these are included, power analyses for related hypothesis tests should be included. Regarding the concerns about connecting points between different project types, however, there has been considerably less work.
From page 70...
... For example, simple online searches of interventions studied via IES grants often result in project or intervention websites, but without any clear indicator of how often the program is adopted. Research on knowledge mobilization suggests that only 17 percent of school and district leaders report accessing research from the WWC "often" (13%)
From page 71...
... Or it may be that even if marketed and available, it is not packaged and supported in the same way that other curricula and programs are. This suggests that there may be a need for approaches that bring together and incentivize partnerships between researchers, communities, education technology companies, publishers, and nonprofits that focus on selling curricula and professional development to schools.
From page 72...
... . This speaks to a need to better understand knowledge mobilization, including how schools and decision makers identify problems and develop solutions; which interventions, curricula, and programs are currently used in schools; how to get promising evidence into their hands; how education leaders harness that evidence to guide action; and what conditions support education leaders to use research more centrally and substantively in their decision making (Farley-Ripple et al., 2018)
From page 73...
... . Drawing on both the five themes introduced in Chapter 1, and on the previous analysis of how and why interventions may or may not be progressing through the existing project types, the committee identified a set of framing principles for education research that inform the revised project structure.
From page 74...
... Decision makers are inundated with potential interventions and professional development services, in addition to frequently adapting and creating their own, and would benefit from guidance on how to efficiently surface and weigh evidence to compare different options. Altogether, this speaks to a need to better understand knowledge mobilization, including how schools and decision makers identify problems and develop solutions; which interventions, curricula, and programs are currently used in schools; how to get promising evidence into their hands; how educational leaders harness that evidence to guide action; and what conditions support educational leaders to use research more centrally and substantively in their decision making.
From page 75...
... In more practical terms, this means revising the underlying project structure. Like the current structure, we envision four project types, each of which can be crossed with a topic area.
From page 76...
... In examining the abstracts of these reviews, nearly all of them focus on synthesizing the results of randomized trials and high-quality quasi-experiments. Yet in the current structure, while they summarize a broad base of causal research, the research itself is considered "exploratory." New: If, as a field, we are to develop and refine interventions that can successfully improve educational outcomes for students, then it is imperative that these interventions consider the diversity of the educational TABLE 4-4  Current Exploration Study Types SOURCE: Klager & Tipton, 2021 [Commissioned Paper]
From page 77...
... To highlight this anchoring in educational context, we call this new project type Discovery and Needs Assessment. These studies would begin in authentic learning environments, with a focus on observing, measuring, and understanding the varieties of practices and processes on the ground and determining gaps between "what is" and "what could be." By emphasizing the need for situating work in authentic school environments, we are also highlighting the need for a broad range of descriptive work that involves primary data collection.
From page 78...
... 2. Development and Adaptation Current: In the current project structure, Development and Innovation grants focus on iteratively developing or refining new interventions for use in schools.
From page 79...
... and effective implementation strategies. We include the word Adaptation in the title to clarify that adaptations to local contexts will occur and that it is incumbent on researchers to develop their interventions with this in mind.
From page 80...
... Certainly, this increased variation will make it more difficult to estimate a statistically significant average treatment effect in a pilot study. But many scholars in both medicine and education research have already argued that the focus of pilot studies should not be on estimation of the average treatment effect or on testing null hypotheses (e.g., see Westlund & Stuart, 2017)
From page 81...
... , thus not necessarily addressing the heterogeneity found in business-as-usual conditions across education contexts. This prioritization of internal validity can also be seen in the predominance of randomized trials in both efficacy and effectiveness studies.9 The fact that randomization to treatment provides clear identification of a cause-and-effect relationship is not to be disputed.
From page 82...
... Furthermore, a single project type for causal questions, which includes both efficacy and effectiveness studies and addresses heterogeneity as well as the average impact, will elevate matters of external validity to be considered on par with the matters of internal validity. The inclusion of quasi-experiments in this category is of particular importance, as some interventions may simply not be able to be studied using randomized trials.
From page 83...
... By elevating these methodologies, the focus becomes clearly on determining the best evidence for the interventions that schools need, instead of on finding interventions that fit the best methods of evidence. Within randomized trials, combining efficacy and effectiveness studies into a single project type also removes what can be arbitrary distinctions between the two.
From page 84...
... in advance, to develop what might be called a "prospective meta-analysis" and to argue clearly for how these studies in combination will answer the questions posed. Taken together, this combination of experimental and quasi-experimental, efficacy and effectiveness studies into a single project type means that decisions regarding the methods, scale, and purpose of the study would need to be aligned clearly with the intervention proposed, the population in need, and the state of knowledge in the field.
From page 85...
... New: We propose a new project type focused on Knowledge Mobilization. We propose the term "knowledge mobilization" rather than "knowledge utilization" or "research evidence use" because we incorporate into this project type the organization and synthesis of bodies of evidence as well as improvement of the use of research evidence in real-world settings.


This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More information on Chapter Skim is available.