The Future of Education Research at IES Advancing an Equity-Oriented Science (2022) / Chapter Skim
Currently Skimming:
6 Methods and Measures
6 Methods and Measures
Pages 129-152
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 129... ...
One of IES's hallmarks since its inception has been its continuous investment in advancing education methods and measures. IES has adopted three primary strategies aimed at improving the quality of research methods in education: (1)
|
From page 130... ...
Through its RFAs and guidance to proposal reviewers -- and in alignment with recommendations for internal validity through the WWC -- IES encourages submitted studies to meet high technical standards. Examples include the requirement that Efficacy and Replication studies be adequately powered, that studies prioritize research designs aligned with causal inferences (e.g., experimental designs, quasi-experimental designs, single subject designs)
|
From page 131... ...
THE FUTURE OF METHODS RESEARCH Summary of Methods Research to Date NCER and NCSER have invested in methodological innovation from their beginnings. This investment was first via unsolicited grants and later through a separate grant program, Statistical and Research Methodology in Education, that funded research relevant to both centers.
|
From page 132... ...
Methods Research Moving Forward In this report, we have argued that education research needs to focus on five crosscutting themes: the heterogeneity of contexts, experiences, and treatment effects; the adaptation of programs and policies to local contexts, leading to different degrees and types of implementation; the need to better understand and test new ways to support the development of knowledge that is useful for decision making; the continued need to take advantage of education technologies; and the need to focus directly on the goal of improving equity in educational experiences. In this section, based upon what has been previously studied and these themes and goals, we propose areas that need new methodological development.
|
From page 133... ...
To date, only three of the methods grants have focused directly on the development and testing of methods for the prediction of local treatment effects. Predicting local effects with precision will require both new statistical methods for analysis, such as machine learning and Bayesian Additive Regression Trees, and more complex research designs, such as factorial, crossover (Bose & Dey, 2009)
|
From page 134... ...
To date, six Statistical Models and Research Methods grants have focused on these topics. However, more methods are needed to study implementation and adaptations made as programs move across places and people (reconceived in Chapter 4 as Development and Adaptation grants)
|
From page 135... ...
The further development, testing, and refinement of these methods will enhance the ability of researchers to study implementation of evidence-based practices in education. Methods for Knowledge Mobilization As the committee noted in Chapter 1 of this report, if the research that NCER and NCSER fund is not useful to or used by its intended audience, it is not meeting the charge mandated under ESRA to effect change in student outcomes.
|
From page 136... ...
However, there is great potential for adapting such methods for use in experimental design of interventions to foster knowledge mobilization that include observation or, for example, video analyses of nationally representative samples of school board meetings (see Box 6-1 for an additional need in the knowledge mobilization space)
|
From page 137... ...
. IES investments in network methods and natural language processing for knowledge mobilization studies could fuel important advances in this area.
|
From page 138... ...
For example, NCER recently awarded five grants under the Digital Learning Platforms to Enable Efficient Education Research Network that will redesign existing digital learning platforms to support research. Education technology data differ from typical data in randomized trials in that they include a vast amount of process data.
|
From page 139... ...
In Chapters 4 and 5, we argued that focusing on interventions that can be studied by randomized trials severely limits the type of interventions that IES-funded studies can focus upon and learn about. Some of the largest effects on student outcomes may, in fact, arise from structural changes that are difficult to randomize.
|
From page 140... ...
Through 2020, the centers have funded 176 measurement studies.1 An analysis of the abstracts of these studies indicates that they can be categorized by their unit of analysis: students, teachers, or "other" (Table 6-1) .2 Collectively, these studies have provided the field with a number of measures related to student outcomes and student characteristics.
|
From page 141... ...
This is evidenced in the broad range of measurement studies focused on student outcomes. At the same time, IES-funded researchers still frequently use standardized test scores and grades as the primary outcomes of their studies.
|
From page 142... ...
-- and the processes and moderators that shape these outcomes -- are important to study in and of themselves. Developing and Validating Measures beyond the Student Level Measures of the structural and contextual factors that shape student outcomes.
|
From page 143... ...
Measures of knowledge mobilization. As discussed in Chapter 4, the committee identified knowledge mobilization as a project type.
|
From page 144... ...
. For example, structurally disadvantaged student populations often experience the classroom setting differently than their structurally advantaged peers; thus, should measures of equity in such student experiences always include an advantaged comparison group?
|
From page 145... ...
. Web-scraping tools, education data mining, and learning analytics and the data that result from these approaches also offer new opportunities for measurement research.
|
From page 146... ...
. RECOMMENDATION 6.1: IES should develop competitive priorities for research on methods and designs in the following areas: • Small causal studies • Understanding implementation and adaptation • Understanding knowledge mobilization • Predicting causal effects in local contexts • Utilizing big data RECOMMENDATION 6.2: IES should convene a new competition and review panel for supporting qualitative and mixed-methods approaches to research design and methods.
|
From page 147... ...
• Developing and validating measures related to educational equity • Using technology to develop new approaches and tools for measurement REFERENCES Alliance for Resource Equity.
|
From page 148... ...
. Statistical process control and interrupted time series: A golden opportunity for impact evaluation in quality improvement. BMJ Quality & Safety, 24, 748–752.
|
From page 149... ...
American Educational Research Journal, 57, 1045–1082. Juran, J.M.
|
From page 150... ...
Paper presented at the Annual Meeting of the American Educational Research Association, April 2021. Means, B., and Harris, C.J.
|
From page 151... ...
Combining critical race theory and quan titative methods. American Educational Research Journal, 56(1)
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.