1

Introduction

A seismic shift in the landscape of public education in the United States occurred at the beginning of the 21st century. Building on decades of momentum, the years 2000–2004 saw federal and state governments passing a suite of policies that would affect virtually every stakeholder in the public education system, and usher in a new era in how the government interacts with schools. Ideas like “accountability” and “school choice,” though not new to individuals already steeped in the work of education policy and teaching, became common parlance in public discourse around education. Education policy at all levels, most notably articulated in the federal No Child Left Behind Act of 2001, placed accountability for student achievement at the heart of the education enterprise and called upon stakeholders to employ evidence-based programming and practices in the service of that aim. Equally important was the new federal insistence on exposing disparities in achievement among students from a variety of demographic subgroups.

In support of those policy efforts, Congress passed the Education Sciences Reform Act of 2002 (ESRA), authorizing the creation of the Institute of Education Sciences (IES) as the research, evaluation, statistics, and assessment arm of the Department of Education, and crystallizing the federal government’s commitment to providing “national leadership in expanding fundamental knowledge and understanding of education from early childhood through postsecondary study” (ESRA, 2002). The overarching goal of this legislation was to build and share reliable information on education with a broad base of constituents and intended audiences, including parents, educators, students, researchers, policy makers, and the general

public. Specifically, ESRA mandates that IES share information on (a) the condition and progress of education in the United States, including early childhood education and special education; (b) educational practices that support learning and improve academic achievement and access to educational opportunities for all students; and (c) the effectiveness of federal and other education programs. With regard to research, the agency’s charge is to build and disseminate a robust evidence of knowledge gained from “scientifically valid research activities” (ESRA, 2002). In the 20 years since its founding, IES has had a field-defining impact on education research in the United States.

Indeed, it is hard to overstate the role that IES has played in shaping the landscape of education research in the United States. In the intervening two decades since its founding, IES has provided funding for education research and statistics through contracts with both public and private research institutions, competitive awards to institutions around the country, and investments in research training programs, grants, and contracts. The work of IES is driven by an emphasis on using scientific research to guide education policy and practice. The agency’s focus on rigor in its funded research has shaped the enterprise of education research, from who has access to research training, to what counts as high-quality research, to what questions researchers are encouraged to ask.

At the same time, the landscape of public education in the United States has changed since IES was founded in 2002, resulting in a different constellation of priorities and political realities than existed at the time IES was founded. As the 20th anniversary of ESRA approaches, it is time to consider ways that IES can improve its current research activities and plan for future research and training in the education sciences. Such an examination can ensure that IES-funded research moves the field forward on issues that are of critical importance to education and special education policy and practice and that improve learner outcomes.

STUDY SCOPE AND APPROACH

In response to a request from the Institute of Education Sciences, the National Academies of Sciences, Engineering, and Medicine through its Board on Science Education convened the Committee on the Future of Education Research at the Institute of Education Sciences to provide guidance on the future of education research at the National Center for Education Research (NCER) and the National Center for Special Education Research (NCSER).¹ IES directs two additional centers not included in this study: the

___________________

¹ Whereas NCER was created when IES was established by ESRA in 2002, NCSER came along 2 years later through a 2004 amendment to ESRA.

National Center for Education Statistics (NCES)² and the National Center for Education Evaluation and Regional Assistance (NCEE). In focusing on the future of educational research at NCER and NCSER, IES tasked the committee with identifying critical problems and issues, new methods and approaches, and new and different kinds of research training investments (see Box 1-1).

This statement of task is directly focused on helping NCER and NCSER strategically fund education research in the coming decade. Given this focus, a committee was assembled with expertise in the four primary elements of the charge. The committee members have a broad range of expertise including education policy, methods in education research, education leadership, education technology, cognition and student learning, training in education research, social-emotional learning, and early learning. In addition,

___________________

² IES concurrently commissioned two other studies from the National Academies. One addresses key strategic issues related to the National Assessment of Educational Progress program, including opportunities to contain costs and increase the use of technology. The second study addresses NCE’’s portfolio of activities and products, operations, staffing, and use of contractors, focusing on the center’s statistical programs.

the committee was composed of scholars working in general education as well as in special education contexts, with several individuals who conduct research across settings. Several committee members are current or former practitioners and/or administrators in both K–12 and higher education settings. For more information on committee members, see Appendix F.

Interpreting the Statement of Task

One of the primary tasks facing a National Academies committee is to determine the bounds of its statement of task. Accordingly, the committee made judgments about the scope of its work. The statement of task clearly directs the committee to focus on NCER and NCSER and excludes other parts of IES such as NCEE and the Regional Education Laboratories within it. However, there were two issues the committee considered that are primarily in the purview of other units in IES, but that have implications for NCER and NCSER.

The first issue is “dissemination” of research and use of evidence generated by research conducted within NCER and NCSER. Based on materials provided to the committee by IES, the committee understood that IES categorizes dissemination of research findings as the purview of the What Works Clearinghouse (WWC) in NCEE. The theory of change, in this sense, is that research funded by NCER and NCSER that meets WWC standards can be included in the WWC repository, where it can be accessed by practitioners and policy makers in search of scientific evidence to support decision making. However, as the committee describes throughout this report, a contemporary understanding of how evidence is used by education stakeholders demands that knowledge mobilization become integrated into the work of researchers from the outset, and so these considerations are within the bounds of this committee’s work.

The second issue is the review processes that govern who receives grants from NCER and NCSER. Reviews are managed by the Office of Science, which is outside of NCER and NCSER, but the committee’s statement of task clearly asks the committee to address “how best to organize the request for applications issued by the research centers to reflect those problems/issues.” So, while the organization of the Office of Science is out of scope, issues pertaining to how to organize reviews for NCER and NCSER are in scope, as confirmed by the IES deputy director for science in her testimony to the committee. Further, to the extent that WWC standards inform how researchers are designing and implementing their projects such that their research could be included in the repository (see more on the WWC in Chapters 2 and 4, and throughout this report), the committee considered

WWC standards as an implicit factor in the request for application (RFA) process, although stops short of commenting on the WWC itself.

Along those same lines, the committee interpreted the statement of task’s four bullets as the primary tasks relevant to our work, and for this reason, focused on the research centers’ activities that have direct bearing on future investments in critical problems or issues, new approaches or methods, training, and organization of RFAs. As described in Chapter 3, NCER and NCSER support research activities across multiple grant competitions, ranging from annual Education Research and Methods grant competitions to funding for Research and Development centers and Research Networks. One of the mechanisms is the Small Business Innovation Research (SBIR) competitions, which provide “seed funding to for-profit small businesses to develop and evaluate new education technology products to improve education and special education” (IES website, 2022).³ Although the committee recognizes the value of this work, we note that the purpose of SBIR grants does not align with the specific tasks outlined in our scope, and therefore have not addressed this program.

The committee recognized that IES is both guided and constrained by the legislative language in ESRA. For this reason, the committee regularly returned to the legislative language included in ESRA to guide deliberations. As the committee made judgments about the future of NCER and NCSER, we continually reviewed ESRA text to ensure that our recommendations were within bounds.

Approach to Gathering and Assessing Evidence

The committee met five times over an 8-month period—four times completely virtually and once in a hybrid virtual/in-person setting. In addition, subgroups of the committee met throughout this period on an as-needed basis. After reviewing the expertise within the committee itself, the committee invited testimony from a number of outside experts in order to augment its expertise. The committee also considered documentation of organizational structure and programming as provided by IES staff, and invited commentary from the public via an open call for input. For details about who provided testimony to the committee and the topics covered, see Appendix A. For a description of public commentary, see Appendix B.

In addition to hearing from outside experts and soliciting public input, the committee sought additional input on scholarly areas in which we deemed further expertise was necessary. The committee commissioned five short papers to help synthesize existing evidence in the field and frame

___________________

³ This sentence was modified after release of the report to IES to remove the suggestion that the use of SBIR competitions was a new mechanism for NCER. See https://ies.ed.gov/sbir/solicitations.asp.

our recommendations. These papers focused on (1) the scope of loss, both personal and educational, facing the nation in the wake of the COVID-19 pandemic; (2) the ways that scholarly understandings of learning have evolved and grown since the founding of IES in 2002; (3) what is known about how evidence is used in education policy and practice; (4) the impact of interventions aimed at supporting diversity, equity, and inclusion in academic peer-review processes; and (5) an analysis of what research topics have been funded through NCER and NCSER since their founding. These papers and their findings have all been considered as scholarly input into the committee’s work. See Appendix C for a list of commissioned papers, and Appendix D for a full description of the methods used in the analysis of funding at NCER and NCSER.

Published, peer-reviewed literature remains the gold standard by which the committee made its judgments. Committee members relied on a combination of peer-reviewed published literature, the input of experts, and their own professional experience in reaching conclusions and developing recommendations. The committee’s statement of task does not call for a synthesis of specific bodies of scholarship. Instead, we were asked to apply our professional judgment to a discrete set of recommendations about the future of IES, an assignment that requires deep expertise across education contexts and content areas, as well as a breadth of professional experience as IES grantees, reviewers, and research consumers. This particular statement of task demanded that the committee consider the prevailing evidence in their respective fields as the foundation for their expert judgment: that is, in the absence of a specific body of evaluative literature about IES, committee members were called upon to apply their own expertise in making recommendations. The committee was not asked to conduct original research or evaluations on how well IES is meeting its stated mandates: Indeed, the committee was directed to focus its energies on the future rather than perseverate over past events. When determining conclusions and formulating recommendations, the committee relied on our professional expertise to interpret multiple kinds of evidence: documents and information provided by IES staff, the five commissioned background papers, and oral testimony regarding the state of education research in the United States, as well as committee members’ own experiences as producers and consumers of education research. Throughout our deliberations, committee members collaborated to ensure collective agreement on how the evidence was interpreted: that is, one individual’s understanding of the literature in their field was not sufficient evidence to support a claim. The committee took particular care to not offer judgment in the absence of sufficient supporting evidence. In such cases, the committee attempted to elucidate ongoing issues or concerns for IES to consider as it moves forward. The conclusions and recommendations outlined in this report, and the process used to author it, reflect the full

consensus judgment of the Committee on the Future of Education Research at the Institute of Education Sciences.

THE CURRENT CONTEXT OF EDUCATION AND CROSSCUTTING THEMES

As the committee began to address its charge, it became clear that to make recommendations about the future of education research at IES, it needed first to understand how the work of NCER and NCSER fits into the current landscape of education and education research in the United States. In doing so, the committee considered how that education landscape has changed since the founding of IES and whether these changes might have consequences for how NCER and NCSER should operate. The committee also considered how the advances in education research generated by IES’s investments to date should inform a renewed set of priorities for the agency.

Current Context

The social and political context of education in the United States is quite different now than when IES was established. The past 20 years have seen major social and political shifts that both directly and indirectly impact education. Support for public education, politically and economically, has vacillated over this time, creating challenges for K–12 and higher education in providing high-quality learning experiences, retaining staff, and maintaining facilities. Political polarization and ideological differences have become heightened, embroiling educators and education decision makers in conflicts that often do not have much to do with student learning and student well-being. These kinds of tensions have become more visible during the COVID-19 pandemic, as exemplified by protests and often open conflict in school board meetings (Kamenetz, 2021).

The student population across preK–12 and higher education has also shifted over the past 20 years, with greater ethnic and racial diversity and growing numbers of students in K–12 who do not speak English as a first language (Irwin et al., 2021). Over this same period, income inequality in the United States has grown considerably, with consequences for the home and community contexts of students (Gamoran, 2015). PreK–12 schools have also become increasingly segregated by class and race (Reardon et al., 2021; Reardon & Owens, 2014; An & Gamoran, 2009). These trends pose increasing challenges for school systems that serve large numbers of students living in poverty, which all too often are the same school systems that have fewer economic resources in the first place.

There have also been rising concerns over the past two decades about the overall well-being of students—their mental health, their sense of be-

longing in school, and their social and emotional growth (National Academies of Sciences, Engineering, and Medicine, 2019a). While the alarm about students’ overall well-being likely reflects issues in the broader society, schools are both called upon to support and nurture learners and themselves can be toxic and unsafe environments. The rise of school shootings, for example, and disciplinary practices that are differentially applied such that Black, Latinx, and Indigenous students and those with disabilities are more likely to suffer negative consequences are in-school phenomena that threaten learners’ health and well-being (GAO, 2018; Gregory, Skiba, & Mediratta, 2017; Beland & Kim, 2016).

The COVID-19 pandemic, in concert with renewed public attention to issues of racial justice, has spotlighted the pernicious inequities that trouble the nation in a wide range of areas, including its education system (NASEM, 2021a). The impacts on schools and communities are innumerable: In addition to unprecedented disruption to schooling and staffing crises, the nation is dealing with profound personal and familial loss as the COVID-19 death toll continues to rise (NASEM, 2021b).

It is not yet possible to articulate a comprehensive analysis of the full scale of loss facing schools and communities, in part because the crisis is still ongoing. Though much media attention has been paid to the notion of “learning loss” as a result of interference with in-person schooling, the committee acknowledges a series of challenges in interpreting existing evidence around this concern. Beyond student achievement, however, there remains an abundance of open questions about how the pandemic will impact education going forward. Among them, what kind of support will communities need to be able to support student learning in the wake of the death of over 940,000 individuals in the United States? How will the nature of schooling change as a result of shifts made during the pandemic? What lessons can be learned from decisions to shift to remote schooling, and what role will technology play in schools going forward? What is the role of schools in attending to the social-emotional needs of students, families, and communities, and what is the role of families in supporting schools in the wake of the pandemic?

These issues and other pandemic-related concerns will necessarily be of paramount importance as the nation continues to battle the pandemic. In recognizing that education research can and should play a pivotal role in helping schools and communities address these critical questions, the committee has considered its work and framed its recommendations with the understanding that the aftershocks of the COVID-19 pandemic will bear on the research community for generations to come.

In addition to these broader social and political trends, insights from advances in research across the many fields that study education—education science, the learning sciences, psychology, sociology, anthropology, eco-

nomics and political science—are providing more nuanced understanding of the processes of learning itself, as well as how education systems function and can be improved (see Chapter 2 for a more in-depth discussion of these advances). These insights are the result, in part, of IES research investments over the past 20 years and they offer guideposts for how IES will need to renew its approach and its portfolio to be relevant for the next 20 years.

For example, there is now wide recognition that learning is a complex cognitive and emotional phenomenon that is situated in specific social and cultural contexts (NASEM, 2018). The experiences that learners have outside of school shape and influence their learning experiences in school. Similarly, there is now a deeper appreciation of the dynamics and challenges of educational improvement and change. Classrooms, schools, and districts are situated within communities and regions across the country that vary on a variety of dimensions. Changes at the school, classroom, or district level need to be understood in context with recognition that a successful program in one setting may not lead to the same outcomes in another setting. There is also increasing recognition of the need to understand and attend to the interlocking elements of the education system. That is, changing what happens in a given classroom for a given student or group of students may be limited in the absence of attention to a broad array of interacting policies and practices that are under the purview of many different actors and decision makers operating at many different levels of the education system.

Crosscutting Themes

In order to make sense of and provide focus to this broad set of contextual issues and take account of advances in the understanding of learning and of education broadly, the committee developed five crosscutting themes: (1) equity in education, (2) changing use of technology, (3) use and usefulness of education research, (4) heterogeneity in education, and (5) implementation and system change. These themes helped the committee to maintain a coherent analysis as we worked through the specific tasks in our charge. Within each task, we have attempted to use these themes as lenses through which to identify salient questions, analyze key issues, and orient our recommendations.

In the chapters that follow, we refer to these five themes to help explain our thinking and contextualize our recommendations, and endeavor to be transparent where it is our judgment of the available evidence undergirding our claims. In the following sections, we describe why we relied on these five crosscutting themes and why they are essential to the ongoing work of IES.

Equity in Education

As the committee’s work commenced, issues around equity in education emerged as one of the most urgent, primary factors that must be centered in decisions about the future work of IES. As noted above, exposing inequities in student achievement across lines of race, class, gender, language minority status, and disability status was a central feature of the No Child Left Behind Act, which set the stage for the founding of IES. In this section, we describe why equity is designated as a crosscutting theme in this report, as well as our approach to operationalizing the theme in this document.

As noted above, the student population in the nation’s schools has become more racially and ethnically diverse over the past 20 years. Students are more likely to speak a language other than English at home, and there is a higher percentage of students who are immigrants (NASEM, 2020). In addition, rising income inequality has increased residential segregation, as families move to places where they can afford the cost of housing, which frequently leads to areas with high concentrations of poverty (Fry & Taylor, 2012). Black and Latinx children are more likely than White children to live in high-poverty areas (NASEM, 2019b). Specifically,

• The rate of Black children living in high-poverty areas in 2016 was about six times higher than that for White children (30% and 5%, respectively). The rate for Latinx children (22%) was about four times that for non-Latinx White children (Annie E. Casey Foundation, 2018).
• The rate of children living in poverty in 2016 was about three times higher for Black children (34%) than for White children (12%). The rate for Latinx children (28%) was more than double that for White children.

Moreover, Black children (12%) were twice as likely as White children (6%) to live in families in which the head of the household did not have a high school diploma. The rate for Latinx children (32%) was more than five times that for non-Latinx White children (Annie E. Casey Foundation, 2018).

Most school districts reflect the demographic and socioeconomic compositions of their neighborhoods. School assignment policies that send all (or many) children from a high-poverty neighborhood to the same school create schools with high concentrations of children living in poverty. Schools serving children from low-income families tend to have fewer material resources (books, libraries, classrooms, etc.), fewer course offerings, and fewer experienced teachers. The educational opportunities available

to students attending these schools are not of the same quality as those in schools in more affluent neighborhoods (Monarrez & Chien, 2021).

These kinds of disparities in access to educational opportunity are deep and enduring characteristics of the American education system. While education is sometimes characterized as the “great equalizer,” the country has not found ways to successfully address the adverse effects of socioeconomic circumstances, prejudice, and discrimination (NASEM, 2019b). Recognizing this, the last two reauthorizations of the Elementary and Secondary Education Act have specified that states need to address achievement gaps between different student groups.

The committee noted that the language in ESRA’s charge to NCER and NCSER puts equity issues front and center, for example calling on NCER⁴

In addition to these federal mandates, the importance of equity also emerges out of decades of research pointing to educational inequity in all facets of the education system. In the committee’s view, educational inequity is one of the paramount challenges facing education researchers, and often the problems that IES and education research broadly are trying address are fundamentally problems of equity. When ESRA mandates that NCER ensure that its funded work is in service of “ensur[ing] that all children have access to a high-quality education,” NCER is being asked to take on questions of equity. This same logic also applies to work designed to address the achievement gap and the multitude of other problems enumerated under the law.

To frame its thinking on this issue, the committee relied on President Biden’s 2021 Executive Order on Advancing Racial Equity and Support for Underserved Communities Through the Federal Government, which outlines in full all federal agencies’ responsibilities related to equity. President Biden has declared that this order applies across his administration, and

___________________

⁴ This sentence was modified after release of the report to IES in order to clarify the role of NCER vs. NCSER in addressing equity issues.

because IES (and therefore NCER and NCSER) fall under the purview of the executive order (EO), it seems clear that organizational and programmatic decisions within IES will need to be consistent with the order. The committee has taken this into account in forming its recommendations.

The EO directs several federal actors to take actions to rectify past inequities and also advance a formal equity agenda in all future work. Of note, the EO directs the heads of all agencies to “assess whether underserved communities and their members face systemic barriers in accessing benefits and opportunities” in their respective programs, and to produce a plan for addressing these barriers. As part of that plan, agencies should identify “whether new policies, regulations, or guidance documents may be necessary to advance equity in agency actions and programs.” Finally, the EO calls on agencies to “consult with members of communities that have been historically underrepresented in the Federal Government and underserved by, or subject to discrimination in, Federal policies and programs [in order to to] evaluate opportunities, consistent with applicable law, to increase coordination, communication, and engagement with community-based organizations and civil rights organizations” (Executive Order 13985, 2021). These directives, and others, are intended to “better equip agencies to develop policies and programs that deliver resources and benefits equitably to all” (Executive Order 13985, 2021).

Ultimately, the EO’s definition of equity and of underserved communities helped focus the committee’s understanding of IES’s obligations:

Throughout this report, the committee has operationalized these definitions of equity and of underserved communities when discussing how equity considerations can and should enter into IES’s decisions. When the committee calls for attention to equity in its findings and recommendations, it is calling for treatment of underserved communities that is actively “fair, just, and impartial.” In the committee’s view, equitable treatment extends beyond diversity goals, though that may be one aim. Indeed, “just” treatment of underserved communities requires active attention to the historic and systemic issues that have perpetuated inequity broadly. As a result, the

committee has endeavored to put these shared understandings of the terms equity and underserved communities to work throughout this report.

The committee’s interpretation of the text of both ESRA and the executive order point to two primary equity aims for NCER and NCSER. First, NCER and NCSER are obliged to fund research that offers insight into and solutions aimed at addressing the equity challenges outlined in ESRA. To fulfill that obligation, it is incumbent upon IES to encourage research that explores issues related to equity and to support the development of an equitable education research enterprise. This report is intended to assist IES responding to both of those aims.

The urgency of addressing equity in education and understanding how inequities in society interact with inequities in schooling has been made even more salient by the events of the COVID-19 pandemic. The impact of the pandemic has varied widely for different communities with particularly devastating impacts for communities of color and communities experiencing poverty. Understanding how to help students, educators, and communities recover from the devastating effects of the pandemic will require a nuanced and deep understanding of equity.

In sum, the attention to equity issues laid out in both ESRA and Executive Order 13985 is rooted in a wealth of education research that posits that attending to equity is a necessary condition for ensuring that education in the United States lives up to its promise. For these reasons, the committee has used equity in education as a crosscutting theme throughout this report. We have attempted to articulate a set of recommendations that, if operationalized, will allow NCER and NCSER to be responsive to President Biden’s commitment to providing the underserved communities defined in his Executive Order with “an ambitious whole-of-government equity agenda” (Executive Order 13985, 2021).

Technology in Education

Though the role that technology plays in education has certainly changed since 2002, it is critical to note that the importance of technology was explicitly included in the ESRA legislation. In fact, ESRA takes care to specify that attending to the role of technology in education (and in particular, the role of technology in supporting student achievement) should be one of the primary foci of IES’s work (ESRA, 2002). Given the centrality placed on technology in the legislative language, the committee recognizes that the unprecedented technological leaps that have occurred in the last 20 years are a critical consideration for any future education investments.

The scope of the change in how schools engage with technology is dramatic. Although most public schools in 2002 had access to the Internet (via Ethernet cables with a student-to-computer ratio of approximately 5:1), the

vast majority of educational technology offerings were limited and most did not take advantage of the Internet (Wells & Lewis, 2006). Low-cost personal computers did not yet exist, despite discounts offered to schools and educators by many companies. The intervening years have seen robust change not only in the nature of technology used, but also in the modalities in which technology is integrated. Teachers, students, and caregivers now make liberal use of smartphones, tablets, and low-cost laptops, and they leverage an increasing number of related applications and web-based platforms for both communication and educational content (U.S. Department of Education, 2021). New genres of technologies are being used for learning and collaboration, such as games (Plass, Mayer, & Homer, 2020), augmented reality, virtual reality (Weiss et al., 2006), among others, and many schools and districts are endeavoring to productively engage social media platforms (Yamaguchi & Hall, 2017). In addition to the proliferation of student-facing learning management systems such as Google Classroom, teachers, administrators, and other staff are now obliged to engage with a battery of education data systems as part of their jobs. This same phenomenon is true in special education contexts: In the past 20 years, access to adaptive technologies has exploded, enabling exciting new possibilities for learning for special education populations (Zimmerman, 2019). And, as noted later in this chapter, the circumstances surrounding schooling in the COVID-19 pandemic have demanded an exponential increase in teacher, student, and caregiver use of technology-based strategies for supporting remote learning (NASEM, 2021a).

Advances in technology, both in systems to support learning and administrative data systems, have led to an explosion of data on students and schools. How best to leverage these systems to support improved student outcomes, while also respecting privacy and ethical use, are critical issues for education at all levels.

Given this substantial shift, the committee found it prudent to consider not only the speed of change prior to 2022 in adoption of technology, but also the likelihood that future decades will experience continued growth and development. Moreover, ESRA is clear in its direction to IES that technology and its use for and in education needs to play a central role in the work of NCER and NCSER. For this reason, the committee identified the use of technology as a crosscutting theme that must be attended to when addressing the foci in its statement of task.

Use and Usefulness of Education Research

A major goal of IES, as outlined in ESRA, is to facilitate the use of evidence to inform education. The very structure of IES is designed to identify and promote effective approaches that have robust, scientific evidence

behind them. Since the founding of IES, however, there have been major advances in understanding how education decision makers and practitioners use evidence in their work and what can make education research more useful.

When IES was established, a common belief in the field was that when interventions were shown to be effective with rigorous scientific testing, they would be discovered and adopted by users in the field (Farley-Ripple et al., 2018; Coburn, Honig, & Stein, 2009). That is, decision makers would immediately turn to the evidence base when they had a problem to solve. Research conducted during the past two decades, however, shows that research use in education rarely works in this linear fashion (Finnigan & Daly, 2014; Best & Holmes, 2010; Davies & Nutley, 2008). Instead, decisions in school systems rely on a variety of factors, only one of which is evidence produced by research (Coburn, Honig, & Stein, 2009). In fact, policy makers and practitioners are unlikely to identify a problem and turn to peer-reviewed literature for a solution (Penuel et al., 2017). Rather, stakeholders are more likely to engage in conceptual use of research: that is, sustained and iterative interaction with a body of work over time, such that it informs how stakeholders ask questions and understand problems (see Chapter 2 for more discussion).

These insights complicate IES’s task of conducting and promoting evidence-based approaches in education. Ensuring that the problems being addressed in education research are meaningful and important to educators and education decision makers is a key challenge. This has been particularly evident during the pandemic when schools sought guidance on how to best support students’ learning during the crisis, and the education research community had difficulty both identifying existing studies that could provide guidance and mounting new research that could be completed and acted upon in a timely way.

As outlined in ESRA, the functions of IES include obligations to “promote the use, development, and application of knowledge gained from scientifically valid research activities,” and “promote the use and application of research and development to improve practice in the classroom.” Thus, the committee understands that IES’s function is not merely to “disseminate,” or inform the public about, research findings, but to take steps to enable their use in practice. As a result, the committee identified use and usefulness of research as a theme that must be consistently addressed. If the research that NCER and NCSER fund is not useful to or used by its intended audience, then it is not meeting the charge mandated under ESRA to effect change in student outcomes. Throughout this report, the committee repeatedly returns to the question of how NCER and NCSER can continue to ensure that the research it funds is both useful and used.

Heterogeneity

In order to fully address the goals laid out for IES in ESRA, education research funded by NCER and NCSER needs to grapple with the wide variation present at every level of the education system. That is, research on how to improve student outcomes will fall short if it does not explicitly address issues of heterogeneity (Bryan et al., 2021; Bryk et al., 2015). This means that “what works, under what conditions, and for whom” (Gutiérrez & Penuel, 2014, p. 22) and why (Cowen, 2019) must be central questions for research. Often, current approaches to determining what is effective for improving student outcomes assume that there is very little to no variation in effect sizes across students, teachers, and schools. However, over the past 20 years, there is mounting evidence that treatment effects vary, sometimes substantially (Weiss et al., 2017).

This concern suggests to instead begin with the assumption that treatment effects can and will vary across students, teachers, and schools. Studies need to treat this heterogeneity as a primary concern, not secondary. Understanding heterogeneity involves more than merely a statistical exercise in computing and finding variation. Explaining what led to that heterogeneity, and then applying the inferences based on past findings to future settings, requires analyzing how conditions differ and how important those differences are in influencing an observed variation (Provost, 2011; Deming, 1953).

Analyzing variation may also help distinguish between the need for systemic change or for targeted action. Calculating the variability of a process may reveal whether it is stable and predictable, or whether the results emerge from an out-of-control process or from separate systems (Provost & Murray, 2011; Deming, 1953). A stable process producing undesirable results needs to shift the entire system to yield improvement; an unstable process requires systemic improvements to detect and correct issues to bring the process under control. However, high variability emerging from separate systems “raises questions about hidden factors and potential systemic inequities to identify and resolve” (Ming & Kennedy, 2020).

The committee notes that IES has, in fact, made multiple efforts to attend to issues of heterogeneity in its tenure. As we discuss in Chapter 4, IES has called for research that better addresses the “whom, where, and under what conditions” questions embedded in research. As with a number of challenges the committee will describe throughout this report, however, a series of structural issues present in IES guidance creates a funding context in which questions of heterogeneity may be less likely to receive support. For this reason, the committee chose to highlight the importance of these issues in establishing recommendations that build in heterogeneity as an assumption, as well as methods to study and explain heterogeneity more fully.

Implementation and System Change

The portfolio of research funded by both NCER and NCSER makes clear that the success of interventions is driven in large part by their implementation. It is also clear that understanding implementation needs to go beyond simply determining if a given intervention is implemented with fidelity. Rather, there is increasing recognition that the process of implementation itself is worthy of study if education research is to provide educators with sufficient guidance on how to improve student outcomes.

Implementation research “is the scientific study of methods to promote the systematic uptake of research findings and other evidence-based practices into routine practice, and, hence, to improve the quality and effectiveness” of interventions, policies, and practices (Eccles & Mittman 2006, p. 1). This definition distinguishes between what is being implemented and how to support its implementation, where “what” refers to the intervention, evidence-based practice, innovation, or “the thing,” while “how” refers to the implementation strategies or “how to support the thing” (Curran, 2020; Fixsen et al., 2005). Identifying relevant factors influencing implementation, and situating them within a theoretical explanation for their influence, allows stakeholders to identify and develop strategies targeting those factors more effectively, ultimately improving desired outcomes. This shift in framing will also clarify where systemic changes might be needed in order to support a more effective implementation of an intervention.

It is the committee’s judgment that if NCER and NCSER are indeed going to support the kind of research outcomes articulated in ESRA, it is critical that funded research engages with issues of implementation. For this reason, the committee considers implementation as a crosscutting theme throughout this report, and endeavors to address how NCER and NCSER might take on implementation and systemic change in support of its stated goals.

AUDIENCES

This report is intended to address the statement of task provided to the committee by its IES sponsors. For this reason, the committee considers IES, specifically NCER and NCSER stakeholders, as the primary audience for this report. However, the committee sees the audience for this report as extending beyond IES: Insofar as NCER and NCSER support a large percentage of the education research community in the United States, this report is intended to reflect that community’s needs and concerns. The committee therefore sees the education research community as an additional but important audience for this report. Finally, the committee recognizes that IES’s scope is limited both by its governing language in ESRA and by its

congressional appropriations. For these reasons, the committee envisions Congress and other relevant policy makers as another audience.

ORGANIZATION OF THIS REPORT

This report is organized to reflect the committee’s recommendations on the items listed in our statement of task. Following Chapters 2 and 3, which describe the background and current organizational structure of IES, the committee turns to the substance of its argument. In Chapter 4, we discuss our recommendations for a structure of project types for organizing funding in NCER and NCSER, and in Chapter 5, we make recommendations for new topics of study. Together, Chapters 4 and 5 cover the research goals and topics that IES uses to organize its work, and these chapters constitute our response to the first element of our charge—to identify problems and issues that should be considered for IES funding. We address the second, third, and fourth elements of our charge in Chapter 6, which focuses on methods and measures; Chapter 7, which examines the future of training; and Chapter 8, which offers commentary on how the request for applications process can be organized to support NCER and NCSER’s future work. We conclude with a chapter offering our vision for the future of education research in NCER and NCSER.

REFERENCES

Cowen, N. (2019). For whom does “what works” work? The political economy of evidence-based education, Educational Research and Evaluation, 25(1–2), 81–98. https://doi.org/10.1080/13803611.2019.1617991.

Curran, F.C. (2020). A matter of measurement: How different ways of measuring racial gaps in school discipline can yield drastically different conclusions about racial disparities in discipline. Educational Researcher, 49(5), 382–387.

Davies, H., Nutley, S., and Walter, I. (2008). Why ‘knowledge transfer’ is misconceived for applied social research. Journal of Health Services Research & Policy, 13(3), 188–190. https://doi.org/10.1258/jhsrp.2008.008055.

Deming, W.E. (1953). On the distinction between enumerative and analytic surveys. Journal of the American Statistical Association, 48(262), 244–255.

Eccles, M.P., and Mittman, B.S. (2006). Welcome to implementation science. Implementation Science, 1, 1. https://doi.org/10.1186/1748-5908-1-1.

Education Sciences Reform Act (ESRA). (2002). Title I of P.L. 107-279.

Executive Order 13985, 86 FR 7009 (January 25, 2021).

Farley-Ripple, E., May, H., Karpyn, A., Tilley, K., and McDonough, K. (2018). Rethinking connections between research and practice in education: A conceptual framework. Educational Researcher, 47(4), 235–245.

Finnigan, K.S., and Daly, A.J. (Eds). (2014). Using Research Evidence in Education: From the Schoolhouse Door to Capitol Hill. Cham, Switzerland: Springer.

Fixsen, D.L., Naoom, S.F., Blase, K.A., Friedman, R.M., and Wallace, F. (2005). Implementation Research: A Synthesis of the Literature. FMHI Publication No. 231. Tampa: University of South Florida, Louis de la Parte Florida Mental Health Institute, National Implementation Research Network.

Fry, R., and Taylor, P. (2012). The Rise of Residential Segregation by Income. Pew Charitable Trusts. https://www.pewresearch.org/social-trends/2012/08/01/the-rise-of-residential-segregation-by-income/.

Gamoran, A. (2015). The Future of Educational Inequality: What Went Wrong and How Can We Fix It? New York: William T. Grant Foundation. http://wtgrantfoundation.org/resource/the-future-of-educational-inequality-what-went-wrong-and-how-can-we-fix-it.

Government Accountability Office (GAO). (2018). Discipline Disparities for Black Students, Boys, and Students with Disabilities. Washington, DC: Author.

Gregory, A., Skiba, R.J., and Mediratta, K. (2017). Eliminating disparities in school discipline: A framework for intervention. Review of Research in Education, 41, 253–278.

Gutiérrez, K.D., and Penuel, W.R. (2014). Relevance to practice as a criterion for rigor. Educational Researcher, 43(1), 19–23. https://doi.org/10.3102/0013189X13520289.

Irwin, V., Zhang, J., Wang, X., Hein, S., Wang, K., Roberts, A., York, C., Barmer, A., Bullock Mann, F., Dilig, R., and Parker, S. (2021). Report on the Condition of Education 2021 (NCES 2021-144). U.S. Department of Education. Washington, DC: National Center for Education Statistics. https://nces.ed.gov/pubsearch/pubsinfo.asp?pubid=2021144.

Kamenetz, A. (2021). A look at the groups supporting school board protesters nationwide. National Public Radio online. https://www.npr.org/2021/10/26/1049078199/a-look-at-the-groups-supporting-school-board-protesters-nationwide.

Ming, N.C., and Kennedy, A.I. (2020). Developing and using indicators for continuous improvement. Teachers College Record (Yearbook), 122(14). https://www.tcrecord.org/Content.asp?ContentId=23462.

Monarrez, T., and Chien, C. (2021). Dividing Lines: Racially Unequal School Boundaries in US Public School Systems. Research Report. Center on Education Data and Policy. The Urban Institute.

National Academies of Sciences, Engineering, and Medicine (NASEM). (2018). How People Learn II: Learners, Contexts, and Cultures. Washington, DC: The National Academies Press. https://doi.org/10.17226/24783.

———(2019a). The Promise of Adolescence: Realizing Opportunity for All Youth. Washington, DC: The National Academies Press. https://doi.org/10.17226/25388.

———(2019b). Monitoring Educational Equity. Washington, DC: The National Academies Press. https://doi.org/10.17226/25389.

———(2020). Changing Expectations for the K–12 Teacher Workforce: Policies, Preservice Education, Professional Development, and the Workplace. Washington, DC: The National Academies Press. https://doi.org/10.17226/25603.

———(2021a). COVID-19 and the K–12 Teacher Workforce: Seizing the Moment to Reimagine Education: Proceedings of a Workshop–in-Brief. Washington, DC: The National Academies Press. https://doi.org/10.17226/26356.

———(2021b). Back in School: Addressing the Well-Being of Students in the Wake of COVID-19: Proceedings of a Workshop–in Brief. Washington, DC: The National Academies Press. https://doi.org/10.17226/26296.

Penuel, W.R., Briggs, D.C., Davidson, K.L., Herlihy, C., Sherer, D., Hill, H.C., Farrell, C., and Allen, A. (2017). How school and district leaders access, perceive, and use research. AERA Open, 3(2), 1–17.

Plass, J.L., Mayer, R.E., and Homer, B.D. (Eds.) (2020). Handbook of Game-based Learning. Cambridge, MA: MIT Press.

Provost, L.P. (2011). Analytical studies: A framework for quality improvement design and analysis. BMJ Quality & Safety, 20(Suppl 1), i92–i96.

Provost, L.P., and Murray, S. (2011). The Health Care Data Guide: Learning from Data for Improvement. San Francisco, CA: John Wiley & Sons.

Reardon, S.F., and Owens, A. (2014). 60 years after Brown: Trends and consequences of school segregation. Annual Review of Sociology, 40, 199–218.

Reardon, S.F., Weathers, E.F., Fahle, E.M., Jang. H., and Kalogrides, D. (2021). Is Separate Still Unequal? New Evidence on School Segregation and Racial Academic Achievement Gaps. CEPA Working Paper No. 19-06. Stanford, CA: Stanford University, Center for Education Policy Analysis.

U.S. Department of Education. (2021). Use of Technology in Teaching and Learning. https://www.ed.gov/oii-news/use-technology-teaching-and-learning.

Weiss, J., Nolan, J., Hunsinger, J., and Trifonas, P. (Eds.). (2006). The International Handbook of Virtual Learning Environments (Vol. 14). Dordrecht, Netherlands: Springer.

Weiss, M.J., Bloom, H.S., Verbitsky-Savitz, N., Gupta, H., Vigil, A.E., and Cullinan, D.N. (2017). How much do the effects of education and training programs vary across sites? Evidence from past multisite randomized trials. Journal of Research on Educational Effectiveness, 10(4), 843–876. https://doi.org/10.1080/19345747.2017.1300719.

Wells, J., and Lewis, L. (2006). Internet Access in U.S. Public Schools and Classrooms: 1994–2005 (NCES 2007-020). Washington, DC: U.S. Department of Education, National Center for Education Statistics.

Yamaguchi, R., and Hall, A. (2017). Compendium of Education Technology Research Funded by NCER and NCSER: 2002–2014. NCER 2017–0001. Washington, DC: U.S. Department of Education, National Center for Education Research.

Zimmerman, E. (2019). AR/VR in K–12: Schools Use Immersive Technology for Assistive Learning. https://edtechmagazine.com/k12/article/2019/08/arvr-k-12-schools-use-immersive-technology-assistive-learning-perfcon.