Appendix A

Gathering and Assessing the Evidence

The committee drew on multiple sources of evidence in response to its charge. The Education Sciences Reform Act was a key reference, as was documentation of organizational structure and programming as provided by Institute of Education Sciences (IES) staff. The committee also held four public sessions with IES staff and experts from relevant areas of research. At the first session, on May 6, 2021, the committee heard testimony from Elizabeth Albro, commissioner of the National Center for Special Education Research [NCSER], and Joan McLaughlin, commissioner of NCSER, who provided an overview of the goals and organization of IES as well as insight into their goals for this study. At the second public session, on May 13, 2021, the committee heard from the director of IES, Mark Schneider, who shared his vision for this study and how he intends to use and engage with this report. The committee also heard from Anne Ricciuti, deputy director of science, who explained the review process for NCER and NCSER competitions, and NCER and NCSER program officers Katherine Taylor, Jacquelyn Buckley, Allen Ruby, Erin Higgins, and Emily Doolittle, who discussed their roles and responsibilities.

The third open session was comprised of a 2-day public meeting with multiple panels (June 29 and July 7, 2021). The first panel offered an opportunity for Elizabeth Albro, Joan McLaughlin, and Anne Ricciuti to update the committee and answer additional questions. The second panel provided the committee an opportunity to better understand where NCER and NCSER fit into the landscape of federal research agencies: the committee heard from James Griffin, chief of the Child Development and Behavior Branch at the National Institute of Child Health and Human Development;

Evan Heit, division director in the Division of Research on Learning at the National Science Foundation; and Gila Neta, program director for implementation science in the Division of Cancer Control and Population Sciences at the National Cancer Institute. Analogously, the committee heard from a series of private foundations that support education research in the third panel of the day, with speakers Bob Hughes, director of K–12 education at the Bill & Melinda Gates Foundation; Na’ilah Suad Nasir, president of the Spencer Foundation; and Jim Short, program director of leadership and teaching to advance learning at the Carnegie Corporation of New York.

The committee then turned to two panels focused on the education research needs of practitioners. In the first practitioner panel, the committee heard from “research brokers”—individuals whose job is to help “translate” researchers for practice communities. This panel included Carrie Conaway, senior lecturer at Harvard Graduate School of Education; Raymond Hart, director of research for the Council of Great City Schools; Emily House, executive director for the Tennessee Higher Education Commission; and Kylie Klein, director of research, accountability, and data in Evanston/Skokie School District 65. The second practitioner panel focused on supporting beneficial research partnerships, and included Elaine Allensworth, Lewis-Sebring Director for the University of Chicago Consortium on School Research; Kingsley Botchway, chief of human resources and equity from the Waterloo Community School District; and Colin Chellman, university dean for institutional and policy research at the City University of New York.

The second day of the third meeting’s public session focused on how and why education research is done. In the first panel, the committee heard about methods and measures in education research from Ryan Baker, associate professor at the University of Pennsylvania; David Francis, Hugh Roy and Lillie Cranz Cullen Distinguished University Chair at the University of Houston; Odis Johnson, Bloomberg Distinguished Professor at the Johns Hopkins University; and Elizabeth Stuart, associate dean for education and professor at the Johns Hopkins University. The next panel focused on assessing impact in education research, with panelists Ana Baumann, research assistant professor at Washington University in St. Louis; Becky Francis, chief executive officer of the Education Endowment Foundation; and Adam Gamoran, president of the William T. Grant Foundation and this committee’s chair. Finally, the committee heard from a series of experts on research training in education: Curtis Byrd, special advisor to the provost at Georgia State University; Julie Posselt, associate professor at the University of Southern California; Sean Reardon, professor at Stanford University; and Katharine Strunk, professor and Erickson Distinguished Chair at Michigan State University.

Finally, the committee held a public session focused on topics in special education (August 10, 2021). As part of that panel, the committee heard

from Beth Harry, professor emerita at the University of Miami; Karrie Shogren, professor and senior scientist at the University of Kansas; Patricia Snyder, distinguished professor at the University of Florida; and Vivian Wong, associate professor at the University of Virginia.

In addition to outside experts, the committee commissioned five short papers to help synthesize existing evidence in the field and frame our recommendations. First, we asked Heather Hough and colleagues at Policy Analysis for California Education to offer insight into the scope of loss, both personal and educational, facing the nation in the wake of the COVID-19 pandemic. Next, we asked Shirin Vossoughi, Megan Bang, and Ananda Marin to consider the ways that scholarly understandings of learning have evolved and grown since the founding of IES in 2001. Third, Kara Finnigan offered insight into what is known about how evidence is used in education policy and practice. Shira Zilberstein, under the supervision of Michelle Lamont, provided a paper on the impact of interventions aimed at supporting diversity, equity, and inclusion in academic peer-review processes. Finally, Christopher Klager, under the supervision of committee member Elizabeth Tipton, conducted an analysis of what research topics have been funded through NCER and NCSER since its founding in 2001. In addition to the full review provided by the committee for all five of these papers, the committee sent the Klager and Tipton paper to several external, independent coders under the supervision of committee member Nathan Jones. These external coders were asked to follow directions outlined in the Klager and Tipton paper to “spot check” 10 percent of the paper’s original coding in order to ensure the coding process was both clear and accurate. This process resulted in 95 percent agreement between the original and external coders. For more information on how the Klager and Tipton paper is used in this report, as well as further details on coding processed, see Appendix D.

These papers and their findings have all been considered as scholarly input into the committee’s work. As noted above, published, peer-reviewed literature remains the gold standard by which the committee made its judgments. The committee also received formal public comment from multiple scholarly organizations and individuals, including the deans of the schools of education associated with the LEARN Coalition, the American Educational Research Association, the National Center for Learning Disabilities, and dozens of others. Committee members have evaluated all documentation from IES as well as outside testimony through the lens of their scholarly expertise: these judgments ultimately form the basis of the committee’s recommendations.

Following the completion of a draft report, the committee sent its work into the National Academies of Sciences, Engineering, and Medicine’s review process. The report was reviewed by 15 independent reviewers, whose

areas of expertise map onto and complement the study committee. The purpose of this independent review is to provide candid and critical comments that will assist the National Academies in making each published report as sound as possible and to ensure that it meets the institutional standards for quality, objectivity, evidence, and responsiveness to the study charge. The committee considered the full range of commentary from each reviewer, and made changes to the report draft in response to that commentary. The review of this report was overseen by Michael Feuer, George Washington University, and James House, University of Michigan. They were responsible for making certain that an independent examination of this report was carried out in accordance with the standards of the National Academies and that all review comments were carefully considered.

Concurrent to this review process, the committee shared a redacted version of the original draft with IES staff for fact-checking purposes. The redacted draft report contained the committee’s understanding on matters of fact only: that is, IES staff were not privy to the committee’s analytic work until after the report review process. Following a fact-checking process internal to IES, IES staff returned a set of comments and suggested edits. The committee considered those suggestions based on the following principles: (1) is the suggested edit an issue of fact or of characterization (facts were corrected as advised); (2) if the suggestion is a characterization, does it fit within the committee’s shared understanding and judgment (suggestions for revised characterization that were aligned with the committee’s judgment were adopted subject to the third principle); and (3) in either case, is the suggestion within the bounds of the committee’s statement of task (if so and if the suggestion met either of the first two principles, the suggestion was adopted). Several committee members facilitated a first round of adjudications of these suggestions, and then each chapter was subjected to a second read for consideration by a different committee member. The entire report was then reviewed by the full committee, and submitted for final consideration to the National Academies’ report review process.