Appendix D

Analysis of IES Funded Topics Commissioned Paper

As the committee began to discuss how to approach this consensus report, it identified the need for an analysis of Institute of Education Sciences (IES) past spending by topic area, with summary data on the topics studied by the National Center for Education Research (NCER) and National Center for Special Education Research (NCSER) grantees over the past 20 years. Chris Klager is a research associate at the Statistics for Evidence-Based Policy and Practice (STEPP) Center at Northwestern University. He was selected to write this paper because he researches the translation and communication of evidence about educational programs for policy makers and practitioners, and also has experience performing analyses related to projects funded by IES. This work was supervised in its entirety by committee member Elizabeth Tipton. This appendix details how the authors gathered the necessary data under six parameters to create summary tables that were utilized by the committee in the body of the report. A full copy of the final paper, which includes the Codebook that the authors created to develop the summary tables, is available at https://nap.nationalacademies.org/resource/26428/READY-KlagerTipton_IES_Topic_Analysis_Jan2022v4.pdf.

The paper addressed a range of research questions listed below regarding the types of studies that have been funded across different time periods and categories:

• What topics have been studied in research funded by NCER and NCSER, and how has the distribution of funded topics shifted over time?
• How have studies of different project types funded by NCER and NCSER changed over time? How are studies connected to one another?
• What types of interventions are studied? Where are these interventions targeted?
• What is the relative funding distribution across topic areas, and what topic areas have received the highest levels of funding?
• What institutions receive grants from NCER and NCSER? How has this changed over time?
• What Methods and Measurement types have been studied under funded grants?

To answer these questions, Klager and Tipton reviewed publicly available data on IES-funded grants, which included information on each of these areas via the inclusion of study abstracts (IES, 2021). They also included data to classify institution types (R1, MSI, and Private) that came from the Carnegie Classification database which is based on information from the Indiana University Center for Postsecondary Research (n.d.).

The authors found that the complete dataset on the IES website has over 2,500 grants and contracts funded by NCER, NCSER, the National Center for Education Evaluation and Regional Assistance (NCEE), and the National Center for Education Statistics (NCES) from 2002 to 2021. The analysis completed was limited to grants funded by NCER and NCSER between 2002 and 2020. They noted that while 2021 awards were announced, it was unclear if all 2021 awards were present in the data that were downloaded at the time this paper was completed, so those awards were excluded. The analytic dataset also excluded awards funded by NCEE and NCES as this was not within the parameters that the committee was charged to examine in our statement of task. Contracts were also excluded, leaving only grants. All analyses in this paper exclude Small Business Innovation Research (SBIR) grants. Although NCER and NCSER issue SBIR awards, they differ from other awards in several ways. SBIR awards fall into either Phase I Development or Phase II Development. They are of a short duration and target small businesses with an emphasis on commercialization of the products that are developed. Many of them are also classified as contracts rather than grants. SBIR is a federal program that operates across federal agencies and is not unique to the Department of Education.

PROJECT TYPES

When trying to define project types, Klager and Tipton explained that over the past 20 years, NCER and NCSER have funded grants in a variety of categories based on two dimensions—the topic of the grant and project type. Over time, the project types have changed, and for much of the past 20 years, these were divided into numbered goals (1 through 5). More recently, this numbering was removed and some categories shifted. Because of these changes in the wording of the request for applications (RFA) and types of studies that fall under each project type, Klager and Tipton saw some simplification in terminology was required to communicate about each.

Historically, the core project structure included five goals:

• Goal 1 – Exploration
• Goal 2 – Development and Innovation
• Goal 3 – Efficacy
• Goal 4 – Effectiveness
• Goal 5 – Measurement

The categorizations that IES provides on its website include variations on these five goals. Additionally, IES funds grants in other programs such as researcher-practice partnerships (RPPs), Training, Methods, and various special programs including large “center” grants that engage in activities that cover multiple goals. The publicly available data on IES’s website about funded grants includes a field called “GoalText,” but not the actual Goal (i.e., 1, 2, 3, 4, 5, etc.) each grant was funded under. Instead, the GoalText field contains a description that characterizes the purpose of the grant. For the purposes of these analyses, Klager and Tipton categorized grants by their GoalText. This means that all grants that were marked by IES (in the GoalText) as Exploration were categorized as Exploration, regardless of the program the grant was funded under.

While Exploration and Development and Innovation projects have remained approximately the same over the history of IES, Efficacy and Effectiveness studies have changed over time. To explore trends over time, Klager and Tipton had to create new categories which involved combining categories in some cases. One important case is with regards to replication grants, which over time moved from Goal 3 to Goal 4 studies, and then to their own project type.

For purposes of comparison, the authors divided out “Initial Efficacy” studies into their own project type and then combined “Replication” and “Effectiveness” trials into a single category. This required them to determine which Efficacy studies were “initial” trials versus “replications.” To do so,

they turned to Chhin, Taylor, and Wei (2018), who categorized all Goal 3 and Goal 4 grants funded by NCER and NCSER between 2004 and 2016 as either a direct or conceptual replication, new evaluation, re-analysis, or longitudinal follow-up. They used the codes applied by Chhin and colleagues (2018) for the grants that they coded to identify replications.

All other grants with GoalText of Efficacy or Efficacy and Replication that were not coded by Chhin and colleagues were coded using the publicly available abstracts. Following the method described in Chhin et al. (2018), Klager and Tipton checked IES abstracts for evidence of the stated purpose of the evaluation and prior efficacy evaluations of the program. If a study cited pilot evaluations only, including previous Development and Innovation grants from IES, or provided no information about the purpose of the study regarding replication, it was coded as a non-replication and was classified as Efficacy for these analyses. If there was evidence of previous efficacy studies or if the stated goal of the grant was for replication, it was coded as a replication and classified as Replication/Effectiveness. The publicly available abstracts provide limited information about each grant. Chhin and colleagues had access to full grant proposals and were able to identify many replications (~50% of 307 grants). Using abstracts, Klager and Tipton identified 32 out of 189 (17%) additional grants that had GoalText indicating an efficacy trial. It is plausible that coding replications from abstracts undercounts the number of replications based on the disparity between Chhin and colleagues’ rate and the rate Klager coded from abstracts. It is unclear, though, if the rate of replications is consistent across time and programs funded by IES.

Table D-1 shows how those GoalText descriptions were categorized for these analyses. Grants were categorized based on the GoalText rather than the programs under which grants were funded. For example, five grants funded as part of the Digital Learning Platforms to Enable Efficient Education Research Network program had GoalText of “Methodological Innovation” and were classified with other grants that also had the “Methodological Innovation” regardless of the programs they were funded under. The “Other” category includes special grant competitions, unsolicited grants, centers established for the study of particular topics, and other projects that cover multiple goals. All grants with GoalText that cover more than one goal (e.g., Efficacy and Development) were classified as meeting multiple goals and were categorized as Other.

TOPICS

Eight topics were formed using the program names that IES provides as the source of funding for each grant. (See Tables D-2 and D-3 for a list of program names where all grants were assigned to a particular topic and

TABLE D-1 Categorization of GoalText into Grant Categories

Table 1. Categorization of GoalText into grant categories

a list of program names for which topics were coded by coders.) In some cases, the program names are descriptive and map well onto a topic, as is the case with the Science, Technology, Engineering, and Mathematics (STEM) program that maps onto the STEM topic used in this analysis. In other cases, the program name is not very descriptive, as in the case of Research Grants Focused on Systematic Replication. In the cases where the program name was not indicative of the type of intervention or idea being studied, the IES abstracts were coded to fit within the topic categories. Because the topics are not mutually exclusive (e.g., a STEM intervention that happens in an Early Childhood classroom could fall into both the STEM and Early Childhood categories), the authors gave preference to School Systems, Age (Early Childhood and Post-Secondary/Adult), then Cognition & Learning, Social & Behavioral, followed by content area (Reading, Writing, Language, Literacy, & ELL; STEM). School Systems was used for interventions that changed the structure of school operations, regardless of content area (e.g., State-wide remedial Algebra program). The Other category captures a small proportion of grants that do not fit well within the seven other topic categories.

TABLE D-2 Programs that Correspond to a Coded Topic

TABLE D-3 Programs for Which a Topic Was Coded

INSTITUTION TYPE

In categorizing institutions that have received IES funds (both NCER and NCSER), Klager and Tipton decided to have universities include hospitals and research centers that are affiliated with a university. Research firms were defined as nonuniversity institutions whose primary work is in the evaluation of products and programs that they did not develop themselves (i.e., external evaluations). This does not mean that they never engage in development of interventions, products, and techniques but that it is not their primary purpose. Developers, on the other hand, engage in basic research and evaluations, primarily on their own products and interventions. Within the Other category, there are several types of institutions although individually, they make up only a very small proportion of grants and funding awarded by IES. These types of institutions include education service providers, scientific organizations, state departments of education, and school districts. All institutions were coded into an institution type based on the description of the institution on its own website, if available, or other internet sources.

R1 classification was based on the classification given to the university at the time the grant was awarded. Classifications are recalculated every few years by the Indiana University Center for Postsecondary Research, with new releases in 2000, 2005, 2010, 2015, and 2018. Minority-Serving Institutions (MSIs) status is based on the 2018 data; thus, it does not reflect any changes in MSI status over time.

EXPLORATION CATEGORIES

Exploration studies include a range of possible study types. To learn more about these, Klager and Tipton divided these studies into different categories. First, they determined if the study involved collecting primary data or if it only included secondary data. If the former, the grant was classified as “primary,” whereas grants that use only secondary data are classified as “secondary.” Additionally, the authors divided the grants into categories based on study design. These designs were coded based upon information in the abstracts, resulting in the following categories: meta-analysis, correlational analyses, randomized experiments (including pilots), and quasi-experiments (causal questions). There were many Exploration grants that had multiple studies with varying analysis plans. In these cases, if there was any experimental study, the grant was classified as experimental. If there was any meta-analysis, the grant was classified as meta-analysis. If the grant did not use an experiment or conduct a meta-analysis, then if there was a quasi-experiment the grant was classified as such. All other grants were showing associations, correlations, or doing mediation analyses.

METHODS GRANTS

Publicly available IES abstracts for Methods grants were coded for type of statistical method employed/developed, products produced, and topic of study. Klager and Tipton classified studies as psychometric (28), statistical models for analysis (23), randomized control trial design (22), and quasi-experimental design (20). Within those classifications, the authors also noted some subclassifications that commonly were funded or which are of interest to the educational methods research community. Relevant subtypes that were coded include value-added models, multilevel models, missing data, power analysis, effect size computation/interpretation, regression discontinuity, interrupted time series, single-case design, heterogeneity, external validity, and local treatment effects. If the abstract indicated the grant dealt with any of the subtypes, the subtype code was applied. Klager and Tipton also coded if the grant mentioned development of software.

LEVEL OF INTERVENTION

Klager and Tipton also sought to understand the level at which an intervention was targeted. Coding the target of the grants from publicly available abstracts was difficult because ultimately, the authors acknowledge that virtually all IES grants seek to affect student outcomes. In many cases, even if the primary agent through which an intervention worked was someone other than the student, the outcome data used to measure impact

was collected from students. Also, it is quite common for studies funded in these categories to have multiple components that target different people. For example, a common occurrence is to have teacher professional development that is accompanied by a curriculum intervention for students.

In cases where an intervention was clearly targeted only or primarily at students, the grant was coded as targeting students. If an intervention had components that affected someone other than students (e.g., professional development for teachers) but those actors were merely delivering an intervention (e.g., a math curriculum) to students, the grant was coded as students as the primary target.

Grants were coded as targeting teachers if they were meant to change teacher practice but did not otherwise affect students except through the changes seen in the teacher. These are primarily tools for teachers or professional development programs that are not intended to train teachers on the use or delivery of a product/intervention to students. The “other” category includes interventions focused on parents, administrator and principals, schools, and school systems. As with teachers, interventions were coded as other if they were designed to affect one of the aforementioned actors and did not otherwise affect students, except through the changes induced in the targeted individual or institution. Coding for parents and administrators as the primary target of the intervention worked in much the same way as teachers; the intervention needed to focus on changing beliefs, skills, or behavior or providing tools for the parents or administrators rather than simply having the parents or administrators deliver the intervention.

For schools and school systems, it is not enough for the program to be delivered to all students or staff in a school or for the unit of randomization to have been the school. Grants targeted at schools and school systems change the structure of schools (e.g., implementing a Montessori model) or are policies that affect schools (e.g., a new accountability system for schools in a state). Using this coding scheme results in most interventions funded by both NCER and NCSER across Development and Innovation, Efficacy, and Replication/Effectiveness targeting students.

This same coding scheme was also used to organize Measurement grants. Abstracts were coded for mentions of various actors for which the measures might be targeted. These include students, teachers, or other actors including schools or school systems.

LIMITATIONS

While Klager and Tipton were able to download the data that form the basis of the paper from the IES website, they noted that these data are limited in that there are categorizations and details about grants that may or may not be present in the public abstracts. The public abstracts tend

to follow a format provided by IES, but it is still sometimes difficult to discern what a grant is about and what sorts of activities the researchers are engaged in. The fields that IES does provide are useful for categorizing by program, but there are many more fields that would clarify the types of grants IES has funded. More concrete categorizations would be useful instead of relying on principal investigators to include information in project abstracts.

REFERENCES