3

Prioritize Topics, Data Content, and Statistical Information to Maintain Relevance

This chapter describes a process that NCES can pursue to review its current and future priorities and to determine desirable changes in its data collections and products, illustrated through examples. The panel believes that making such choices now would be premature, and that these choices are best made by NCES as part of its strategic-planning process. The strategic plan and its implementation plan should also allow for changes over time to accommodate changing priorities or to meet immediate special needs. Information availability also changes over time, and the optimal approach for collecting data today may differ from the optimal approach 5 years from now.

This chapter also elaborates on the important high-value topics in education, discussing the data-content needs and areas in which NCES can advance those topics. While survey research has been NCES’s standard approach to data collection, some of these needs might best be met through administrative data and linkages to other data sources. Using such resources could limit respondent burden, possibly improve data quality, and maximize NCES’s effectiveness in this time of limited financial resources.

PRIORITIZING TOPICS

To determine topics to be given the highest priority, the panel first gathered information from stakeholders (see Appendix F and Chapter 1). The panel created a list of over 100 pertinent education topics and rated all topics based on predetermined criteria, by first and most critically examining the importance or value of the topic, and then examining the level of effort, or whether NCES could make progress on the topic (see Appendix C).

Topics were assigned a yes/no determination for all criteria. No attempt was made to assert equal value for the 19 criteria; however, the number of criteria satisfied can serve as a rough measure of the importance and/or feasibility of the topic. About half (48%) of the topics satisfied 8 or more of the 10 criteria with regard to importance, reflecting a broadly based need for many types of data. Many of the topics were interrelated; for example, there were multiple topics related to measuring equity.

The prioritization process was created to address NCES’s limited resources, which may necessitate hard choices going forward. A high rating in the panel’s prioritizing process reflects a broad consensus that a topic is important and satisfies multiple needs. However, NCES will need to consider its entire package of data products, which may lead to different conclusions than generated by the individual rankings. For example, NCES may discover duplication in its data products, even those of high importance, or a topic may appear low in importance but satisfy a key need. Decisions to keep or drop a data product should be made by NCES, ideally assisted by a consulting group, and based on NCES’s strategic plan.

The panel evaluated topics in terms of enduring priority—that is, topics that have been important and will continue to be important for the next 7 years. Thus, the presence of a COVID-19 learning gap received a relatively low rating (meeting 3 of the 10 criteria on importance) compared with more encompassing topics like equity, access, and technology. This does not mean that NCES should not collect information on timely topics like COVID-19. In fact, the panel views the NCES School Pulse Panel on COVID-19 as an excellent example of innovation and flexibility that needs to be more widely incorporated. NCES’s strategic plan should systematize decision making and prioritization of work while incorporating enough flexibility to adapt to sudden changes in strategic priorities, such as those posed by the COVID-19 pandemic.

ALIGN ACQUIRED DATA CONTENT WITH HIGH-PRIORITY TOPICS AND QUESTIONS

NCES performs, and has performed, a wide variety of data collections, ranging from relatively small surveys on highly specific topics (e.g., through the now discontinued Fast Response Survey System [FRSS] and the current School Pulse Panel) to large systems such as the National Assessment of Education Progress (NAEP), which includes surveys of principals, teachers, and students, along with assessments of student knowledge and skills, and which has expanded to support state-level estimates.

Some of NCES’s surveys have recently been discontinued due to the lack of staffing (e.g., FRSS and the School Survey on Crime and Safety [SSOCS]). NCES will continue to face hard choices about which data

collections to keep and which to drop, and these choices need to be based on the strategic plan.

RECOMMENDATION 3-1: NCES should conduct a top-to-bottom review of its data-acquisition activities, to prioritize topics most relevant to understanding contemporary education, and to discontinue activities that are disproportionately costly and burdensome relative to their value.

Prioritize Equity and Access Issues

The related concepts of equity, access to education, and opportunity to learn stand out in importance. Issues relating to equity are major news topics, important to federal, state, and local policy makers, and a major research focus. Inequality precludes full use of our nation’s human resources and negatively affects societal cohesion. Measures related to equity stood out among the 112 topic areas reviewed by the panel, with multiple measures satisfying all 10 criteria of importance: socioeconomic status, urban/rural/suburban location, race/ethnicity data, gender, English proficiency, mobility, disability, and “professional and academic areas in which Blacks are underrepresented.”¹

This section delves into issues relating to equity to illustrate how NCES might broadly review its measures to better align them with its priorities. This exercise also illustrates how reorganization within NCES could be beneficial; as we note elsewhere, NCES’s current structure is compartmentalized based on the primary data source or focus of individual surveys. We discuss organizational structure later in this report, and different structures have distinct strengths and weaknesses. Still, creating a comprehensive structure to review topics such as equity might help to facilitate increased collaboration across the surveys to, for example, share experiences in utilizing new data sources or addressing particular research topics. There is a need for a comprehensive examination of how the Center’s surveys address each topic area, helping to ensure that all surveys are compliant (or that noncompliance is intentional). By contrast, an organizational structure focused on the primary data source of individual surveys may leave important topics unaddressed or inconsistently handled.

Studies of equity require multiple types of measures: demographic measures, to identify groups that are of interest; process or implementation

___________________

¹ This topic is defined in law and does not mean that other underrepresented groups are unimportant. See 20 U.S. Code, Part B – Strengthening Historically Black Colleges and Universities, § 1061 Definitions. Available: https://www.law.cornell.edu/uscode/text/20/1061 [March 2022].

measures, to determine whether groups are treated differently; and outcome measures, to examine whether groups experience different outcomes.² NCES’s current strengths lie in measuring demographics and outcomes.

• Demographic data: It is routine for NCES surveys to contain basic demographic characteristics such as race/ethnicity and sex, and sometimes to include disability status and socioeconomic characteristics such as education levels and household income or poverty status.³ Further, the Common Core of Data (CCD) and Integrated Postsecondary Education Data System (IPEDS) measure demographic distributions within schools, districts, and higher education institutions. NCES might consider expanding its data collections to include other dimensions of inequality addressed by federal laws, which include national origin, sexual orientation, and gender identity. However, such data are considered more personal and intrusive, and may be harder to collect. Reactions to such unequal treatment may be easier to collect without requiring the assignment of labels to individuals.
• Outcome data: NCES also has multiple outcome measures, including assessments of students’ knowledge and skills (measured in NAEP and several longitudinal surveys), student retention, and degree attainment.⁴ Note that measures of outcomes are broader

___________________

² Another way of categorizing equity measures is as outcomes versus access, as in NASEM (2019a). This report uses slightly different categories because: (1) often there is value in having basic demographic information without a specific research question in mind, especially in public-use data files that are meant to support multiple research uses; and (2) some types of educational processes, such as imposing disciplinary actions, do not fit well as either outcomes or access. Access might be measured either through basic demographic measures (e.g., whether schools differ in their race/ethnicity compositions) or process (e.g., whether schools use tracking).

³ Income or poverty status is one of the most difficult characteristics to measure because such information is considered highly personal and confidential. Some surveys collect income levels using broad categories, making the request less intrusive. Often, eligibility for free or reduced-price lunches or school Title I status are used as surrogate measures of poverty levels, but participation in the free lunch program is lower for secondary-school students, making the measure less reliable at that level. At the postsecondary level, participation in Pell Grants is often used as an indicator of financial need, but students who are eligible may choose not to apply, and students’ levels of financial need may vary across colleges. The National Postsecondary Student Aid Study collects the most complete financial information, with finances as a primary focus. The National Education Longitudinal Study of 1988 and earlier studies also collected a household items index. Household items also can be used as a measure of cultural capital, which is another potentially important factor in education. The Education Longitudinal Study of 2002 asked for total family income using 13 categories.

⁴ NCES assessments are not discussed in detail here because they are being addressed in a separate study. However, they are relevant for creating a picture of the kinds of data needed and currently collected.

• than just final outcomes, and include disparities in academic readiness, self-regulation and attention skills, engagement in school, and performance in coursework (NASEM, 2019a). NCES also supports the Statewide Longitudinal Data Systems (SLDS) Grant Program through grants and performs annual surveys to monitor SLDS’s progress. SLDS contains both demographic and outcome data. The Program for the International Assessment of Adult Competencies (PIAAC)⁵ is both a source of data on the success of the education system with regard to literacy and a measure of need, particularly with regard to immigrant populations. Measuring outcomes is often difficult without longitudinal follow-up, and following up with students, especially after they leave college, can be difficult. A snapshot of one time point will lack data on later outcomes, such as graduation and job attainment.
• Implementation data: Measuring whether the education process is applied equitably is a significant weak point, but some data are available. Implementation data are often interrelated with issues of access, such as whether there are disparities in access to effective teaching, enrollment in rigorous coursework, and high-quality academic supports. Data from CCD and IPEDS can be used to examine whether students are unequally distributed across schools and colleges in terms of their demographic characteristics. Transcript data can be used to determine whether specific groups of students tend to follow different courses of study. The Parent and Family Involvement Survey collects data on parent involvement and school choice. NCES developed the Department of Education (ED) School Climate Surveys as survey instruments that schools, districts, and states can use to monitor school climate. Though the data do not belong to NCES, they could possibly be systematized to produce process information. Process data are difficult to collect because the process of interest may vary from one study to another and may consist of intangibles that are difficult to measure or are not routinely measured. There may also be questions about the accuracy of reporting, particularly for data viewed as either damaging or self-serving. Still, some types of data on school or district policies are relevant, such as whether a school uses tracking to separate high-achieving and low-achieving students. NCES has inconsistently collected data on tracking (e.g., in the 2017–2018 but not the 2020–2021 National Teacher and Principal Survey).

___________________

⁵ After a prepublication version of the report was provided to NCES, the program name was corrected throughout the document to reflect the Center’s current vehicle for collecting information on literacy.

• One way to expand data collection in this area is to collect subjective data. For example, do students (or staff) perceive that they are treated unequally based on their groups? How do students decide which courses to take or which careers to pursue? Other types of process data may be available from school or district data. Data on disciplinary actions (such as suspensions and expulsions) might be considered process data or outcome data, depending on research goals. NCES also reports on, but does not collect, data from the Campus Safety and Security Survey, conducted by the Office of Postsecondary Education within ED. These are aggregate data at the institution or campus level and might be considered partial measures of the campus environment.

The types of equity data needed depend on the application. Data sources such as CCD, SLDS, and IPEDS are valuable due to their comprehensive coverage. They can be used to determine the distribution of various demographic groups in schools and colleges, to create statistical or purposive samples for surveys or other research designs, or they can be merged with other types of data to add key equity data. Relating equity data to specific topics of interest, such as student achievement or literacy, is another application. NCES has several surveys of this type, including NAEP, Education Longitudinal Study, PIAAC, and SSOCS, though other surveys could be appropriate depending on the analytic goal. Finally, one might consider creating a data source specifically focused on equity. Such a source might explore attitudes towards equity (including both self-perception and attitudes towards others) and experiences relating to equity within the educational environment.

Data can also be classified as objective, record-based data; other objective data; or subjective data. Some researchers prefer one data type over another, but each type has advantages and disadvantages.

• Objective, record-based data include attendance records, health records, course transcript data, and other administrative data. Record-based data may not be in electronic format, and even when they are, the data may be divided among multiple databases and with varying formats, even within a single school or district. There is a common perception that electronic data can be readily processed, but considerable work may be required to format and prepare data before they are ready for analysis, and records may not contain all the data that are needed (e.g., because the person maintaining the data may have different uses for it than a researcher does). One role for NCES could be to promote standardization of both which data are stored and how record data are maintained, so that data may be more easily shared and processed.

• Other types of objective data are not record based. These might include school policies such as those regarding disciplinary actions. Some of these data might be available electronically, perhaps as posts on a school website, but not stored in databases or in a format that can be readily analyzed statistically. The technological tools for web scraping and data mining are progressing rapidly and may ultimately provide a useful means for collecting and analyzing web-based data. However, it may be less expensive and more reliable to collect such data through a survey using statistical sampling.
• Subjective data are often dismissed as less reliable but may be the only source of certain data. There are also situations in which subjective data are directly useful. For example, many incidents of unequal treatment will never be reported on official databases, and self-reports can augment data available on databases. Further, even if self-reports are incomplete or reflect misperceptions on the part of the person reporting unequal treatment, the perception may itself be a matter of interest.

SLDS is a tremendous resource supported and monitored by NCES, though the Center is not involved in data collection or maintenance. The Common Education Data Standards program is critical in setting standards for state data systems and could be a vehicle through which NCES could lead on equity-related efforts, by helping states collect more disaggregated data on race/ethnicity, gender identity, and other data on populations of interest. Additional types of data could be used to address equity issues. U.S. states and local governments have the primary responsibility for elementary and secondary education, so they, not the federal government, are often the best source of program data. However, the Elementary and Secondary Education Act (especially Title I) and the Individuals with Disabilities Education Act involve the federal government in elementary and secondary education, and the Higher Education Act (especially Titles III and IV) involves the federal government in postsecondary education, so federal program data could be a rich resource. NCES might also contribute by setting standards for school/district/college databases, in terms of contents and definitions. Even if the standards are voluntary, producers and purchasers of database software might view the standards as an important target which, over time, will increase the shareability of data.

Collecting Data on All Levels of Education

Data covering all levels of education (early childhood, elementary and secondary, higher education, and adult education) are necessary for a complete picture of the educational process, and for understanding how various

parts are interrelated.⁶ Generally, NCES has been strongest when measuring traditional education and at traditional ages, but a longer and more comprehensive time frame is needed. A student’s difficulty at one educational level may result in continuing or greater difficulties at higher levels and may lead to the need for greater compensatory strategies in later years. For example, postsecondary education is increasingly involved in remedial or developmental education. To fully measure the impact of education, it is also helpful to measure posteducation outcomes such as employment and income.

The panel finds that topics in early childhood education are particularly important, such as access to early childhood education and state and local agencies’ early childhood school readiness activities. In 2001, NCES conducted the Early Childhood Longitudinal Study (ECLS) with a birth cohort, but more recent ECLS cohorts have started with kindergarten, leaving a data gap for younger children.⁷ NCES also studies early childhood through the National Household Education Survey (NHES). Research on early childhood is challenging because much of early childhood falls outside of the standard education infrastructure, and is thus less amenable to standard research approaches. Young children may be at home, with friends or relatives, or in daycare centers. Besides household surveys such as NHES, NCES might consider studies of the regulatory and certification structures set up to monitor and improve daycare, and may wish to create linkages to data from public-assistance programs, since economically disadvantaged children are a major focus of interest. (Creating linkages is discussed in greater detail later in this chapter.)

Career and technical education (CTE) is an often-ignored area that is increasingly important as the U.S. addresses workforce issues. NCES measures CTE that occurs within high schools and traditional postsecondary education but lacks robust data on the growing number of noncredit and certificate programs for adult learners that are operated by community colleges and private providers. With CTE, NCES may also need to broaden the types of data it collects; for example, retention and attainment of a degree or certificate may be less important than measuring participants’ success in obtaining jobs or upgrading occupational skills. Sometimes a single course or sequence of courses may be all that is needed.

Related to, but broader than, CTE is adult education, including adult literacy, and NCES’s efforts in this area need strengthening. IPEDS does not collect data on noncredit courses in higher education, and much of adult education falls outside of the traditional education infrastructure. PIAAC

___________________

⁶ Gabriela Katz, “StriveTogether Cradle to Career Network,” presentation to the National Academies of Sciences, Engineering, and Medicine, August 6, 2021.

⁷ See: https://nces.ed.gov/ecls/ [March 2022].

and the NHES are NCES’s primary vehicles for examining the education status of the adult population. Much information is needed, including the number of non-Workforce Innovation and Opportunity Act programs, where programs are located, what services are provided, how instruction is delivered (i.e., online, in classrooms, or a hybrid), and what outcomes are attained, as well as adult-learning demographics, practitioner demographics, and funding methods.⁸ Additionally, more data are needed on adult learners, such as their motivations, resilience, self-regulation, sensory difficulties, mental illnesses, cognitive challenges, and chronic health issues. Additional information is needed about adult learning programs, describing existing resources, instructors’ backgrounds, instructors’ employment status (e.g., full-time, part-time, volunteer), and the professional development instructors receive. Education statistics need to be crafted to capture the fact that the student population is older than it was in the past, and that adults comprise a larger fraction of all levels of schooling beyond high school.

Certain populations are of special interest, including the homeless, the incarcerated and those on parole, and English-language learners who are not literate in their native languages. Some such education occurs within the secondary school system, but it also falls within adult education and career and technical education.

One tool for providing a comprehensive view of education is through longitudinal studies, which can monitor individual students’ progress through the education system and transitions into the workforce. These studies can incorporate multiple components (i.e., of students, parents, teachers, and administrators), while also measuring change over time and student outcomes, such as test scores and academic progress. The data can be used to develop and investigate theories about what makes education effective or ineffective. However, new sequences of surveys start relatively infrequently, and sometimes they are not well timed for monitoring trends or current education issues. NCES might explore ways of more frequently updating surveys to include current topics, even if they do not fit the longitudinal structure. Sometimes others might pay NCES to add modules on special topics, either incorporating a module within a survey (as the National Science Foundation did when adding a teacher transcript request form to the National Education Longitudinal Study of 1988, or by conducting a follow-up survey (as performed by the University of Texas at Austin when conducting a midlife follow-up of High School & Beyond respondents).⁹

___________________

⁸ Daphne Greenberg, “A Vision and Roadmap for Education Statistics,” presentation to the National Academies of Sciences, Engineering, and Medicine, August 23, 2021.

⁹ See: https://sites.utexas.edu/hsb/ [March 2022].

Linking to Other Types of Data Relevant to Education

NCES’s longitudinal studies provide one kind of linkage—linking data on students over time—but linkages to other databases could also be useful by expanding the types of data that are available, sometimes increasing accuracy by using administrative data rather than self-reports, while lessening respondent burden. Students often receive services outside of schools (sometimes through referrals from the schools) that affect their success in education, and linking to measures of such services provides a more complete picture of the students’ situations. For example, 22 percent of states link child care data with social-services data and 16 percent link it with health data.¹⁰ At another level, 30 percent of states link early childhood education programs with workforce data to examine issues such as supply and demand, professional development, and supports to retain an effective early-childhood-education workforce).¹¹ Additional sources of administrative data on children and families for linkages can be found in an Administration for Children and Families report (Holman et al., 2020).

Linking data might also involve creating partnerships with other agencies and levels of government using alternative types of data. For example, ED collects a great deal of information about family finances through its grant and loan programs (some of which is used by the National Postsecondary Student Aid Study [NPSAS]), and merges student-aid-participant data with Internal Revenue Service tax data to produce employment-outcome statistics for the College Scorecard. Other high-value statistical products could be developed with other datasets. For example, the Department of Health and Human Services has individual-level information on Medicaid, Medicare, and quarterly earnings and unemployment insurance (in the National Directory of New Hires) and aggregate information on other human-services programs administered by states. The Department of Agriculture holds aggregate data on participants in the Supplemental Nutrition Assistance Program and school lunch programs that states administer. The Department of the Treasury’s Office of Tax Analysis has expressed interest in collaborating with other federal agencies, and its data, along with data related to unemployment insurance, might be used to better link education with workforce outcomes. The U.S. Department of Labor maintains a Registered Apprenticeship Program, which may provide a valuable supplement to the data that NCES collects from schools and colleges. Privacy concerns can occur when data are shared across agencies, but mechanisms for protecting privacy can be employed.

___________________

¹⁰ Carlise King, “Early Childhood Data Integration Goals and Trends,” presentation to the National Academies of Sciences, Engineering, and Medicine, August 6, 2021.

¹¹ Ibid.

NCES might also consider ways of making its data more linkable to support the work of others. For example, conducting evaluations (other than on methodological issues) is outside the scope of NCES, but NCES survey data might be used to better support the extraction of specialized subsets of schools or colleges that could be used in evaluations. NCES already supports such work in the sense of providing CCD and IPEDS, but NCES also collects more detailed information about school practices in survey systems such as the National Teacher and Principal Survey (NTPS), SSOCS, ECLS, and the High School Longitudinal Study of 2009 that might be useful to researchers desiring more detailed data about schools when selecting study participants. Allowing the use of these survey systems to support sampling would raise issues of access to restricted-use data, and NCES would need to review its systems for protecting confidentiality.

Document the Broader Educational Environment

Much of what happens in education is affected by the educational environment, which might be broadly defined as including administrative infrastructure (e.g., workforce development, curricula, finance and management, and school context), educational tools (e.g., use of technology and online teaching), and comparative data (e.g., international data). NCES is already actively conducting research in these areas, but much remains to be done.

Administrative Infrastructure

A core task for the education system is the development and maintenance of the teaching workforce and supplemental staff, which includes teacher training, recruiting, hiring, school placement, addressing turnover across schools, and improving retention in education. Teacher education and skills, credentials, and certifications—especially for high-need areas such as special education—are all very important. Other high-priority topics for NCES include teacher and staff diversity, the representativeness of teachers and staff of the community and school population, the teacher pipeline, and teacher compensation. NCES already collects data in these areas through the NTPS and the Beginning Teacher Longitudinal Study; additionally, Baccalaureate and Beyond includes questions for those interested in becoming teachers. NCES has tracked the representativeness of teachers in its reports (NCES, 2020), but more data on the training and hiring of teachers would be useful.

Curricula are another important part of the administrative infrastructure. Curricula are a challenging area to study because they vary from state to state and between schools. Still, this variation presents an opportunity for research. NCES can serve an important role in curricular studies in

several ways. First, it can continue to create and update classification systems, which help to systematize data from school transcripts and allow studies of course taking on topics such as science, technology, engineering, and math (STEM) education and career and technical education/vocational education.

Second, NCES can continue to produce detailed curricular and pathway data through studies such as the High School Transcript Study (HSTS). These data allow researchers to examine students’ progress through secondary and postsecondary education, and support policy analysis (e.g., examining the impact of high school graduation requirements on students’ course taking and achievement [Chaney et al., 1997]). The alignment of high school mathematics curricula with existing labor-market needs is an example of an area in which detailed curricular and pathway data would be helpful. Some have expressed concern that the mathematics skills taught in the algebra-geometry pathway in years 3 and 4 of high school are little used by most people, while the mathematics of data science is regularly used and, if more fully incorporated into high school curricula, could help many students build more useful labor market skills.¹² Studies such as HSTS help to measure which pathways students follow and allow researchers to evaluate the outcomes of those various pathways.

Third, as in the Trends in International Mathematics and Science Study, NCES can support deeper investigations of curricula, not only measuring which courses students take, but also delving deeper into topics including how advanced the curricula are, the time spent on the curricula, and the degree of focus on topics. Such data can also be used to address equity, including whether curricular changes might ameliorate the under-representation of women and minorities within STEM.¹³

Another component of administrative infrastructure deserving additional NCES attention is funding and expenditures for students, K–12 schools, and postsecondary institutions. NCES collects finance data on states and school districts (but not individual schools) through the U.S. Census Bureau’s F-33 survey,¹⁴ which is released in the CCD; on college costs through IPEDS; and on student finances through NPSAS. NCES could deepen its finance and expenditure data collection to investigate the impact of various school-funding formulas and approaches, resource disparities, and effective resource use. In addition, there is a growing focus on the cost of postsecondary education, and students’ decisions on whether to attend college and which colleges to attend are greatly affected by the cost

___________________

¹² Jo Boaler, “Mathematics Is a Subject in Need of Change,” presentation to the National Academies of Sciences, Engineering, and Medicine, August 23, 2021.

¹³ Ibid.

¹⁴ See: https://nces.ed.gov/ccd/f33agency.asp [March 2022].

of higher education. NCES could do more to capture data on the cost of postsecondary education, and the Center might also consider collecting school-level finance data.

Finally, NCES could collect additional data on school context, which involves many concepts related to students’ learning environments. For example, school context data includes state policy contexts, school sectors or types (e.g., charter, magnet, private, virtual, and traditional public schools), school climate, geospatial differences (such as distance to students, geographic location [e.g., state], and urbanicity). Schools may also employ tools such as wraparound services, trauma-informed discipline, student engagement in discipline, peer models, and peer practice, which all contribute to school context. Some aspects of school context are so diverse that it is difficult for surveys or other traditional data systems to properly characterize them. Additional research is required in these areas to determine the types of data that would be most useful and the best modes to obtain them.

Note that topics involving administrative infrastructure are often interrelated with equity issues—a top priority. Schools with high poverty levels tend to have teachers who are less experienced (Gagnon and Mattingly, 2012), and higher turnover rates (NCES, 2021d). They also tend to have fewer financial resources and learning environments that are less supportive. Thus, collecting data on these topics helps with the goal of addressing equity.

Acquiring and Using Appropriate Tools

Access to and use of technology is another highly important area, and one that is rapidly changing. From 1994 to 2000, NCES conducted annual surveys of public schools to measure their access to the internet, at which point access had become almost universal (98%); similarly, the ratio of students to instructional computers decreased to 5:1, though schools with the highest poverty levels had fewer computers per student with access to the internet (Cattagni and Farris, 2001). By 2020–2021, 45 percent of schools reported having a computer for each student (Gray and Lewis, 2021). While access to computers and the internet has been increasing rapidly across the U.S. population, the types of equipment and access among the poor, both at the elementary/secondary level and the postsecondary level, continues to be an equity issue. The panel applauds NCES’s efforts to monitor issues such as access and recommends that such research be continued with refined measures to better document individual student differences.

The uses of technology have also changed greatly. While schools may once have had computers in the absence of the training or tools to make full use of them, in 2020–2021, 47 percent of schools reported that their teachers

used technology for classroom work that would not otherwise be possible (Gray and Lewis, 2021). NCES may both benefit from and be of value to other federal agencies that are conducting technology-related research. For example, the Office of Planning, Evaluation and Policy Development within the U.S. Department of Education sponsored a study of digital-learning resources for instructing English learners (Zehler et al., 2019), and NCES could both advise other federal agencies on research approaches and adapt its own approaches based on what is learned. Professional development is another important technology-related issue. As technology continues to advance and new tools are developed, teachers will need to be instructed in their use. NCES has been active in measuring the use of technology, through both specialized surveys and more general surveys that include questions on technology use. This is an area in which NCES will need to be nimble to stay current, as old questions become outdated and new questions develop.

In addition to being a subject of research, technology can also be a tool for collecting and disseminating new types of data. When students interact with digital learning tools, data can be collected to monitor student learning and what facilitates it. NAEP collects data on digital NAEP test-takers, measuring how long students spend on each question and how that correlates with response accuracy.¹⁵ IES has created a competition called XPRIZE to encourage the collection and analysis of such data,¹⁶ and supports SEERNet as another research tool.¹⁷ Though NCES does not conduct evaluations, it can and should create data systems to facilitate research.

Another aspect of technology worthy of special attention is the rapid transition to online education. Even prior to the COVID-19 pandemic, online education was growing rapidly in postsecondary and adult education, and this growth has further increased with the pandemic, making inroads into elementary and secondary education (Lederman, 2021; NCES, 2021g; U.S. Census Bureau, 2021f). Though online education may not work for all students, some students and parents prefer it, and some school districts are opting to create permanent online options (Lurye, 2021).

Online education raises a host of issues. For example, online education can improve equity by making content, courses, and modes of instruction more broadly available, but this potentially positive impact may be muted or inequity may even increase if disadvantaged groups lack the technology, expertise, or infrastructure to make full use of online education. Online education also changes the way education is performed: at the simplest level, this might include the ability to rewatch a lecture or teaching session; it affects how test security is enforced and how cheating is prevented; and it

___________________

¹⁵ See: https://www.nationsreportcard.gov/process_data/ [March 2022].

¹⁶ See: https://www.xprize.org/challenge/digitallearning [March 2022].

¹⁷https://seernet.org/ [March 2022].

may eventually allow virtual reality to replace student labs. NCES will need to stay nimble to monitor the key issues that arise. Currently, NCES collects relatively little information on online education. Some topics concerning online education are evaluative in nature and outside of NCES’s mandate; however, NCES can actively monitor online education that is currently in place, assess how extensively it is used, and determine who uses it.

Developing Cross-State and International Comparisons of the Educational Environment

The U.S. system of delegating education to states and localities results in tremendous diversity in curricula, policies, and practices. Although that diversity may sometimes complicate education research, it also represents an opportunity for experimentation. By tracking the diversity of educational environments, NCES can both provide preliminary data that might be used to examine the impacts of various policies and practices, and also help researchers to design specialized samples for investigating education issues. To date, NCES has not been strongly involved in monitoring state and local policy differences, though NAEP is designed to provide state-level statistics for some states and these data have supported research on education reform (Grissmer et al., 2000). NCES also supports the SLDS. By increased monitoring of differences across states and localities, NCES can supply valuable data.

International studies expand researchers’ abilities to make comparisons across cultures and education systems. The panel applauds NCES’s participation in international studies. International comparisons also provide important challenges. For example, some countries sample in ways that generate a misleading view of an entire nation. Additionally, the United States is much more heterogeneous than many other countries, and there are important differences in the way education is managed between countries (i.e., the United States has no national curriculum, and states, localities, and private institutions set their own standards and agendas). When participating in international studies, NCES needs to document these types of differences and indicate how they interact with the research findings.

Congressional Mandates

Almost half of the topics examined (52 topics, or 46%) are mandated by Congress as areas for NCES to collect data. These mandates place obvious constraints upon NCES, and they do not always correspond with the panel’s ranking of importance. Some of these mandates extend back for decades, and NCES may wish to explore with Congress whether needs for some types of data have changed, perhaps because of changing priorities

or because better alternatives are now available. NCES might also reexamine whether some data collections are overly comprehensive. There may be situations in which data collections on mandated topics could be reduced in scope, still meet the mandates, but leave more resources for other topics.

CONCLUSION 3-1: Congressional mandates constrain NCES’s data collection priorities yet may no longer reflect what is important for understanding contemporary education.

RECOMMENDATION 3-2: NCES should revisit priorities mandated by Congress and, where appropriate, make recommendations for changes.