Appendix B
Data Sources and Collection Approaches
This appendix describes selected data sources relevant to this study, including traditional data sources already used by NCES. The primary legacy data source for statistical agencies is data that can be derived only through probability sample surveys, such as longitudinal and cross-sectional surveys. Administrative data have also been extensively used for statistical purposes by federal statistical agencies including NCES, but expansion of their use has been encouraged by the U.S. Office of Management and Budget (U.S. OMB, 2014a). New sources of data include commercial data (available for purchase), data available through web scraping, wearable recording devices, transcribing of video and/or audio recordings, and others. Here, we refer to alternative data sources to include administrative data (because new uses are emerging) in addition to commercial/proprietary and web-scraped data, data available from transcription of video and audio recordings, as well as sources/methodologies that may lead to new data in the future. We provide definitions as used in this report; they are not meant to be definitive descriptions for the field of statistics.
This report recommends that NCES expand its use of alternative data sources and new collection methodologies. This has been an active area of research for all statistical agencies in recent years as online data sources proliferate and novel ways to use data sources emerge. This is part of a natural evolution and modernization. New data sources and approaches may provide cost-effective ways to counter some of the challenges with the traditional sample survey approach (e.g., surveys are expensive and time-consuming, respondents may find them burdensome, and achieving high response rates has become more challenging). However, alternative data
sources have their own challenges, and creative methods are needed for evaluating and using them.
PROBABILITY SAMPLE SURVEYS
In probability sample surveys, some or all members of a population are selected to participate, each with a known probability of selection. When all members are selected, the probability of selection is one and the survey is called a census. An advantage of probability sample surveys is that the tabulated responses represent the entire population, and a measure of accuracy can be calculated. Probability-based surveys form the bulk of government data collections.
Two common types of surveys are cross-sectional, in which respondents are surveyed at a single point in time, and longitudinal, in which the same respondents are surveyed over multiple points in time with the intention of measuring change over time. A third alternative is experience sampling, in which the time period is also part of the sampling. Experience sampling has been useful for capturing “in-the-moment” states of mind and responses to ongoing situations (e.g., a class). Shernoff et al. (2014) provide an interesting example.
Surveys can incorporate a variety of measurements in addition to those obtained from questionnaires: tests or assessments, record collection, classroom observations, and physical measurements, such as blood pressure. Other measurement approaches made possible by technology include studies that involve video (and its coded version), audio (e.g., of classrooms), and text (e.g., syllabi and lesson plans). Hiebert et al. (2005) provide a useful example of a study using video recordings.
As illustrated in Table B-1, NCES has an extensive history of repeated longitudinal surveys, especially of students in grades K–5, secondary, postsecondary (and beyond) NCES longitudinal surveys typically include interviews with students, parents, and teachers, as well as administrative data and transcript study results, based on coding and summarizing transcript information in a consistent way. Additional value is derived from these surveys by the use of selected follow-on studies (sometimes funded by other parties) that target longitudinal survey participants with additional questions at a later date. The follow-on studies of the High School and Beyond (1982) sophomore cohort conducted in 2015 and 2021 are one example.1
___________________
1 For additional information, see: https://sites.utexas.edu/hsb/ [March 2022].
TABLE B-1 NCES Public Data Sources (Selected)‡
| Level of Education | NCES Products | Survey, Administrative or Other | Characteristics, Topics, etc. | Frequency, Lag Time, etc. | Link |
|---|---|---|---|---|---|
| Pre-K | Early Childhood Longitudinal Survey, Birth Cohort (ECLS-B) | Longitudinal survey | Longitudinal survey of children from birth through K. | Initial collection in 2001. Four interviews at age 9 months, 2 years, preschool, and kindergarten. | https://nces.ed.gov/ecls/birth.asp |
| K–12 | Early Childhood Longitudinal Survey, Kindergarten Cohorts (ECLS-K) | Longitudinal survey | Longitudinal surveys of children from K through grade 8 (1998–99), or from K through grade 5 (2010–11, and 2023–24) | Survey first fielded for grades K–8 beginning in 1998–99, K–5 in 2010–11, and 2023–24. | https://nces.ed.gov/ecls/ |
| Level of Education | NCES Products | Survey, Administrative or Other | Characteristics, Topics, etc. | Frequency, Lag Time, etc. | Link |
|---|---|---|---|---|---|
| K–12 | School Crime Supplement (SCS) to the Bureau of Justice Statistics National Crime Victimization Survey (NCVS) | Cross-sectional survey | NCVS is a household survey that collects information from household members age 12 or older about crime victimization within the last 6 months. The School Crime Supplement (SCS) collects information from students age 12 to 18 about victimization, crime, and safety at school (public, private elementary, middle, and high schools). The SCS asks about school-related topics such as alcohol and drug availability; fighting, bullying, and hate-related behaviors; fear and avoidance behaviors; gun and weapon carrying; and gangs at school. | As of September 2021, SCS data were available for 1989, 1995, 1999, 2001, 2003, 2005, 2007, 2009, 2011, 2013, 2015, 2017, and 2019. | https://nces.ed.gov/programs/crime/surveys.asp |
| K–12 | School Survey on Crime and Safety (SSOCS) | Cross-sectional survey | SSOCS is a sample survey of the nation’s public schools designed to provide estimates of school crime, discipline, disorder, programs, and policies. SSOCS is administered to public primary, middle, high, and combined school principals in the spring of even-numbered school years. Due to staffing and funding issues, SSOCS 2022 will be the final collection | Data collected every other year in even numbered years. Most recent data product available in September 2021 was from 2017–2018. However, data used in Report on Indicators of School Crime and Safety (Irwin et al., 2021) | https://nces.ed.gov/surveys/ssocs/ |
| K–12 | Middle Grades Longitudinal Study (MGLS2017) | Longitudinal survey | The Middle Grades Longitudinal Study of 2017–18 (MGLS:2017) follows a nationally representative sample of students as they enter and move through the middle grades. The study is focusing on student growth in mathematics and literacy skills. | Two rounds of collection: first round 2018 (students in 6th grade in 2017), second round 2020. | https://nces.ed.gov/surveys/mgls/ |
| K–12 | Secondary Longitudinal Sample Surveys | Longitudinal surveys | A series of six longitudinal studies following middle- or high-school students through school and sometimes beyond. Depending on the survey, the data include surveys of students, parents, teachers, school administrators, student assessments in math and English, and high-school transcripts. NLS:72, HS&B:82, NELS:88, ELS:2002, HSLS:09 and HS&B:22. | New longitudinal series are periodically created to provide updated data. The most recent are ELS:2002, following 10th graders in 2002 and 12th graders in 2004 through secondary and postsecondary years | https://nces.ed.gov/surveys/els2002/ |
| Level of Education | NCES Products | Survey, Administrative or Other | Characteristics, Topics, etc. | Frequency, Lag Time, etc. | Link |
|---|---|---|---|---|---|
| K–12 | Fast Response Survey System (FRSS) | Survey system | This system has conducted multiple surveys per year on special topics sometimes requested by NCES and sometimes by other ED agencies using this as a vehicle. The survey respondents have included public and private elementary and secondary schools, elementary and secondary school teachers and principals, local education agencies, public libraries, and school libraries. The system ended recently because of staffing issues at NCES. FRSS was part of a larger project, the Postsecondary Quick Response Information System, which included surveys of postsecondary institutions. | Surveys were conducted on an as-requested basis, sometimes on entirely new topics and sometimes with essentially the same survey repeated over multiple years. The report for the last survey, FRSS 110, was released in November 2021. | https://nces.ed.gov/surveys/frss/ |
| K–12 | Private School Survey (PSS) | Cross-sectional survey | PSS is a biennial universe collection of private elementary and secondary schools. PSS generates biennial data on the total number of private schools, teachers, and students and builds an accurate and complete list of private schools to serve as a sampling frame for NCES surveys of private schools. Information collected includes: religious orientation; level of school; size of school; length of school year, length of school day; total enrollment (K–12); number of high-school graduates, whether a school is single-sexed or coeducational and enrollment by sex; number of teachers employed; program emphasis; existence and type of kindergarten program. | The PSS began with the 1989–90 school year and has been conducted every 2 years since. The most recent data files available in September 2021 were from 2017–18. | https://nces.ed.gov/surveys/pss/ |
| K–12 | National Teacher and Principal Survey (NTPS) | Cross-sectional survey | NTPS collects extensive data on American public and private elementary and secondary schools every 2 to 3 years. Teachers, principals, and schools are components of the NTPS survey system. NTPS provides data on characteristics and qualifications of teachers and principals, teacher hiring practices, professional development, class size, and other conditions in schools. NTPS replaces the Schools and Staffing Survey (SASS) which was last conducted in the 2011–12 school year. | Administered in years 2015–16, 2017–18, 2020–21, replaced the Schools and Staffing Survey (last conducted in 2011–12) | https://nces.ed.gov/surveys/ntps/ |
| Level of Education | NCES Products | Survey, Administrative or Other | Characteristics, Topics, etc. | Frequency, Lag Time, etc. | Link |
|---|---|---|---|---|---|
| K–12 | School Pulse Panel (SPP) | Survey, panel | SPP is a monthly panel study to look at the impact of the coronavirus pandemic on K–12 public schools. It will produce nationally representative data with a quick turnaround. Content will be revised quarterly. SPP is conducted by the U.S. Census Bureau on behalf of NCES. | Clearance requested from OMB in June 2021 | https://www.census.gov/programssurveys/school-pulse-panel.html |
| K–12 | Common Core of Data (CCD) | Survey to collect aggregates from administrative records | CCD is a comprehensive, national database of all public elementary and secondary schools and school districts. Aggregate data reported via EDFacts by state and local education agencies, or schools based on admin strative records. Fiscal data collected by the Census Bureau. | Annual nonfiscal data for 2020–21 released June 28, 2021. Annual fiscal data for 2017–18 available in September 2021. | https://nces.ed.gov/ccd/ |
| K–12 | Education, Demographic and Geographic Estimates (EDGE) | A special tabulation of the U.S. Census Bureau’s American Community Survey (ACS) data | The EDGE program uses data from the U.S. Census Bureau’s American Community Survey to create indicators of social, economic, and housing conditions for school-age children and their parents for school districts. It uses spatial data collected by NCES and the U.S. Census Bureau to create geographic locale indicators, school point locations, school district boundaries, and other types of data to support spatial analysis. | Updated annually based on 5-year ACS. As of September 2021 data released for 2005–2009 through 2015–2019. Data can be used to link school district-level aggregates of characteristics of school-age children, the parents of school-age children, and the total population to survey data. | https://nces.ed.gov/programs/edge/ |
| K–12 | National Assessment of Educational Progress (NAEP) is a sample survey combined with an assessment. | Survey with assessment | NAEP, also known as “the Nation’s Report Card,” is the only nationally representative and continuing assessment of what America’s students know and can do in various subject areas. Since 1969, assessments have been conducted periodically in reading, mathematics, science, writing, U.S. history, civics, geography, and the arts. In addition to tests of students’ knowledge, NAEP includes surveys of students, teachers, and principals. | Reading and mathematics every 2 years, and other subjects periodically. | https://nces.ed.gov/nationsreportcard/ |
| Level of Education | NCES Products | Survey, Administrative or Other | Characteristics, Topics, etc. | Frequency, Lag Time, etc. | Link |
|---|---|---|---|---|---|
| K–12 | High School Transcript Studies (HSTS) | Other, a system of collecting and coding transcript data for inclusion in surveys | NCES’s HSTS collect information that is contained on the student high school record—i.e., courses taken while attending secondary school; information on credits earned; year and term a specific course was taken; and final grades. When available, information on class rank and standardized scores is also collected. Once collected, information (e.g., course name, credits earned, course grades) is transcribed and standardized (e.g., credits and credit hours standardized to a common metric) and can be linked back to the student’s questionnaire or assessment data. NAEP focuses on grades 4, 8, and 12. | Periodic, depending on the survey. | https://nces.ed.gov/surveys/hst/ |
| K–12 | Civil Rights Data Collection (CRDC) | Collection from administrative records | The U.S. Department of Education (ED) Office for Civil Rights (OCR) collects data from local education agencies on key education and civil rights issues in our nation’s public schools through the CRDC. The CRDC collects a variety of information including student enrollment and educational programs and services, most of which is disaggregated by race/ethnicity, sex, limited English proficiency, and disability. | School-level data collected in 2000, 2004, 2006, 2009–2012, 2011–12, 2013–14, 2015–16, 2017–18, 2019–20 (as of September 2021). Recent collections cover all public schools. Information collected by the CRDC is used by OCR as well as other ED offices, policy makers and researchers outside of ED. | https://www2.ed.gov/about/offices/list/ocr/data.html |
| K–12 | Common Education Data Standards (CEDS) | Other, standards | CEDS are a national, collaborative effort to develop voluntary, common education data standards for a key subset of K–12 (e.g., demographics, program participation, course information) and K–12-to-postsecondary education transition variables. The intention is to facilitate the use of common definitions across state data systems. | https://nces.ed.gov/programs/ceds/ |
| Level of Education | NCES Products | Survey, Administrative or Other | Characteristics, Topics, etc. | Frequency, Lag Time, etc. | Link |
|---|---|---|---|---|---|
| K–12 | Classification of Secondary School Courses (CSSC) and Codes for Exchange of Data (SCED) | Other, classification | To analyze student transcript data, NCES developed the CSSC using data from the initial transcript collection in HS&B. In 2007, NCES released SCED, a course classification system designed to facilitate schools’ and districts’ maintenance of secondary-level transcript data over time and transfer of those data among districts and states. | This taxonomy was also used to code courses from high-school transcript studies throughout the 1980s, 1990s, and 2000s. | See https://nces.ed.gov/pubs2019/2019417.pdf for methodological report. |
| K–12 | Beginning Teacher Longitudinal Study (BTLS) | Longitudinal survey | BTLS was a study of a cohort of beginning public school teachers initially interviewed as part of the 2007-08 Schools and Staffing Survey through the 2011–12 school year. The study was intended to create an unfolding “story” by following this cohort of first-year teachers. | One cohort. Data collected in 2007–08, 2008–09, and 2010–11, 2011–12, and 2012–13. Web says data were released in 2015. But web only has data for the first 3 waves. | https://nces.ed.gov/surveys/btls/ |
| Universities and colleges | National Postsecondary Student Aid Survey (NPSAS) | Survey | The NPSAS is a comprehensive study that examines how students and their families pay for postsecondary education. It includes nationally representative samples of undergraduate and graduate students, as well as students attending public and private less-than-2-year institutions, community colleges, 4-year colleges, and major universities. Both students who receive financial aid and those who do not receive financial aid participate in NPSAS. NPSAS has been conducted every 3 to 4 years since 1987. Student interviews and administrative records are used to provide exceptional detail concerning student financial aid. The latest data are available for the 2015–16 academic year. Data collection for 2019–20 has ended and data are currently being processed. | Conducted every 3 to 4 years since 1987. Data for 2019–20 ended in January 2021. | https://nces.ed.gov/surveys/npsas/ |
| Level of Education | NCES Products | Survey, Administrative or Other | Characteristics, Topics, etc. | Frequency, Lag Time, etc. | Link |
|---|---|---|---|---|---|
| Universities and colleges | Beginning Postsecondary Students (BPS) | Longitudinal survey | Each cycle of BPS follows a cohort of students who are enrolled in their first year of postsecondary education. The study collects data on student persistence in, and completion of, postsecondary education programs (including postsecondary transcript studies), their transition to employment, demographic characteristics, and changes over time in their goals, marital status, income, and debt, among other indicators. BPS tracks students’ paths through postsecondary education and helps answer questions of policy interest, such as why students leave school, how financial aid influences persistence and completion, and what percentages of students complete various degree programs. |
|
https://nces.ed.gov/surveys/bps/ |
| Universities and colleges | Baccalaureate and Beyond Longitudinal Study (B&B) | Longitudinal survey | B&B examines students’ education and work experiences after they complete a bachelor’s degree, with a special emphasis on the experiences of new elementary and secondary teachers. Following several cohorts of students over time, B&B looks at bachelor’s degree recipients’ workforce participation, income and debt repayment, and entry into and persistence through graduate school programs, among other indicators. It addresses several issues specifically related to teaching, including teacher preparation, entry into and persistence in the profession, and teacher career paths. B&B also gathers extensive information on bachelor’s degree recipients’ undergraduate experience, demographic backgrounds, expectations regarding graduate study and work, and participation in community service. |
|
https://nces.ed.gov/surveys/b&b/ |
| Level of Education | NCES Products | Survey, Administrative or Other | Characteristics, Topics, etc. | Frequency, Lag Time, etc. | Link |
|---|---|---|---|---|---|
| Universities and colleges | Integrated Postsecondary Education Data System (IPEDS) | Survey that collects administrative records data | IPEDS, established as the core postsecondary education data collection program for NCES, is a system of surveys designed to collect data from all primary providers of postsecondary education. IPEDS is a single, comprehensive system designed to encompass all institutions and educational organizations whose primary purpose is to provide postsecondary education. The IPEDS system is built around a series of 12 interrelated surveys to collect institution-level data in such areas as enrollments, program completions, faculty, staff, finances, and academic libraries. | Data collected annually in three waves using 12 survey instruments.* | https://nces.ed.gov/ipeds |
| Universities and colleges | Classification of Instructional Programs (CIP) | Other, classification | CIP provides a taxonomic scheme that supports the accurate tracking and reporting of postsecondary fields of study and program completions activity in IPEDS. An important product of the CIP effort is the crosswalk of CIP program codes to the Standard Occupational Classification (SOC) System, which is referred to as the CIP/SOC Crosswalk. This crosswalk matches postsecondary programs of study that provide graduates with specific skills and knowledge to occupations requiring those skills or knowledge to be successful. | Classifications updated in 1980, 1985, 1990, 2000, 2010, 2020 | https://nces.ed.gov/ipeds/cipcode/Default.aspx?y=56 |
| Level of Education | NCES Products | Survey, Administrative or Other | Characteristics, Topics, etc. | Frequency, Lag Time, etc. | Link |
|---|---|---|---|---|---|
| Universities and colleges | Postsecondary Education Quick Information System (PEQIS) | Survey system | PEQIS was established in 1991 to conduct brief surveys of postsecondary institutions or state higher education agencies on postsecondary education topics of national importance as identified by NCES or another part of the department. Surveys were generally limited to two to three pages of questions, with a response burden of about 30 minutes per respondent. Most PEQIS institutional surveys used a previously recruited nationally representative panel of approximately 1,600 institutions. The system has ended because of staffing issues at NCES. PEQIS was part of a larger project (Quick Response Information System) which included surveys concerning elementary and secondary education FRSS. | The most recent PEQIS survey was conducted in 2013, covering the 2012–13 school year. | https://nces.ed.gov/surveys/peqis/ |
| Universities and colleges | National Study of Postsecondary Faculty (NSOPF) | Survey | NSOPF was a nationally representative sample of full- and part-time faculty and instructional staff at public and private not-for-profit 2- and 4-year institutions in the United States, designed to provide data about faculty and instructional staff to postsecondary education researchers and policy makers. There are no plans to repeat the study. Rather, NCES plans to provide technical assistance to state postsecondary data systems and to encourage the development of robust connections between faculty and student data systems so that key questions concerning faculty, instruction, and student outcomes, such as persistence and completion, can be addressed. | Conducted in 1987–88, 1992–93, 1998–99, and 2003–04. | https://nces.ed.gov/surveys/nsopf/ |
| Level of Education | NCES Products | Survey, Administrative or Other | Characteristics, Topics, etc. | Frequency, Lag Time, etc. | Link |
|---|---|---|---|---|---|
| Adult education/career and technical education | Career and Technical Education (CTE) | Other, a data product built from existing NCES collections | CTE Statistics is the NCES reporting system for national information on CTE and workforce preparation. The program compiled information from a variety of existing NCES data collections that examine students, schools, teachers, and adults in general. Information is provided on CTE participation and CTE staff in public high schools, the education and work outcomes of public high-school graduates, and on CTE participation, outcomes, and providers at the subbaccalaureate level. Information is also available on adults’ occupational certifications and licenses, and on adults’ skills. The CTE program was discontinued in 2019 due to staffing shortage. Contracted web tables are still under production, but no staff are assigned to manage the program. | Website provides links to data and reports in three general areas: secondary/high school, postsecondary/college, and adult. | https://nces.ed.gov/surveys/ctes/ |
| Adult education/career and technical education | National Assessment of Adult Literacy (NAAL) | Survey plus evaluation | The 2003 NAAL is a nationally representative assessment of English literacy among American adults age 16 and older. Sponsored by NCES, NAAL is the nation’s most comprehensive measure of adult literacy since the 1992 National Adult Literacy Survey (NALS). NAAL not only provides information on adults’ literacy performance but also on related background characteristics that are of interest to researchers, practitioners, policy makers, and the general public. | 2003 | https://nces.ed.gov/naal/ |
| Adult education/career and technical education | Program for the International Assessment of Adult Competencies (PIAAC) | Survey with assessment | PIAAC is a cyclical, large-scale study that was developed under the auspices of the Organisation for Economic Co-operation and Development. The goal of PIAAC is to assess and compare the basic skills and the broad range of competencies of adults around the world. The assessment focuses on cognitive and workplace skills needed for successful participation in 21st-century society and the global economy. | Conducted in 2012, 2014, 2017, and 2021 | https://nces.ed.gov/surveys/piaac/ |
| Level of Education | NCES Products | Survey, Administrative or Other | Characteristics, Topics, etc. | Frequency, Lag Time, etc. | Link |
|---|---|---|---|---|---|
| Pre-K, K–12, Lifelong Learning† | International Activities Program (IAP) | Surveys combined with assessments | The IAP supports a variety of activities to make international comparative data available on education and learning. These include the International Early Learning Study, Progress in International Reading Literacy Study, Trends in International Mathematics and Science Study, International Computer and Information Literacy Study, Program for International Student Assessment, and the Teaching and Learning International Survey. Also, listed under adult education, is the Program for the International Assessment of Adult Competencies. Typically these studies combine surveys with assessments. | https://nces.ed.gov/surveys/international/ | |
| All | October Supplement to the Current Population Survey (CPS) | Survey | Selected household member reports for all members of household. Basic CPS: Household membership and characteristics; demographic characteristics; and labor force participation. October Supplement: Basic annual school enrollment for preschool, elementary, secondary, and postsecondary students; and educational background information needed to produce dropout estimates on an annual basis. | Data collected annually by the U.S. Census Bureau on behalf of the Bureau of Labor Statistics and National Center for Education Statistics. | https://nces.ed.gov/surveys/cps/ |
| Level of Education | NCES Products | Survey, Administrative or Other | Characteristics, Topics, etc. | Frequency, Lag Time, etc. | Link |
|---|---|---|---|---|---|
| All | National Household Education Survey: with modules (NHES) | Cross-sectional survey | NHES is a national household survey system that has been used to collect data on a variety of topics. Surveys are conducted by the U.S. Census Bureau. Early childhood modules: Early Childhood Program Participation Survey (ECPP) and, in prior years, in the NHES School Readiness Survey (SR). These are household surveys of families with children from birth through age 6, not yet enrolled in K. Homeschooling and Parental Involvement in Education: NHES Parent and Family Involvement in Education (PFI) Survey collects data on homeschooled children in grades equivalent to K–12 as well as collecting data about students who are enrolled in kindergarten through grade 12. Career/Technical Training Module: The Adult Training and Education Survey (ATES) module collected data about adults ages 16 to 65 not enrolled in high school. |
The SR survey was conducted in 1993 and 2007. The ECPP surveys occurred in 1991, 1995, 2001, 2005, 2012, 2016, and 2019. The next survey is planned for 2023. PFI data were collected in 2012, 2016 and 2019. ATES was collected once in 2016. | https://nces.ed.gov/nhes/; see also a list of previous topics at https://nces.ed.gov/nhes/publications.asp |
SOURCE: NCES document provided to the panel, “List of NCES Programs—Statistics Budget.”
NOTES:
‡The table source, a full listing of NCES’s statistics budget programs, is available on request from the project’s Public Access File. Available: https://www8.nationalacademies.org/pa/information.aspx.
*After a prepublication version of the report was provided to NCES, this sentence was corrected to reflect more accurately how IPEDS is collected.
†After a prepublication version of the report was provided to NCES, this label was corrected to reflect more accurately the levels of education covered by the IAP.
NCES also has an active program of cross-sectional surveys, including household surveys such as the education module of the Current Population Survey, the School Crime Supplement to the National Crime Victimization Survey, and the National Household Education Survey (NHES). NHES targets segments of the population with special modules that have included early childhood education, family involvement in education, homeschooling, and adult education. The National Principal and Teacher Survey is another example of a cross-sectional survey. NCES assessment surveys, such as the National Assessment of Educational Progress, include student tests or assessments embedded in probability sample surveys.
ADMINISTRATIVE RECORDS
Administrative data traditionally refer to data collected by governments for other than statistical purposes (e.g., through the process of administering a program). Administrative data may include financial data about a program, summary statistics about participants or program features, and highly specific data about individuals, businesses, or institutions. Administrative data about individuals collected by the government are called a system of records and must be protected under the Privacy Act of 1974. There are occasional challenges in sharing such data because of consent requirements. Similar rules regarding the protection of lists of individuals have been adopted by many state and local governments, businesses, and institutions. The Office of Management and Budget used the following definition of administrative data:
“Administrative data,’ for purposes of this Memorandum, refers to administrative, regulatory, law enforcement, adjudicatory, financial, or other data held by agencies and offices of the government or their contractors or grantees (including states or other units of government) and collected for other than statistical purposes. Administrative data are typically collected to carry out the basic administration of a program, such as processing benefit applications or tracking services received. These data relate to individuals, businesses, and other institutions” (U.S. OMB, 2014a, p. 4).
Couper (2013, p. 146) describes administrative data as “data provided by persons or organizations for regulatory or other government activities. Users may assume that the data are confidential and used only for the intended purpose by the agency collecting the data.” There has been considerable effort within the federal government to identify and facilitate the use of administrative data for statistical purposes. M-14-06 (U.S. OMB. 2014), cited above, furthered that goal.
NCES relies on administrative data collected by state and local agencies. These agencies provide the data NCES needs for its Common Core of
Data and for its Integrated Postsecondary Education Data System. NCES is prohibited by law from compiling a national database of individually identifiable information on individuals,2 and hence does not take possession of the administrative microdata for individuals; instead, it asks state and local education agencies to provide aggregate information. NCES also plays a key role in assisting other offices within the Department of Education in the collection of administrative data for programmatic purposes.
OTHER DATA SOURCES
Because of the limitations of traditional data sources, statistical agencies are augmenting those sources with private-sector data and a variety of new data sources and technologies. For agencies compiling information about purchases, sales, or prices, scanner data (available for purchase from the private sector) and credit card transactions or bank data can be valuable.
“Private retailers and manufacturers have a long history of collecting consumer data, often for market research purposes” (NASEM, 2021a, p. 68). Some companies sell proprietary data from standing panels of households. These nonprobability samples of willing participants simulate probability samples by targeting invitations to particular types of people, and by using eligibility criteria and poststratification to guide participation and weight the responses to be representative of the intended population. “Granularity is among the strengths of commercial data, and some data are available on a weekly basis. At the same time, these data are collected for marketing or other purposes, are not nationally representative, are not well documented, and coverage may vary across geographic areas” (NASEM, 2021a, p. 68).
“Data originating from commercial and other sources provide information not available elsewhere” (NASEM, 2021a, p. 68). However, one of the challenges with using these data is determining their quality and coverage, key to understanding how the data can best be used. (See NASEM, 2020, pp. 76–79, for more detail on challenges).
NONPROBABILITY SAMPLE SURVEYS
With the growth of web surveys has come growth in nonprobability sampling, in which web posts or advertisements ask people to volunteer for a survey, often in return for an incentive. Such surveys can be used to simulate probability samples by targeting invitations to particular types of
___________________
2 See 20 U.S. Code § 9572(a) National Database. Available: https://www.law.cornell.edu/uscode/text/20/9572 [March 2022].
people, and by using eligibility criteria and poststratification to guide who is allowed to participate and to weight the responses to be representative of the intended population. As noted above, commercial firms are selling data from nonprobability panels; however, a number of academic institutions have also found those data valuable. Examples include the University of Southern California’s Understanding America Survey, which includes some education modules; RAND Corporation’s American Educator Panels; and the American School District Panel.
TRADE ASSOCIATION AND OTHER MEMBERSHIP DATA
Trade associations, professional societies, and other organizations may maintain useful datasets concerning their members or customers. Some may agree to share data for use in research projects, under appropriate conditions. Sometimes these data may be used to develop statistical samples. For example, there is no national list of teachers, but there are professional associations with membership lists, such as the National Education Association and the American Federation of Teachers. However, such frames are not comprehensive and are potentially subject to bias.
WEB SCRAPING
In a presentation to the National Academies of Sciences, Engineering, and Medicine Panel on A Consumer Food Data System for 2030 and Beyond in 2019 Carma Hogue of the U.S. Census Bureau, defined web scraping as “an automated process of collecting data from an online source. Web crawling is an automated process of systematically visiting and reading web pages.” (NASEM, 2020, p. 202).
The U.S. Census Bureau’s Economic Directorate has been researching alternative data sources and big-data methodologies, including web scraping, for 4–5 years. They have concluded that their surveys of federal, state, and local governments are most likely to benefit because “much of the data to be collected on surveys are available online. Currently, analysts manually access data from websites. If Census could develop an automated way to scrape that data, it could reduce respondent and analyst burden.” (NASEM, 2020, p. 202). In addition, “many private companies have terms of use on their websites that prohibit web scraping and web crawling. Government websites do not tend to have such restrictions.” (NASEM, 2020, p. 202)
In terms of education data, districts and schools have increasingly placed important documents containing course offerings, course prerequisites, course registration procedures, school discipline policies, dress codes, event calendars, and more online. Much of this material exists to inform
students and parents. However, web scraping could harvest these documents, code them, and prepare data products.
Haber (2021) provides an example of an education-related analysis of charter schools based on web-scraped data. As a very different web-scraping project, the National Agricultural Statistics Service, has explored web extraction to provide early detection of a disease that impacts pig inventories (NASEM, 2019b, p. 37).
SOCIAL MEDIA
The identification of influenza outbreaks was one of the early uses of analysis of social media communications (e.g., Facebook, Instagram, etc.) for information gathering. Alessa and Faezipour (2018) review these efforts. In their abstract, they state “Many studies have shown that social networking sites can be used to conduct real time analysis for better predictions.”
Torres et al. (2021, p. 1) present a more complex, education-related study. The abstract concludes
The Child Trends News Service sought to broaden access to science-based information to support families during the pandemic through television news, testing whether digital media can be used to understand parents’ concerns, misconceptions, and needs in real time. This article presents that digital media data can supplement traditional ways of conducting audience research and help tailor relevant content for families to garner an average of 90 million views per report.
COGNITIVE INTERVIEWING/TESTING
Cognitive interviewing is often used as a tool in survey development to verify how respondents understand survey questions and whether they can and will answer accurately. For example, “think-aloud” is a common approach, in which the respondent is asked to read the survey question aloud and then verbalize his/her thoughts in preparing an answer. The interviewer both observes the respondent’s reactions (e.g., whether the respondent shows hesitation or confusion in responding to the question, and whether the respondent refers to records to obtain a response) and may probe with additional questions, such as why the respondent hesitated or how the respondent interpreted a particular word in the question. Cognitive testing may be used in an iterative manner, using later rounds to test the changes adopted based on earlier rounds.
Crafts et al. (2016) used cognitive interviewing for developing and testing the National Science Foundation’s (NSF’s) Microbusiness Innovation Science and Technology Survey (later incorporated within NSF’s Business
R&D Innovation Survey). They found that, rather than reading and using the definitions of research and innovation provided in the questionnaire, respondents commonly used their own definitions, which were quite different. NSF revised the questionnaire to eliminate those terms, instead breaking definitions into multiple components, each requiring a yes/no response. Additional testing showed that the reformulated questions obtained more accurate responses.
FOCUS GROUPS
Focus groups are a qualitative research tool used to develop or test a survey questionnaire, or in other research contexts such as to interpret the results from a data collection or simply to examine a topic in depth without any attempt to relate the results to a survey. Unlike an interview, the interaction among focus group participants is an important part of the process (Flores and Alonso, 1995). Focus groups are kept small to encourage participation from all participants. They may be used early in a study as a type of exploratory research, to determine which concepts are important, or later, to help in the interpretation of results.
In an evaluation of a U.S. Department of Labor grant to a consortium of community colleges providing training in advanced manufacturing, one early result was that student retention rates in the program were low (Westat, 2016). Researchers conducted focus groups with students to determine how students felt about retention and program completion, and they found that students sometimes felt their goals were met prior to completing the program, while retention and program completion were of lesser importance to the students. For example, some students found that taking a few courses was sufficient for obtaining a job, and some decided that, once they received outside certification, they no longer needed a college certification.
