National Academies Press: OpenBook

Understanding the Quality of the 2020 Census: Interim Report (2022)

Chapter: 2 Frameworks for Understanding the Decennial Census and Its Quality

« Previous: 1 Introduction
Suggested Citation:"2 Frameworks for Understanding the Decennial Census and Its Quality." National Academies of Sciences, Engineering, and Medicine. 2022. Understanding the Quality of the 2020 Census: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/26529.
×

2

Frameworks for Understanding the Decennial Census and Its Quality

Tasked as we are with assessing the quality of the 2020 U.S. Census, we are obliged to clarify our thinking about what constitutes “quality” in the census context. This chapter articulates these points at a basic, conceptual level, as context for the analyses that the panel expects to complete during its work. We begin in Section 2.1 by describing the fundamental concept of error and the types of error in the census—and the even more foundational observation that error is inevitable in the census. Against this backdrop, the quality of the decennial census can be described as a composite of the accuracy—the relative absence of or mitigation of error, substantial or systemic—in both the data produced by census operations and the manner in which those procedures were carried out. This raises a second major issue that we describe in Section 2.2, which is that there is no unique or simple scorecard: quality assessment depends critically on the filters through which individual census operations or the census as a whole are viewed, the purposes of the decennial census and its data and their fitness for use in meeting those purposes. We close in Section 2.3 with a brief review of the major methodologies by which census error and quality are assessed.

2.1 TYPES OF ERROR IN A DECENNIAL CENSUS

In the pure statistical sense, error is simply the difference between an estimate and the associated true (but unknown) value. Error may be positive or

Suggested Citation:"2 Frameworks for Understanding the Decennial Census and Its Quality." National Academies of Sciences, Engineering, and Medicine. 2022. Understanding the Quality of the 2020 Census: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/26529.
×

negative valued—in the census context, this translates to net overcount (estimate greater than true value) or net undercount (estimate smaller than true value) of particular geographic or demographic group. However, despite the popular connotation of the word—habitually equating “error” with “mistake”—it is important to bear in mind, in discussing census evaluation, that the presence of census errors is not necessarily an indicator that a mistake or something harmful has occurred. Indeed, our predecessor Panel to Review the 2000 Census put it neatly and succinctly in declaring that (National Research Council, 2004:21):

Error is an inevitable part of the census, and perfection—the absence of all error—is an unrealistic and unattainable standard for evaluation. The sources of census error are numerous, and many are difficult to control. In this light, census evaluators must be aware of potential sources of error, estimate their potential effects, and develop strategies to more fully measure and (when possible) minimize errors.

That predecessor Panel to Review the 2000 Census also succinctly summarized the two major categories comprising census error: errors in coverage and errors in response (National Research Council, 2011:20–21). The first of these, coverage errors, arise when persons or housing units are not represented in the census results exactly once and in the right place—as is the basic census goal—but are instead missed entirely or recorded multiple times. These may develop from deviations from the census residence rule and concepts, such as college students being placed at both their school address and their parental homes or being missed at the school location.1 The promotion of the 2020 Census was heavily keyed toward the ease of online response, without needing to wait to enter the identifier code on a physical census mailing, and instead relying on the Non-ID Processing operation to resolve the geographic location. This strategy necessarily incurs the risk of some coverage error and focuses attention on studying the capacity of the Census Bureau’s Non-ID Processing and other data processing operations to handle potentially duplicative responses. More fundamentally, coverage errors may occur due to imperfections in the Census Bureau’s Master Address File, its inventory of physical addresses used to conduct and determine the progress of the count. Improving address list development was a major focus area for innovation and improvement in the 2020 Census, but—despite all best efforts—it is exceedingly difficult to maintain an absolutely

___________________

1 The final residence criteria for the 2020 Census were published in the Federal Register on February 8, 2018 (83 FR 5525), following a period of public comment and reaction. Every U.S. census has been grounded in the concept of counting every person at their usual residence, which the 2020 rule defined consistently with the 2010 rule: “the place where they live and sleep most of the time.” The 2020 residence criteria continue that people in certain types of group quarters facilities (such as adult correctional facilities) “are counted at the group facility,” and “people who do not have a usual residence, or who cannot determine a usual residence, are counted where they are on Census Day.” The associated list of residence situations included with the rule describe the Census Bureau’s interpretation of how different living situations (e.g., shipboard crews or people with seasonal or work-week residences) are resolved under the rule. The history and concepts of census residence rules are developed in great detail by National Research Council (2006).

Suggested Citation:"2 Frameworks for Understanding the Decennial Census and Its Quality." National Academies of Sciences, Engineering, and Medicine. 2022. Understanding the Quality of the 2020 Census: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/26529.
×

complete address list that accounts for churn (both creation and elimination) of housing and address stock, and addresses are subject to both omission and duplication. It follows that gaps and duplicates in the address list produce omissions and duplicates of the persons in those housing units.

A second major category of census error is response error—instances in which the information provided on a census questionnaire or data record is inaccurate. Some part of the population will always be loath to participate in the decennial census, and may not choose to complete questions other than the total population count in the household (if they even provide that); still others may deliberately provide misleading information, whether motivated by distrust in the government or other reasons and despite legal requirements to provide complete and truthful responses.2 Other response errors may arise out of basic misunderstandings (or fear) rather than purposeful misrepresentation: immigrant communities may be particularly hesitant to provide full information (particularly given the extensive public discussion and attention generated by the potential inclusion of a citizenship question in the 2020 Census). Response error can also arise from misinterpretations of the census residence rule and concepts, such as respondents being unclear as to how to report newborn children or the recently deceased or how to account for fluid housing arrangements (such as the doubling-up and sharing of housing generated by the Great Recession in the 2010 Census and COVID-19 stay-at-home and caregiving compliance in the 2020 Census). The U.S. census has long used proxy enumeration, such as information supplied by a neighbor or landlord, as a last resort when no response is obtained from a household, and the Census Bureau has established procedures for imputing values for nonresponse. Moreover, the 2020 Census was always planned to make wider use of data from administrative records sources, including being the first U.S. census in which households could be enumerated using administrative records sources (after an in-person attempt) if there was sufficient confidence in the records data. Each of these methods—enumeration by proxy or administrative records, and imputation for nonresponse—are done to try to come as close to truth as possible, but it remains possible that they are discrepant from the true count and characteristics of the households in question. Thus, they necessarily incur some risk of error. Response error can also be subtler in nature, as in misinterpreting the examples in the race and Hispanic origin questions as prompts for place of birth or in the phenomenon of respondents reporting their age with a round number ending in 0 or 5. Finally, some types of response error may also be introduced by the mode of data collection, such as whether an enumerator is conducting a face-to-face interview or taking an interview by phone.

The unique circumstances of the 2020 Census dramatically increase the profile of two additional classes of census error, the first of which is processing

___________________

2 Truthful response to the census is mandated by 13 U.S.C. § 221–225.

Suggested Citation:"2 Frameworks for Understanding the Decennial Census and Its Quality." National Academies of Sciences, Engineering, and Medicine. 2022. Understanding the Quality of the 2020 Census: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/26529.
×

error (or procedural error), and is the type of error that may arise from the mechanics of census operations or systems processing. As we discuss below, the census involves a very complex mesh of information technology and computer systems; consequently, procedural or processing error is an inevitable risk of the modern census experience. These are the types of error that the Census Bureau works vigorously to detect and stem in its post-collection data processing and editing routines3 and, in the past, have included such things as calculation of age being misspecified, the crosswalk of codes between a group quarters location and the individual units within it being scrambled, or misapplied geolocation codes. Between November 2020 and April 2021, the Census Bureau intently sought to resolve such “anomalies” before issuing apportionment counts and, later, redistricting data. Again, such processing error is normal for something as complex as the decennial census, and it is a class of error that has always been the focus of much painstaking detection and correction by the Census Bureau staff. But it is one that came to new prominence in the 2020 cycle.

The second, newly prominent class of census error drawing attention in the 2020 Census is statistical noise, and it differs from the others in that it is willful, being directly introduced by the Census Bureau. For decades, previous censuses have effectively added noise to census data as a safeguard of respondent privacy. Typically, this was done by data swapping—exchanging some person/household records within small areas to try to thwart direct one-to-one identification of census returns based on auxiliary data. Importantly, the extent and exact nature of the swapping remains a closely held secret in the Census Bureau, lest the process be made amenable to reverse engineering. However, as we will discuss later, the Census Bureau opted to implement a new Disclosure Avoidance System (DAS) for the 2020 Census, premised on the cryptographic concept of differential privacy—infusing a controlled amount of random noise into almost every statistic to be published, all in the context of a master privacy-loss budget of the disclosure risk that can be tolerated for various tabulations. The actual noise added to a specific cell remains unknown, but the distributional properties of that noise are both known and publicly shareable. As we discuss in brief in this report, and will discuss in more depth in the final report, the mechanics of the DAS are considerably more complicated than represented here; additional information about the new disclosure avoidance methodology can be found in the proceedings of a National Academies of Sciences, Engineering, and Medicine (2020) workshop on the 2020 Census data products and in the handbook released by the Census Bureau following the issuance of 2020 Census redistricting data (U.S. Census Bureau, 2021e). It is important to emphasize here that the error associated with noise infusion is different from the other

___________________

3 The application of these routines progress from basic compilation of census returns in the Decennial Response File to a more refined Census Unedited File and then to the Census Edited File; these milestones in the post-collection data processing are described in the Glossary in Appendix A.

Suggested Citation:"2 Frameworks for Understanding the Decennial Census and Its Quality." National Academies of Sciences, Engineering, and Medicine. 2022. Understanding the Quality of the 2020 Census: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/26529.
×

error classes because of its deliberate addition and because it is algorithmic with published parameters. However, this does not imply that the consequences of its use are easily appreciated.

2.2 WHAT IS THE DECENNIAL CENSUS? CENSUS QUALITY IN THE CONTEXT OF CENSUS DATA USES AND PURPOSES

Just as it is important to be clear about the reality that error is an inevitable, inherent part of the census and that absolute perfection (absence of error) is an unattainable standard, it is also necessary to be clear on another foundational point: our mandate is to assess the quality of the 2020 Census (and implications for 2030), but that is a question that lacks a single, specific answer. Understanding the quality of a census involves understanding the relative absence or control of error in the census data and in the way individual census operations were conducted, and forming a composite assessment of quality is a multifaceted problem. One’s interpretation of the quality of a decennial census, and how one takes stock of the error inherent in census results, depends vitally on how one views the role and function of the census, how one intends to use the resulting census data, and how one weights these various uses and purposes. Each of the many fundamental roles that are simultaneously fulfilled by the decennial census brings with it a different set of priorities and criteria for assessment; in this section, we describe some of the basic roles of the U.S. census and the implications for quality assessment.

At root, the decennial census is the “actual Enumeration” required by Article I, Section 2 of the U.S. Constitution to produce the “whole number of persons in each State” required by Section 2 of the Fourteenth Amendment. This is the foundational, constitutional mandate for the census, to support apportionment and the related use case of legislative redistricting. From a quality standpoint, this role puts a high weight on both absolute numerical accuracy of census counts (i.e., a full enumeration or itemization of each person) and relative/distributive accuracy of census counts across states (and geographic areas, generally). It is the spirit of “Enumeration”—of being counted—that makes the decennial census so important for some sociodemographic and geographic groups, to demonstrate their existence and their vitality in numbers. (Conversely, it may also drive the desire by some to not be counted, as with people with vulnerable immigration statuses.)

This “actual Enumeration” plank is a vitally important one, and definitely a valid perspective. But, as already noted, the complete absence of error is an unattainable and unreasonable standard for a census, and even though it is sometimes tempting to treat census figures as unimpeachable truth simply by dint of being “the census,” they are not. Rather, it is important to understand the decennial census is an array of statistical estimates, made subject to error,

Suggested Citation:"2 Frameworks for Understanding the Decennial Census and Its Quality." National Academies of Sciences, Engineering, and Medicine. 2022. Understanding the Quality of the 2020 Census: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/26529.
×

of demographic and geographic population groups. In some application areas, users of these statistical estimates understandably demand a high degree of numerical accuracy and care relatively less about distributional accuracy; such application areas include the use of population thresholds for qualifying for governmental funding programs (where being above or below the threshold is of greatest importance) and in refining legislative and other governmental boundaries to meet equal-population and other districting requirements. But other application areas are better able to account for inherent error properties, so long as the distributional accuracy and characteristics of the data are solid.

From another perspective, the decennial census is an information collection mechanism of the grandest scale: it is a sample survey of the whole, comprising both self-report and interview responses, that collects vital information in a consistent manner. Since the first count in 1790, the U.S. census has never been limited to just a simple head-count of households. The array of socioeconomic information gathered by the census grew sharply over the decades, evolving into a detailed long-form questionnaire asked of just a sample of the population and, with the 2010 Census, into the parallel and ongoing American Community Survey (ACS). This means that whereas the “short-form-only” 2010 and 2020 Censuses included only a limited number of questions, those questions are asked of everyone in the nation and in consistent format and content, making the decennial census a very rich resource for chronicling and understanding race and ethnic origin, within-household relationships and family structures, sex and age distributions, and home occupancy and ownership in the United States. From this perspective, the content and structure of the census questionnaires themselves and the possible effects of the mode of data collection (e.g., paper, Internet, telephone, personal interview) become very important in assessing the quality of a census.

The U.S. census has always used addresses and housing unit locations as its organizational base; the quality and completeness of the Census Bureau’s Master Address File (MAF) and associated geographic databases are inextricably bound to the quality and completeness of the resulting data, for it is the location of those addresses that will determine the location of the people comprising the households. Accordingly, the decennial census is as much an inventory of housing and residential addresses/locations as an enumeration of persons. The 2000–2020 Censuses have made the development and maintenance of a continuous MAF a priority (rather than previous decades’ model of rebuilding the base address list anew for each census). But, as already noted in the previous discussion of the nature of census error, the MAF is itself subject to coverage error and measurement difficulties; if an address does not appear on the MAF, or it is handled incorrectly in the MAF, then it is increasingly likely that the household(s) at that address will not be represented properly in census returns. Measures of census error and quality should include attention to churn and stability, and general issues of quality, in the MAF that underlies the census.

Suggested Citation:"2 Frameworks for Understanding the Decennial Census and Its Quality." National Academies of Sciences, Engineering, and Medicine. 2022. Understanding the Quality of the 2020 Census: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/26529.
×

Relatedly, it should also be acknowledged that MAF quality depends in large part on the quality of the spatial location (map spot) associated with addresses, whether captured directly as geographic positioning system coordinates or interpolated from the topology of road networks—making the census, in part, a large-scale exercise in cartography and map-building.

Each decennial census is a massive, nonmilitary logistical mobilization of personnel and infrastructure, both the large temporary workforce recruited to perform and manage enumeration functions and the physical infrastructure built up to support census activities. Figure 2.1 provides a basic graphical overview of the 35 component operations of the 2020 Census, as represented in version 4.0 (and the final pre-census) version of the Census Bureau’s master Operational Plan for the 2020 Census (U.S. Census Bureau, 2018). The 35 component operations depicted in the figure had to work in parallel or in sequence, as appropriate, with each other in order to complete census work, and they vary in the extent to which they changed from their 2010 Census analogue. Bottlenecks or operational problems in any component operation could directly affect the conduct of downstream operations, with errors potentially cascading as well. Accordingly, in considering the quality of the census, it is important to assess the manner and the timeliness with which work was actually done—taking care to avoid automatic assumptions that the completion of an operation means that it was completed well. For instance, challenges in recruiting and retaining temporary enumerators with essential language skills—or simply the perseverance to do the job in circumstances as demanding as those in 2020—could point to possible issues in the quality of resulting data from field operations.

The individual operations represented in Figure 2.1 all rely on information technology (IT) systems and the effective flow of information to sustain and complete operations. That is, the decennial census is a complex, interconnected “system of systems” and associated business processes, and so the quality of data that results from the census will depend in part on how effectively the component systems integrate and work together—thus making it possible for census workers to get the job done. The concept of the census as a system of systems also makes the decennial census a daunting feat of IT systems engineering, demanding attention to how effectively system operational requirements were developed and communicated along the way, how well those requirements were manifested in the final system releases, and how effectively both in-house and outsourced systems development resources were coordinated.

The decennial census is undeniably a major and costly program of the U.S. federal government and, like all government programs, warrants attention to how well it achieves performance and budgetary goals. Cost reduction (or at least containment of per-housing-unit cost) was a major driver for development in the years preceding the 2020 Census. Understandably, then, some part of broader calculus in determining the success and the quality of the 2020 Census

Suggested Citation:"2 Frameworks for Understanding the Decennial Census and Its Quality." National Academies of Sciences, Engineering, and Medicine. 2022. Understanding the Quality of the 2020 Census: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/26529.
×
Suggested Citation:"2 Frameworks for Understanding the Decennial Census and Its Quality." National Academies of Sciences, Engineering, and Medicine. 2022. Understanding the Quality of the 2020 Census: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/26529.
×

is whether the census and its component operations satisfied the mantra of performing on-time and on-budget.

Finally, for this listing, it is important to consider the decennial census as a flagship program of U.S. federal statistics, and national statistics generally. The decennial census and its data products become the essential reference and benchmark for other household survey measures and the calculation of economic indicators, health/disease incidence rates, crime victimization rates, and myriad other important statistics—at all levels of geography. The census also informs the updating of population estimates between the decennial years. Consequently, it is incumbent that the quality and success of the decennial census be judged, in part, by how the Census Bureau adhered to (or was permitted to adhere to) the standards expected of impartial statistical agencies. Our panel’s parent Committee on National Statistics regularly issues its Principles and Practices for a Federal Statistical Agency, the most recent version of which is summarized in Box 2.1. These principles and practices set some high and challenging bars for assessing the quality and success of the census as a major statistical program: the quality of the census must be judged, in part, by whether its results are relevant, credible, trustworthy, and fit for use in essential applications, by its conduct independent of political influence, and by its commitment to improvement and innovation.

Suggested Citation:"2 Frameworks for Understanding the Decennial Census and Its Quality." National Academies of Sciences, Engineering, and Medicine. 2022. Understanding the Quality of the 2020 Census: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/26529.
×

2.3 HOW CENSUS ERROR AND CENSUS DATA QUALITY MAY BE ASSESSED

Over the decades, the basic methodologies for assessing the quality of census data have typically been of two general sorts: comparison of the census data to some alternative but relevant data source or extensive interrogation of the census data themselves and the paradata from the procedures and operations that yielded them. In briefly describing these fundamental evaluation approaches, the now familiar refrain arises—the unique and innovative circumstances of the 2020 Census create major new opportunities for census evaluation, but some extremely daunting challenges as well.

Within the broad class of evaluation based on comparison to other data sources, the two leading methods over the decades have been demographic analysis and coverage measurement using a postenumeration survey. The first of these, demographic analysis, is a long-standing approach premised on constructing a population estimate completely independent of the census returns through accounting—adding births and immigrants, subtracting deaths and emigrants. Births and deaths are generally deemed to be measured well by the nation’s Vital Statistics program and reference to Medicare records, but estimating immigration and emigration is notoriously difficult (even absent the challenge of considering legal immigration or citizenship status in those figures). Accordingly, the most recent decennial censuses have taken to producing a range of demographic analysis estimates, varying primarily by the assumptions made about the intercensal rate of growth of components of the immigrant population. The nuances involved in this accounting measure are considerable; Robinson (2010) reviews some of the historical issues and Jensen et al. (2020) describe the specific methodology used in 2020.

The key strengths of demographic analysis as a census evaluation tool include its intuitive simplicity—it is easily understood, the challenges of measuring its individual components notwithstanding—and the fact that its development is fully independent of decennial census operations. Indeed, for both the 2010 and 2020 Censuses, the first national-level demographic analysis estimates were released before the first census estimates (the national population and the state-level apportionment totals).4 But it has substantial limitations as well, beyond the difficulty of estimating migration. Demographic analysis estimates are generally only available for the national level and for very broad race/ethnicity, sex, and age groups, at the national level, simply because the necessary data inputs are typically available only at the national

___________________

4 The Census Bureau held a news conference on the first national demographic analysis estimates to accompany the 2010 Census on December 6, 2010, a few weeks prior to the release of 2010 Census totals. For the 2020 Census, initial demographic analysis figures were released on December 15, 2020—several months before the apportionment totals would be released in April 2021.

Suggested Citation:"2 Frameworks for Understanding the Decennial Census and Its Quality." National Academies of Sciences, Engineering, and Medicine. 2022. Understanding the Quality of the 2020 Census: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/26529.
×

level of geography. Thus, while it conceptually includes the whole population, demographic analysis is limited in the extent to which it can be examined in disaggregate ways, such as group quarters relative to the general household population or between urban and rural areas. Moreover, the accuracy of those demographic analysis estimates does still depend on race and Hispanic origin being defined and coded in comparative ways in the two censuses and the other component measures, which was not the case in 2020. For 2020, the Census Bureau introduced new approaches to coding of multiple race and Hispanic origin entries, and additional work and analysis remains to be done to facilitate comparisons of 2020 Census data with 2010 Census results and other data sources.

The second major technique for evaluating census quality, and the principal way in which undercounts and overcounts have been diagnosed in recent censuses, is the fielding of a postenumeration survey (PES). In brief, a postenumeration survey proceeds by identifying a sample of blocks and effectively conducting an independent census for those blocks, including address listing that is independent of the census Master Address File and interviewing respondents as soon after the completion of census nonresponse follow-up operations as is practicable (because the survey is also meant to capture a snapshot of the resident population as of the same April 1 reference date as the census itself). The returns of this PES comprise what is dubbed the P-sample, and the census returns from the blocks are called the E-sample. The P- and E-samples are carefully and intensively matched to each other, and statistical dual-system estimation is used to estimate the number of individuals who were missed by both the census and the PES (based on the distribution of individuals missed or counted in error in one source but not the other). Designed with great care and executed in proper scale, the PES is capable of yielding estimates of coverage error for detailed sociodemographic groups and, indeed, was the focus of intense debate in previous censuses about its applicability for statistical adjustment of census totals to correct for coverage issues. Statistical adjustment has been off the table in the 2010 and 2020 Censuses, but the PES remains a crucial diagnostic tool.

That said, there are strong grounds for concern about the utility of the 2020 PES, because the same operational challenges that affected the 2020 Census may have magnified effects on the PES. The independent listing of addresses for the P-sample was able to be completed before the full shutdown of field operations due to the COVID-19 pandemic, but all other PES operations had to be postponed—until after the completion of nonresponse follow-up in the census proper. Moreover, the basic gambit of the PES is that it is done exclusively through personal, face-to-face interview—it is all fieldwork, without the relative luxury of leaning more heavily on mail and Internet self-response. So the 2020 PES represents a major, compounded problem: it is difficult to recruit and retain a field staff, in November 2020, to conduct interviews in the midst of a still-

Suggested Citation:"2 Frameworks for Understanding the Decennial Census and Its Quality." National Academies of Sciences, Engineering, and Medicine. 2022. Understanding the Quality of the 2020 Census: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/26529.
×

escalating pandemic, interacting with a public that may be less amenable to open their doors to interviewers and that was being asked to recall and report on living situations on April 1, 2020, months after the fact (living situations that may also have been seriously disrupted by the pandemic). Moreover, PES respondents might understandably be confused or concerned about being contacted given the strong publicity surrounding the end of 2020 Census field data collection on October 15, 2020. Follow-up and matching operations for the 2020 PES were still ongoing in November 2021, and it remains to be seen how the 2020 PES will be able to account for recall bias and other measurement errors. Another limitation of the PES is that it is restricted to the household population, and so is not designed to and not capable of assessing error in the enumeration of group quarters such as correctional facilities, college student housing, and health care facilities.

Increased use of administrative records data has been an aspirational goal of census planners for several decades, and increased use of these data in support of enumeration operations was one of the four major innovation areas built into the 2020 Census design. Though the concept of a pure administrative records-based U.S. census may remain a distant target, comparison to a composite of multiple administrative records sources remains in the present an interesting possibility for the assessment of census coverage and error. The same records data that were used in 2020 to fill in responses, in cases when no response could be obtained and confidence in the records-based information for a household was high, could be directly compared to the census results as a whole—subject to considerable care, of course. Many population subgroups that are hard-to-count in the census context, such as the extremely impoverished, may be similarly unlikely to show up in tax rolls and other government program records.5 But, indeed, some of the Census Bureau’s planned projects for its 2020 Census evaluations and experiments will attempt to draw census coverage lessons from comparison to administrative records sources.

As we noted at the beginning of this chapter, census quality is a product of the accurate implementation of the individual operations and procedures used to plan and conduct the census as well as of the resulting data themselves. Accordingly, besides comparisons to some other data source, the other major category of census evaluation technique is the intensive analysis of operational and process data and quality indicators that describe how well census operations were executed. These operational paradata are the principal way, if not the only way, for some important census operations (such as group quarters enumeration) to be assessed, and are crucial to understanding the dynamics of

___________________

5 See, for instance, the 2016 report of a Hard-to-Count working group of the Census Bureau’s National Advisory Committee at https://www2.census.gov/cac/nac/reports/2016-07-admin_internet-wg-report.pdf, the brief synopsis of the 2010 Census Match Study (a complete match of 2010 Census returns to an administrative records composite) at https://mccourt.georgetown.edu/news/who-is-missing-from-administrative-data/, and McClure et al. (2017).

Suggested Citation:"2 Frameworks for Understanding the Decennial Census and Its Quality." National Academies of Sciences, Engineering, and Medicine. 2022. Understanding the Quality of the 2020 Census: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/26529.
×

census nonresponse operations. The Census Bureau itself is the only entity that can analyze much of its own data and operational paradata and, to its credit, it has continually placed a high priority on its data quality checks in processing census returns, as in the painstaking reviews that forced the Census Bureau to miss its statutory deadline in 2020 rather than force a known, seriously flawed result. Moreover, the Census Bureau has commendably made a program of research and evaluation a part of each U.S. decennial census since 1950. Previous panels of the National Academies of Sciences, Engineering, and Medicine have properly chided some of the previous census evaluation programs as being overly focused on the progress and completion of census operations rather than the quality of their execution. Still, the Census Bureau’s own attempt to take stock of how well its operations worked in practice are very valuable contributions. The increased automation of census field operations in the 2020 Census raises new and exciting possibilities for the analysis of operational process data—some of which will have and should have been used by Census Bureau managers in real time during the count itself, this operational dashboard monitoring of information being an important capability enabled by new operational control systems. And the Census Bureau has commendably paired its releases of 2020 Census apportionment and redistricting data with more quality indicator metrics than have been provided at similar stages in earlier decennial censuses. We return to these points in closing this report, but note here that we look forward to the Census Bureau itself making use of analytical capabilities arising from the 2020 Census paradata and continuing to be extensive in providing more detailed quality metric data both with this panel and with other researchers.

Suggested Citation:"2 Frameworks for Understanding the Decennial Census and Its Quality." National Academies of Sciences, Engineering, and Medicine. 2022. Understanding the Quality of the 2020 Census: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/26529.
×

This page intentionally left blank.

Suggested Citation:"2 Frameworks for Understanding the Decennial Census and Its Quality." National Academies of Sciences, Engineering, and Medicine. 2022. Understanding the Quality of the 2020 Census: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/26529.
×
Page 23
Suggested Citation:"2 Frameworks for Understanding the Decennial Census and Its Quality." National Academies of Sciences, Engineering, and Medicine. 2022. Understanding the Quality of the 2020 Census: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/26529.
×
Page 24
Suggested Citation:"2 Frameworks for Understanding the Decennial Census and Its Quality." National Academies of Sciences, Engineering, and Medicine. 2022. Understanding the Quality of the 2020 Census: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/26529.
×
Page 25
Suggested Citation:"2 Frameworks for Understanding the Decennial Census and Its Quality." National Academies of Sciences, Engineering, and Medicine. 2022. Understanding the Quality of the 2020 Census: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/26529.
×
Page 26
Suggested Citation:"2 Frameworks for Understanding the Decennial Census and Its Quality." National Academies of Sciences, Engineering, and Medicine. 2022. Understanding the Quality of the 2020 Census: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/26529.
×
Page 27
Suggested Citation:"2 Frameworks for Understanding the Decennial Census and Its Quality." National Academies of Sciences, Engineering, and Medicine. 2022. Understanding the Quality of the 2020 Census: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/26529.
×
Page 28
Suggested Citation:"2 Frameworks for Understanding the Decennial Census and Its Quality." National Academies of Sciences, Engineering, and Medicine. 2022. Understanding the Quality of the 2020 Census: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/26529.
×
Page 29
Suggested Citation:"2 Frameworks for Understanding the Decennial Census and Its Quality." National Academies of Sciences, Engineering, and Medicine. 2022. Understanding the Quality of the 2020 Census: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/26529.
×
Page 30
Suggested Citation:"2 Frameworks for Understanding the Decennial Census and Its Quality." National Academies of Sciences, Engineering, and Medicine. 2022. Understanding the Quality of the 2020 Census: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/26529.
×
Page 31
Suggested Citation:"2 Frameworks for Understanding the Decennial Census and Its Quality." National Academies of Sciences, Engineering, and Medicine. 2022. Understanding the Quality of the 2020 Census: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/26529.
×
Page 32
Suggested Citation:"2 Frameworks for Understanding the Decennial Census and Its Quality." National Academies of Sciences, Engineering, and Medicine. 2022. Understanding the Quality of the 2020 Census: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/26529.
×
Page 33
Suggested Citation:"2 Frameworks for Understanding the Decennial Census and Its Quality." National Academies of Sciences, Engineering, and Medicine. 2022. Understanding the Quality of the 2020 Census: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/26529.
×
Page 34
Suggested Citation:"2 Frameworks for Understanding the Decennial Census and Its Quality." National Academies of Sciences, Engineering, and Medicine. 2022. Understanding the Quality of the 2020 Census: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/26529.
×
Page 35
Suggested Citation:"2 Frameworks for Understanding the Decennial Census and Its Quality." National Academies of Sciences, Engineering, and Medicine. 2022. Understanding the Quality of the 2020 Census: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/26529.
×
Page 36
Next: 3 Other Evaluations of the 2020 Census »
Understanding the Quality of the 2020 Census: Interim Report Get This Book
×
Buy Paperback | $35.00 Buy Ebook | $28.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

The decennial census is foundational to the functioning of American democracy, and maintaining the public's trust in the census and its resulting data is a correspondingly high-stakes affair. The 2020 Census was implemented in light of severe and unprecedented operational challenges, adapting to the COVID-19 pandemic, natural disasters, and other disruptions. This interim report from a panel of the Committee on National Statistics discusses concepts of error and quality in the decennial census as prelude to the panel’s forthcoming fuller assessment of 2020 Census data, process measures, and quality metrics. The panel will release a final report that will include conclusions about the quality of the 2020 Census and make recommendations for further research by the U.S. Census Bureau to plan the 2030 Census.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!