THE DECENNIAL CENSUS is the federal government’s largest and most complex peacetime operation. Mandated in the U.S. Constitution to provide the basis for reapportioning seats in the U.S. House of Representatives, the census has many purposes today: redrawing congressional and state legislative district boundaries; allocating federal and state program funds; planning and evaluating federal, state, and local government programs; providing the basis for updated population estimates; and informing researchers, the private sector, the media, and the public about the characteristics of population groups and geographic localities and how they have changed over time.
The National Research Council’s Committee on National Statistics established the Panel to Review the 2000 Census in 1998 at the request of the U.S. Census Bureau. The panel was given a broad charge:
to review the statistical methods of the 2000 census, particularly the use of the Accuracy and Coverage Evaluation Program and dual-systems estimation, and other census procedures that may affect the completeness and quality of the data.
The panel’s findings in this final report cover the planning process for 2000, major innovations in census operations, the completeness of population coverage in 2000, and the quality of both the basic demographic data collected from all census respondents and the detailed socioeconomic data collected from the census long-form
sample (about one-sixth of total households). We recommend improvements in the planning process and design for the 2010 census that flow from our evaluations of 2000; however, our recommendations are not intended to be comprehensive in terms of 2010 planning. The Panel on Research on Future Census Methods is assessing the Census Bureau’s plans for 2010.
Our overall conclusion is that the 2000 census experienced both major successes and significant problems. Despite problems with the census address list, the completeness of population coverage and the quality of the basic demographic data were at least as good as and possibly better than in previous censuses. However, many of the long-form items on such topics as income and employment, as well as the data for residents of college dormitories, prisons, nursing homes, group homes, and other group quarters, were no better and in some cases worse than in previous censuses. That we found problematic areas should not detract from the success of the 2000 census in providing relevant, timely data for many uses. It should also make clear the importance of a thorough evaluation of the quality of census data and operations for users to understand the census information and for planners to improve future censuses.
THE ROAD TO 2000
The 2000 census planning began in a climate of concern about the perceived failures of the 1990 census—one that saw a substantial decline in public cooperation and, despite higher per household costs than in the 1980 census, resulted in worse coverage of minorities, renters, and children relative to other population groups. The Census Bureau’s initial design to remedy these problems for 2000 relied on much greater use of statistical techniques in the census enumeration, but this plan encountered opposition from members of Congress and others. As a result, the Bureau had to contend with externally imposed last-minute changes in design, delayed budget decisions, consequent changes in plans, and insufficient time for operational trials. All of these problems increased not only the costs of the 2000 census but also the risk that it could have been seriously flawed in one or more respects.
In light of this experience, the panel recommends that the Census Bureau, the administration, and Congress agree on the basic
design for the 2010 census no later than 2006 in order to permit an appropriate, well-planned dress rehearsal in 2008. In particular, this agreement should specify the role of the new American Community Survey (ACS). Further delay will undercut the ability of the ACS to provide, by 2010, small-area data of the type traditionally collected on the census long-form sample and will jeopardize 2010 planning, which currently assumes a short-form-only census (Recommendations 3.1 and 3.2).
ASSESSMENT OF 2000 CENSUS OPERATIONS
We find that the 2000 census was generally well executed despite many obstacles and even though agreement on the final 2000 design was not reached until spring 1999. The dedication of the Census Bureau staff made possible the success of several operational innovations. Because of the last-minute changes in plans, the ample funding appropriated by Congress in spring 1999 was essential for the successful execution of the census.
Two major successes of the 2000 census were that the decline in mail response rates observed in the 1980 and 1990 censuses was halted and that most operations were well conducted and completed on or ahead of schedule. The use of a redesigned questionnaire and mailing strategy and of an expanded advertising and outreach program contributed to the improved mail response rate, which, in turn, helped reduce the cost and time of follow-up activities. Contracting for selected data operations, using improved technology for capturing the data on questionnaires, and aggressively recruiting enumerators and implementing nonresponse follow-up—all innovations in 2000—contributed to the timely execution of the census.
Two major operational problems of the 2000 census were the error-plagued development of the Master Address File (MAF) and the poorly managed enumeration of residents of group quarters. The use of multiple sources to build a Master Address File—a major innovation in 2000—was appropriate in concept but not well executed. The Bureau’s early plans for constructing the MAF assumed that the U.S. Postal Service’s Delivery Sequence File would provide most of the needed updates to the 1990 list in mailout/mailback areas, but the Postal Service list was not complete or accurate enough for this purpose. The consequent changes in schedules and operations
and the variability in local efforts to update the MAF contributed to census enumeration errors, including a large number of duplicates. ThenumberofduplicateswouldhavebeenlargeryetiftheBureau had not mounted an ad hoc operation in summer 2000 to weed out duplicate MAF addresses.
The census is the only comprehensive source of information on the group quarters population, which constitutes large proportions of some communities. The group quarters enumeration in 2000 was not well conducted, experiencing such deficiencies as poorly controlled tracking of enumerations and errors in assigning group quarters to the correct geographic areas. There was no program for evaluating the completeness of coverage of group quarters residents.
Looking to improve census operations in 2010 from the experience in 2000, we recommend that the Census Bureau give high priority to research in three areas:
Targeted second questionnaire: Because a second questionnaire sent to nonresponding households is known to significantly improve response rates—but feasibility issues prevented its use in 2000—the Census Bureau must proceed quickly to work with vendors to develop cost-effective, timely ways to mail a second questionnaire to nonresponding households in the 2010 census, in a manner that minimizes duplicate enumerations (Recommendation 4.1).
Improved MAF development procedures: Because a complete, accurate Master Address File is critical not only for the 2010 census but also for the 2008 dress rehearsal, the new American Community Survey, and other Census Bureau surveys, the Bureau must develop more effective procedures for updating and correcting the MAF than were used in 2000. In particular, the Bureau must develop procedures for obtaining accurate information to identify housing units within multiunit structures and redesign its Local Update of Census Addresses partnership program to benefit state and local governments that participate (Recommendation 4.3).
Better enumeration of group quarters: Because of the growing importance of group quarters populations, the Census Bureau must thoroughly evaluate and completely redesign the processes related to them for the 2010 census, adapting the design as needed
for different types of group quarters. This effort should include consideration of clearer definitions for group quarters, redesign of questionnaires and data content as appropriate, and improvement of the address listing, enumeration, and coverage evaluation processes for group quarters (Recommendation 4.4).
ASSESSMENT OF POPULATION COVERAGE IN 2000
Much valuable information is available from the 2000 Accuracy and Coverage Evaluation (A.C.E.) Survey about the completeness of the census enumeration for the household population. Changes in estimation methods between the most recent A.C.E. Revision II, completed in March 2003, and the 1990 Post-Enumeration Survey (PES) make it difficult to compare coverage estimates between the two censuses. Nonetheless, there is sufficient evidence to conclude that net undercount rates for population groups were reduced in 2000 from 1990 and, even more important, that differences in net undercount rates between historically less-well-counted groups (minorities, children, renters) and other groups were reduced as well. From this result and from analysis of available information for states and large counties and places, it is reasonable to infer that differences in net undercount rates among geographic areas were also probably smaller in 2000 compared with 1990.
The reduction in differential net undercount was an achievement of the 2000 census. Contributing to this outcome, however, were large numbers of duplicate census enumerations and large numbers of wholly imputed census records (three times as many such records as in 1990, including imputations for members of enumerated households who lacked all basic characteristics as well as imputations of people into housing units that were presumed to be occupied). Both the duplicate enumerations and the imputed records generally accounted for larger proportions of historically less-well-counted groups than of other groups.
Despite the reduced differences in net undercount rates and an estimate that the 2000 census overcounted the total population by a small percentage (a first in census history), such groups as black men and renters continued be undercounted in 2000, whereas other groups were overcounted. Reducing such differences will be an important goal for the 2010 census. Also in 2000, gross errors of er-
roneous enumerations and omissions were almost as high as in 1990. The distribution of coverage errors across geographic areas is hard to discern from the available information; there is no information about coverage of group quarters residents.
The Census Bureau’s Three Decisions Against Adjustment
In March 2001, October 2001, and March 2003, the Census Bureau announced that it would not adjust the 2000 census results for incomplete coverage of some population groups (and overcounting of other groups). In the panel’s view, all three decisions are justified. The Bureau’s March 2001 and October 2001 decisions are justified given: (1) its conclusion in March that evaluation studies were not sufficient to determine the accuracy of the A.C.E. population estimates and (2) its conclusion in October, after further study, that the original A.C.E. population estimates were too high. The Bureau’s March 2003 decision not to use the A.C.E. Revision II coverage measurement results to adjust the 2000 census base counts for the Bureau’s postcensal population estimates program is justified as well. Although the Revision II estimation work was thorough, innovative, and of high quality, the results are too uncertain to be used with sufficient confidence about their reliability for adjustment of census counts for subnational geographic areas and population groups.
Sources of uncertainty stem from the small samples of the A.C.E. data that were available to correct components of the original A.C.E. estimates of erroneous enumerations and non-A.C.E. residents and the consequent inability to make these corrections for other than very large population groups; the inability to determine which of each pair of duplicates detected in the A.C.E. evaluations was correct and which should not have been counted in the census or included as an A.C.E. resident; the possible errors in subnational estimates from the choice of one of several alternative “correlation bias” adjustments to compensate for higher proportions of missing men relative to women; the inability to make correlation bias adjustments for population groups other than blacks and nonblacks; and the possible errors for some small areas from the use of different population groups for estimating erroneous census enumerations and census omissions.
That estimated net undercount rates and differences in net undercount rates for population groups (and, by inference, subnational areas) were smaller in 2000 than in 1990 contributed to the panel’s agreement with the Census Bureau’s decision not to use the A.C.E. Revision II results to adjust the 2000 census counts. The smaller the measured coverage errors in the census, the smaller would be the effects of an adjustment on important uses of the census. Because the benefits of an adjustment are less when coverage errors are small, a high level of confidence is needed that an adjustment would not significantly increase the census errors for some areas and population groups. In our judgment, the A.C.E. Revision II estimates, given the constraints of the available data for correcting the original A.C.E. estimates, are too uncertain for use in this context.
Performance of the Accuracy and Coverage Evaluation
The original A.C.E. was well conducted, and, in many respects, an improvement over the 1990 PES, achieving such successes as timely data collection, low missing data rates, and improved quality of matching. However, inaccurate reporting of household residence in the A.C.E.—which also occurred in the census—led the original A.C.E. to substantially underestimate duplicate census enumerations in 2000. Many of these duplications occurred because respondents were confused about or misinterpreted census residence rules, so that, for example, college students were often counted at home as well as at their residence hall, and people with both summer and winter homes were often counted twice.
The Census Bureau commendably dedicated resources to the A.C.E. Revision II effort, which used the original A.C.E. and evaluation data to completely reestimate net undercount (or overcount) rates for several hundred population groups (poststrata). This work exhibited high levels of creativity and effort devoted to a complex problem. From innovative use of matching technology and other evaluations, it provided substantial additional information about the numbers and sources of erroneous census enumerations and, similarly, information with which to correct the residency status of the independent A.C.E. sample. It provided little additional information, however, about the numbers and sources of census omissions.
Demographic analysis uses records of births, deaths, and net immigration to produce updated population estimates for age and sex groups for blacks and for all others. The original 2000 demographic analysis estimates were subsequently revised in October 2001. There are sufficient uncertainties in the revised net immigration estimates (particularly the illegal component) and a revised assumption of completeness of birth registration after 1984, compounded by the difficulties of classifying people by race, so that the revised demographic analysis estimates cannot serve as the definitive standard of evaluation for the 2000 census or the A.C.E. However, the estimates were useful in helping to identify coverage problems in the census and the A.C.E., and the method is important as the basis for postcensal population estimates.
Coverage Evaluation in 2010
A large postenumeration survey is an essential part of a coverage measurement program for the U.S. census. Demographic analysis, which is limited to producing net national-level coverage estimates for blacks and all others by age and sex, cannot substitute for a postenumeration survey. Underscoring the need for a large survey in 2010 is the prospect that the Census Bureau will make extensive use of methods developed from 2000 A.C.E. research on matching census records by name and birthdate to reduce duplicate enumerations in the 2010 census. If that is done and if in 2010 the number of omissions were to be as large as or larger than in 2000, the consequence could be a substantial net undercount of the population and an increase in differential undercoverage among population groups. In those circumstances, it would be essential to have an improved coverage evaluation survey built on the experience of the 2000 A.C.E. to understand the coverage achieved in 2010 and to inform the public about it. Data from an improved survey would be available to develop reasonably reliable estimates of coverage errors, which would be needed not only to explain the accuracy of the census but also to permit adjustment of some or all of the data should that be considered necessary.
We therefore recommend that the Census Bureau and the administration should request, and Congress should provide, funding for
the development and implementation of an improved A.C.E. Program for the 2010 census. The A.C.E. survey in 2010 should be large enough to provide estimates of coverage errors that provide the level of precision targeted for the original (March 2001) A.C.E. estimates for population groups and geographic areas. An important area for improvement is the estimation of components of gross census errors (including types of erroneous enumerations and omissions), as well as net error. Understanding gross errors is valuable for identifying areas for improvement in census processes. Other important areas for improvement include the inclusion of group quarters in the A.C.E. (they were excluded in 2000); improved questionnaire content and interviewing procedures about place of residence; a simpler procedure for treating people who moved between Census Day and the A.C.E. interview; the development of poststrata for estimation of net coverage errors by using census results and statistical modeling as appropriate; and the investigation of possible correlation bias adjustments for additional population groups (Recommendation 6.1). We also recommend that the Census Bureau strengthen its program to improve demographic analysis estimates, particularly estimates of net immigration and measures of uncertainty for the demographic results (Recommendation 6.2).
It is clear from the experience with the 2000 A.C.E. that evaluation of coverage errors in the census takes time. It also takes time to evaluate census processes and the quality of the census data for individual items. In the panel’s view, adequate evaluation of the census block-level data for congressional redistricting is not possible by the current deadline. Congress should consider moving this deadline to allow more time for evaluation of the completeness of population coverage and quality of the basic demographic items before they are released (Recommendation 6.3). Users should be aware that census counts at the block level—whether based on the enumeration or adjusted for coverage errors—are subject to high levels of variability; these data should be aggregated to larger geographic areas for use.
ASSESSMENT OF COMPLETE-COUNT AND ADDITIONAL LONG-FORM-SAMPLE DATA
The 2000 census data for items asked of everyone (age, sex, race, ethnicity, household relationship, housing tenure) were quite complete at the national level—missing data rates ranged from 2 to 5
percent (including records with one or more missing items and people who were wholly imputed). Rates of inconsistent reporting for these items were also low. However, some population groups and geographic areas exhibited high rates of missing data and inconsistent reporting for the basic items.
Missing data rates for the 2000 census long-form items were moderately to very high (10 to 20 percent or more) for over one-half of the items, which cover such topics as citizenship, veterans status, migration, disabilities, education, work experience, transportation to work, occupation, income, housing characteristics, and others. Missing data rates also varied widely among population groups and geographic areas. By comparison with 1990, missing data rates were higher in 2000 (often substantially so) for most of the long-form items asked in both years. Missing data rates for long-form items for group quarters residents in 2000 were much higher than the rates for household members and considerably higher than the rates for group quarters residents in 1990. Only scattered evidence is available as yet about possible systematic reporting errors for long-form items (e.g., systematic underreports or overreports of income or education).
One reason for higher missing data rates in 2000 than in 1990 is that the 2000 census placed greater reliance on imputation routines to supply values for missing and inconsistent responses, in contrast to the greater reliance on telephone and field follow-up to obtain answers for missing items in 1990. The greater reliance on imputation in 2000 contributed to the timely execution of the census. Whether it produced distributions for characteristics that were less accurate than the distributions that would have been produced from additional follow-up is not known. There is scattered evidence that the imputation procedures sometimes produced untoward results—for example, extremely high unemployment rates for some types of group quarters residents.
We recommend that the Census Bureau should develop procedures to quantify and report the variability of the 2000 long-form-sample estimates due to imputation, in addition to the variability due to sampling and whole-household nonresponse. The Bureau should also study the effects of imputation on the distributions of characteristics and the relationships among them and conduct research on improved imputation methods for use in the American
Community Survey (or the 2010 census if it includes a long-form sample; Recommendation 7.1).
We recommend that the Census Bureau should make users of the 2000 census long-form-sample data products (Summary Files 3 and 4 and the Public Use Microdata Samples) aware of the high missing data rates and measures of inconsistent reporting for many long-form items. The Bureau should also inform users of the need for caution in analyzing and interpreting these data (Recommendation 7.2). Users should note that the Census Bureau has reissued employment status tabulations to provide data for the household population and to exclude group quarters residents because of high missing data rates and imputation errors for them.
We recommend that the Census Bureau’s planning for the 2010 census should include research on the trade-offs in costs and accuracy between imputation and additional field work for missing data (Recommendation 4.2). We also recommend that the Census Bureau publish distributions of characteristics and item imputation rates for the 2010 census and the American Community Survey (when it includes group quarters residents) that distinguish household residents from the group quarters population (at least the institutionalized component). Such separation would make it easier for data users to compare census and ACS estimates with household surveys and would facilitate comparative assessments of data quality for these two populations (Recommendation 7.3).
ASSESSMENT OF 2000 CENSUS RESEARCH
The Census Bureau planned two major evaluation efforts for 2000: one set of evaluations addressed population coverage, using results from the census, the A.C.E., and demographic analysis; the other set assessed the full range of census processes and examined measures of completeness and consistency of reporting for content items. Under tight time constraints, the Census Bureau staff charged with evaluating coverage conducted comprehensive and insightful research of high quality. Their results for the A.C.E. (and demographic analysis) were timely, well-documented, and provided useful information to stakeholders and the public. However, the evaluation program for census operations and data quality appears to have experienced difficulties in obtaining usable data for analysis,
perhaps because the evaluation staff perspective was not taken into account in designing census databases, such as the MAF. The results from this evaluation program were slow to appear and are often of limited use for understanding the quality of the 2000 census or for planning the 2010 census.
In addition to pursuing improvements for coverage evaluation in 2010 as outlined above, the Census Bureau must materially strengthen the evaluation component for census operations and data quality in 2010 (and in the current testing program). The Bureau should mine data sources created during the 2000 census process, such as the A.C.E. data, extracts from the MAF, a Master Trace Sample of data collection and processing information for a sample of addresses, and a forthcoming match of census records with the March 2000 Current Population Survey, to address important outstanding questions about 2000 data quality and suggest research to improve the 2010 census and the American Community Survey. The Bureau should identify important areas for evaluations in 2010 to meet the needs of users and census planners and set evaluation schedules accordingly; relatedly, it should design and document 2010 data collection and processing systems so that information can be readily extracted to support timely, useful evaluation studies. The Bureau should use graphical and other exploratory data analysis tools to identify patterns (e.g., mail return rates, imputation rates) for geographic areas and population groups that may suggest reasons for variations in data quality and ways to improve quality (such tools could also be useful in real-time management of census operations). The Bureau should accord priority to development of technical staff resources for research and evaluation of both coverage and content in 2010 (Recommendations 9.1 and 9.2).
Finally, the Census Bureau should share preliminary analyses with outside researchers for critical assessment and feedback. To help the Bureau evaluate population coverage and data quality in the 2010 census, the Bureau should seek ways—using the experience with the Panel to Review the 2000 Census as a model—to furnish preliminary data, including microdata, to qualified researchers under arrangements that protect confidentiality (Recommendation 9.3). The panel benefited greatly from the ability to have ready access to detailed evaluation data. In our view, greater sharing with re-