THE U.S. DECENNIAL CENSUS IS INTEGRAL to the regular realignment of power and resources mandated by Article I, Section 2, of the U.S. Constitution. In addition to apportioning the U.S. House of Representatives, the census provides the basic population count information used to redraw congressional and other legislative districts. Census data also serve a wide variety of other purposes, including the allocation of government funds, the construction of frames and weights for many surveys, and the development of numerous statistical measures and indexes.
The 2010 census will be the nation’s 23rd decennial enumeration. Historically, the decennial census has been characterized by both change and continuity. Marshals on horseback have given way to temporary enumerators working for a permanent Census Bureau, methods based solely on personal visits to those relying principally on the mail, and a long-form sample of respondents providing additional socioeconomic data to the continuous American Community Survey. The machinery of census-taking has changed as well, from handwritten ledgers to Hollerith punch cards to the first UNIVAC computer to modern document-scanning technology. Yet the 2010 census—like all of its predecessors—will inherit some key concepts that have been part of the census experience since the first enumeration in 1790. The de jure standard of trying to associate people with their single place of “usual residence” that will be used in the 2010 census is directly descended from the “usual place of abode” standard dictated by the first U.S. Congress for the 1790 census (1 Stat. 101). The act that authorized the first census in 1790 also made response to census inquiries mandatory of all residents, as is still the case in modern statute.
Systematic research and development (R&D) to improve census methodology, pretest changes in procedures, and evaluate census results is an integral part of the modern census. Such R&D was not possible in the early decades of census-taking because there was no permanent census office with appropriate staffing, and statistical methods for testing and evaluation were in their infancy. An exception that illustrates the importance of research and testing occurred for the 1850 census: a temporary Census Board, created in 1849, solicited outside expert advice that resulted in marked improvements in the design of the census forms that were used by U.S. marshals to collect the data. The improvements were based on an 1845 census of Boston directed by Lemuel Shattuck, a founder of the American Statistical Association (Hacker, 2000a).
A permanent Census Bureau was established in 1902, and the 1910 census included perhaps the first major experiment conducted in conjunction with a U.S. census. In order to boost awareness of the 1910 census and save interviewing time, census enumerators in large cities (100,000 population or greater) distributed “advance schedules” a few days prior to the beginning of the actual count. Although respondents were encouraged to fill out the schedule in advance of the enumerator visit, enumerators were not permitted to simply accept a filled-out form as a response to the census; simply “prepar[ing] the way for the enumerator by announcing his approaching visit and informing the people precisely of the questions to be answered” was the primary aim of the experiment, while potential time savings and gains in accuracy were described as the “secondary object” (U.S. Department of Commerce and Labor, 1911:115–116). Three decades later, the Census Bureau’s plan for the 1940 census included the first formal, structured set of evaluations of census content and quality in a decennial census; the procedural history of the 1940 census commissioned by the Bureau describes a series of analyses of the data quality for individual question items as well as limited work to estimate the level of underenumeration in the census (Jenkins, 1983:96–104). The 1950 census included the first systematic set of pretests and experiments to plan the enumeration, along with the first set of experiments conducted as part of the census and a large number of post-census evaluations (Goldfield and Pemberton, 2000a).
The Census Bureau uses several terms to delineate its census R&D activities, including pretest or test, experiment, and evaluation. Although usage is not always consistent, in this terminology, “tests” involve collection of data from respondents between censuses. Some tests shake down operations and uncover potential problems in the field; other tests are more properly termed experiments, in that they involve designed comparisons of alternative questionnaires, field procedures, and other aspects of the census. The Bureau, however, typically reserves the term “experiment” for instances when data collection to compare alternative methods is conducted as part and parcel of
the census itself. Finally, “evaluations” involve after-the-fact comparisons of census results with other surveys or data sources. Many evaluations assess operational or data quality by using information that is tracked and collected as census operations take place. Other evaluations, notably the postenumeration survey that is commonly used to assess coverage errors in the census, involve original data collection. Our panel suggests that fixation on these labels is not as helpful as thinking in terms of census “research and development” generally.
Since the 1950 census, some program of both experimentation and evaluation has been a fixture of the decennial census process. These formal programs have developed into major enterprises: not counting studies directly related to the postenumeration survey and coverage measurement, the 2000 census included a slate of 91 formal evaluations (pared from an original list of 149 possibilities), 5 formal experiments, and 3 studies based on qualitative, ethnographic observation of the census. No exception from its recent predecessors, the upcoming 2010 census has been designed to include a formal slate of experimentation and evaluation; early in the planning process, the Census Bureau dubbed this effort the 2010 Census Program of Evaluations and Experiments, or CPEX for short.
THE PANEL, ITS CHARGE, AND PREVIOUS REPORTS
The program evaluations of the 2000 census were sharply criticized by the National Research Council (2004a:Finding 1.11) Panel to Review the 2000 Census as “slow to appear, [often] of limited value to users for understanding differences in data quality[, and] of limited use for 2010 planning.” Recognizing that the experiments and evaluations of the 2000 census did not provide an adequate basis for 2010 census planning, the U.S. Census Bureau requested that the Committee on National Statistics (CNSTAT) of the National Research Council convene this Panel on the Design of the 2010 Census Program of Evaluations and Experiments in early 2007. The panel was charged to:
review the U.S. Census Bureau’s program of research, evaluation, and experimentation for the 2010 census. The panel will consider priorities for evaluation and for evaluation and experimentation in the 2010 census. It will also consider the design and documentation of the Master Address File and operational databases to facilitate research and evaluation, the design of experiments to embed in the 2010 census, the design of evaluations of the dress rehearsal and 2010 census processes, and what can be learned from the pre-2010 testing that was conducted in 2003–2006 to enhance the testing to be conducted in 2012–2016 to support census planning for 2020. Topic areas for research, evaluation, and testing that would come within the panel’s scope include
questionnaire design, address updating, nonresponse follow-up, coverage follow-up, unduplication of housing units and residents, editing and imputation procedures, and other census operations.
This is the panel’s third and final report. At the Census Bureau’s request, the panel issued a first, interim report in late 2007 (National Research Council, 2008b) with the express intent of reviewing an initial list of general topics prepared by the Census Bureau’s CPEX staff to identify key priorities. In that report, we focused in particular on topics for experimentation that were likely to improve census quality, reduce census costs, and provide a basis for further research during the 2010–2020 decade. Accordingly, we identified two areas that were either entirely absent from the Bureau’s initial list or mentioned only in limited fashion: the use of Internet data collection and the use of administrative records for a variety of census purposes. We further recommended that an alternative questionnaire experiment—a staple of previous census experimentation programs—center primarily on the presentation of residence concepts and accurate elicitation of household counts.
Subsequently, Census Bureau staff briefed the panel about its chosen roster of experiments and evaluation studies for 2010; we describe the Bureau’s roster in more detail in Appendix B. In brief, in its selections, the Census Bureau chose not to pursue any form of Internet data collection experiment and also declined to study uses of administrative records other than their limited use in the 2010 Coverage Follow-Up operation. As expected, an alternative questionnaire experiment was chosen for the 2010 experiment roster, but its focus is almost exclusively on subtle and hard-to-distinguish differences in the wording and presentation of race and Hispanic-origin questions. The panel had remaining concerns about the Bureau’s selected research priorities. Moreover, great uncertainty surrounded the progress of the Census Bureau’s “replanning” effort following the discovery of major problems with its Field Data Collection Automation contract; it was unclear to the panel (and other external observers, such as the U.S. Government Accountability Office, 2008, 2009b,d) whether the operational control systems that manage the flow of information during the census process would be adequately tested prior to deployment. The panel was also concerned that, in the haste to simply get operational systems in a functional state, the Bureau might not give proper attention to archiving data or creating “spigots” of operational data in order to enable evaluation of census processes. Because of these concerns, the panel issued a second report in February 2009 (National Research Council, 2009a) in the form of a letter to then-acting Census Bureau Director Thomas Mesenbourg. The letter report provides the panel’s detailed critique of the four experiments known to the panel at the time.
To give the work of this panel a unified presentation—and because neither of our previous two reports had been printed in final, typeset volumes—
OVERVIEW OF THIS REPORT
This is our panel’s third and final report; it is also somewhat different in tenor from the two reports that preceded it because—quite deliberately, and despite the formal name of our panel—the report has relatively little to do with suggesting changes or structures for the Census Bureau’s 2010 CPEX program. We provide an updated description of what we know of the current plans for CPEX in Appendix B, because the Bureau has made some significant revisions to the program since the time of our letter report. But, although we critique the CPEX plans, we refrain from suggesting changes in this report for the simple reason that major change is effectively impossible due to the fast-approaching date of the 2010 census itself. At this writing, in late 2009, the operational exigencies of the decennial census and related programs are such that plans for experiments had to be finalized by spring 2009, if only to facilitate printing of questionnaires and selection of experimental samples. Hence, we do not offer specific comments or suggested refinements of the experiments to be conducted in 2010 because it is no longer feasible for such changes to be made.
Likewise, we reiterate our concern from previous reports about the acquisition of operational data from the 2010 census for evaluation purposes—it is impossible to speak meaningfully of the types of analyses that could be conducted in 2010 census evaluation studies without better knowledge of the operational data that will be available. But we must also accept that there is little to be gained by suggesting specific practices in this report because the Census Bureau’s focus is now on getting its systems in order for the main count, a process that has grown more risky due to the need in 2008 to replan census operations and revert to paper-based nonresponse follow-up. We hope and trust that evaluation-ready data will be saved during the census process, but there are no further words at this point that can ensure that this is done.1
Instead, we turn in this final report to the broader themes raised by our charge and our work in the previous reports. Rather than the specifics of the 2010 CPEX, it is our intent in this report to speak of census research and development generally—to describe the role that research should play during the 2010–2020 decade to put the 2020 census on solid footing. In doing so, we also build on important contributions and themes raised by some of
In late 2009, the Census Bureau described late changes to the CPEX program that include a “Master Trace Project” to retain and study 2010 census operational data. Because of this timing and because details of the CPEX changes are not yet available, we cannot comment further on these steps in the report. However, we describe them in introducing Section 3–A.2.
our predecessor National Research Council panels, including the Panel to Evaluate Alternative Census Methods (National Research Council, 1994), the Panel on Research on Future Census Methods (“2010 Census” panel; National Research Council, 2004b), and the Panel on Residence Rules in the Decennial Census (National Research Council, 2006).
In Chapter 2, we discuss what we think to be the most critical drivers for census planning and research: the balance between census cost and census quality. In Chapter 3, we offer a critique of the Census Bureau’s current strategies for research, experimentation, and testing—in contrast with the Bureau’s legacy of past research—before suggesting general directions for research in the 2010–2020 period. We also conclude that more formal, structural changes in the Census Bureau’s organization for research are critical for success and elaborate those structural recommendations in Chapter 4. Two appendices to the report describe past and present Census Bureau research programs in detail; Appendix A describes the research, testing, and evaluation programs that preceded and were conducted in line with the 1950–2010 censuses, and Appendix B summarizes what we know about the 2010 CPEX program.