National Academies Press: OpenBook
« Previous: Summary
Suggested Citation:"1 Vision for the American Community Survey." National Research Council. 2015. Realizing the Potential of the American Community Survey: Challenges, Tradeoffs, and Opportunities. Washington, DC: The National Academies Press. doi: 10.17226/21653.
×

1

Vision for the American Community Survey

The American Community Survey (ACS) is the result of efforts—dating back to the first census in 1790—to collect data for public policy purposes through the decennial census process. Presidents and Congresses have regularly reaffirmed the value of the data collected through this platform by their use of the data and by requesting new questions to be added. The need for more frequent data collections, such as mid-decade censuses, has also been voiced repeatedly over the years.

The Census Bureau has met the challenges associated with growing needs for the data through innovative redesigns. For the past few decades, a long-form survey was used as part of the decennial census to collect detailed population and housing data from a sample of the U.S. population. This long-form survey became the main source of socioeconomic information for areas smaller than the whole nation.

The primary motivation for the change from the long-form decennial data collection to the ACS was to produce more frequent and more timely estimates, especially for small areas that may change significantly over the course of a decade. The data provided for small areas, such as counties, municipalities, and neighborhoods, are what made the long-form sample unique, and those data are what make the ACS unique today. Although there are other federal surveys that produce data on similar socioeconomic topics in greater content detail, their sample sizes are not large enough to be able to provide the same granularity as the ACS. The ACS is the primary source of data for anyone wishing to understand the characteristics and needs of small communities and administrative entities from a local

Suggested Citation:"1 Vision for the American Community Survey." National Research Council. 2015. Realizing the Potential of the American Community Survey: Challenges, Tradeoffs, and Opportunities. Washington, DC: The National Academies Press. doi: 10.17226/21653.
×

perspective. The survey also provides data about small population groups that are not available from other sources.

THE ACS IN CONCEPT AND IMPLEMENTATION

The concept of the ACS was proposed in the 1990s, with the goal of moving from the decennial collection of the data to continuous measurement throughout each decade. The idea was based on Leslie Kish’s earlier work on rolling sample design (Kish, 1981), and it entailed replacing the large decadal long-form sample with smaller monthly samples to collect data each month in all geographic areas covered by the survey. The Census Bureau hoped that the data from this design, collected over several years and pooled, would provide estimates of the population and housing characteristics that had previously been produced by the decennial long-form sample, with the same level of precision as the data from the long-form sample, even for small geographic areas.

In the mid-1990s, the Census Bureau began testing possible approaches to implementing the new continuous measurement survey. This testing was followed by a demonstration stage between 2000 and 2004, when data were produced at the national and state levels, as well as several large geographic areas. Full implementation began in 2005, with a sample of housing units, followed by a combined sample of housing units and group quarters in 2006.

While the data collected through the census long-form sample provided a snapshot every 10 years, with the ACS the Census Bureau publishes annual cumulated 1-year estimates for geographic entities with populations of at least 65,000, 3-year estimates for geographic entities with populations of at least 20,000, and 5-year estimates for all statistical, legal, and administrative entities, including areas as small as census block groups.

As part of the implementation of the ACS, the Census Bureau developed new data processing, estimation, data review, and data dissemination tools and methods to enable the release of the data before the end of the calendar year following the year in which they were collected. This means that the releases are not only more frequent, but data are also released more quickly after the data collection has been completed than was the case with the census long-form sample. This is particularly important for informing policy making in smaller, rapidly changing geographic areas, where once-a-decade measurements of social, economic, or housing characteristics could quickly become obsolete.

Another significant change that occurred with the transition from the decennial data collection to continuous measurement was a shift from relying on a large temporary workforce to highly trained, professional staff

Suggested Citation:"1 Vision for the American Community Survey." National Research Council. 2015. Realizing the Potential of the American Community Survey: Challenges, Tradeoffs, and Opportunities. Washington, DC: The National Academies Press. doi: 10.17226/21653.
×

to carry out the various activities associated with the survey operations, including in-person follow-up visits to nonrespondents. This change led to lower rates of refusal to complete the questionnaire, as well as to an improvement in the quality of the information that is collected, primarily due to lower item nonresponse rates. In addition to data quality improvements, the change also led to increased operational efficiency. A further benefit to the Census Bureau was the reduction in the burden imposed on the other parts of the decennial census operations by the simultaneous collection of the long-form data.

Despite a highly successful implementation, a major challenge for the ACS is producing precise estimates for small geographic areas. The initial hope for the survey was to produce estimates with a similar level of precision to the data from the census long-form sample, by cumulating 5 years of data, with the estimates representing averages over the 5-year period.

For the 2000 census long-form sample, the overall sampling rate was 1 in 6, which translated to approximately 18 million housing units. Initially, sample sizes of 500,000 housing units monthly (or 6 million housing units annually) were proposed for the ACS (Alexander, 1993b), but it quickly became evident that a survey of that size would be prohibitively expensive. After additional research, the Census Bureau determined that a sample half that size, that is, 250,000 housing units monthly or 3 million housing units annually, would generate acceptable levels of precision (U.S. Census Bureau, 2009).

For the first few years of the ACS, the annual sample size was a little under 3 million housing units. Starting in June 2011, this was increased to approximately 3.5 million housing units annually, due largely to concerns voiced by data users about the precision of the estimates, particularly in smaller geographic areas. Table 1-1 shows the ACS sample sizes since its inception, including both housing units and group quarters. During the first 5 years of data collection (2005-2009), the average annual percentage of addresses in the sample was 2.2.

The 2011 increase brought the sample size over 5 years somewhat closer to the size of the long-form sample, but nonresponse follow-up is different. For the long-form sample, the Census Bureau followed up with all nonrespondents as part of the decennial operations; in the ACS nonrespondents are sampled for follow-up, which increases the design effect of the survey estimates and widens confidence intervals around the estimates. Unmailable addresses, which do not receive follow-up in the ACS, further reduce the number of completed interviews. Ultimately, the number of responding households is about two-thirds the size of the initial ACS sample. Furthermore, the annual sample sizes are not adjusted for a natural growth of the population in the sampling frame over time, so the effective rate of sampling is declining.

Suggested Citation:"1 Vision for the American Community Survey." National Research Council. 2015. Realizing the Potential of the American Community Survey: Challenges, Tradeoffs, and Opportunities. Washington, DC: The National Academies Press. doi: 10.17226/21653.
×

TABLE 1-1 Initial ACS Sample Sizes and Completed Responses, 2006-2012

Year Housing Units Group Quarters Residents
Initial Addresses Selected Final Number of Responses Initial Sample Selected Final Number of Actual Responses Final Synthetic Casesa
2012 3,539,552 2,375,715 208,551 154,182 137,086
2011 3,272,520 2,128,104 204,553 148,486 150,052
2010 2,899,676 1,917,799 197,045 144,948 N/A
2009 2,897,256 1,917,748 198,808 146,716 N/A
2008 2,894,711 1,931,955 186,862 145,974 N/A
2007 2,886,453 1,937,659 187,012 142,468 N/A
2006 2,885,384 1,968,362 189,641 145,311 N/A

aFinal actual responses are the responses obtained from sampled group quarters residents. Synthetic interviews for group quarters residents were created by imputing the characteristics of interviewed group quarters persons into group quarters facilities that were not in the sample for that year or other time period.

SOURCE: Data from U.S. Census Bureau; available https://www.census.gov/acs/www/methodology/sample_size_and_data_quality/ [August 2014].

Overall, while sample sizes have proven adequate for larger geographic areas, the unique strength of the ACS design, as envisioned, was the survey’s potential to produce population and housing estimates for the smallest of geographic areas (such as tracts, block groups, and school districts), as well as small demographic groups. This vision has not been achieved because for small areas the margins of error associated with the estimates can be very large.

PANEL CHARGE

Now that the ACS is nearly 10 years old, this is a good time to assess its evolution to date and consider how it can be enhanced going forward. The Census Bureau asked the Committee on National Statistics of the National Research Council to convene a panel and provide input on changes that the ACS office should consider over the course of the next few years in order to further improve the ACS data (see Box 1-1). In addition to the need for addressing the primary challenge associated with the precision of the estimates, the ACS is also at a stage in its natural evolution at which it is timely to evaluate other aspects of the survey and associated processes to identify opportunities for refinement based on the experience of the first

Suggested Citation:"1 Vision for the American Community Survey." National Research Council. 2015. Realizing the Potential of the American Community Survey: Challenges, Tradeoffs, and Opportunities. Washington, DC: The National Academies Press. doi: 10.17226/21653.
×

BOX 1-1
Statement of Task

An ad hoc panel will conduct a study to address priority technical issues for the American Community Survey (ACS) as the survey enters its next decade. The panel will consider how the Census Bureau could improve performance in several areas, which may ultimately lead to improved data products. The panel should conduct its work on the assumption that increases in ACS resources may not be possible.

  • The panel will focus on methods and approaches to improve the accuracy of demographic, social, economic, and housing information produced from the ACS for the smallest geographic areas and population groups and will advise the Census Bureau on how to communicate the changes to data users in ways that facilitate effective use of the data.
  • The panel will also consider data collection processes that can more efficiently meet national and local needs for new content in the broader context of the fundamental mission of the ACS.

few years. The panel was asked to focus on two priority areas: identifying methods that could improve the quality of the data available for small areas and suggesting changes that would increase the survey’s efficiency in responding to new data needs.

PANEL APPROACH AND INITIAL ASSESSMENT

Since the survey was launched in 2005, many stakeholders have come to depend on the ACS data. Federal agencies use the ACS data to inform policy makers, assess programs, and distribute funds. A study found that in 2008, ACS data or data derived from the ACS were used by 184 federal domestic assistance programs to guide the geographic distribution of $416 billion in funds, representing 29 percent of all federal assistance (Reamer, 2010). State and local agencies use the ACS to evaluate the need for new services, such as roads, schools, and hospitals. Businesses use the ACS for information about potential markets, such as where people who might be interested in their services are concentrated. Other frequent users of the data include nongovernmental organizations, organizations serving American Indians and Alaska Natives, emergency planners, academic researchers, and journalists. The broad range of stakeholders demonstrates the survey’s success, but it also represents a more complex challenge in terms of priori-

Suggested Citation:"1 Vision for the American Community Survey." National Research Council. 2015. Realizing the Potential of the American Community Survey: Challenges, Tradeoffs, and Opportunities. Washington, DC: The National Academies Press. doi: 10.17226/21653.
×

tizing decisions for how to meet the growing needs and expectations for ACS data, especially when operating with limited resources.

The panel’s deliberations led to the formulation of three guiding principles that provide a framework for the discussions and recommendations in this report:

  • While comparisons to the decennial long-form survey provide a useful context for understanding the evolution of the ACS, the unique strength of the survey is not in replicating the long-form survey, but in meeting data needs that can best be addressed by a large national survey with the design characteristics of the ACS. The needs for small area data evolve and so do the methods and tools available for accomplishing the survey’s objectives.
  • How well the survey is meeting data needs for small administrative entities and population groups can only be truly assessed from the perspective of the broad range of stakeholders who use the data. In the design of a successful survey, there is no substitute for a thorough understanding of data users’ needs.
  • Tradeoffs will have to be made. Some of these tradeoffs are inevitable choices between competing survey design objectives (such as speed, accuracy, and level of detail). Other tradeoffs are imposed by resource limitations, particularly a sample size that is insufficient for producing adequately precise data for all small geographic areas and groups. Given the role of the ACS as a national resource, design decisions, such as the optimal allocation of sample among geographic areas in order to improve the precision of the estimates, are not simply statistical questions: they also involve policy decisions. The panel was not charged with assessing these matters from a policy perspective, but nonetheless emphasizes that a solid understanding of stakeholder needs is necessary for informed policy decisions.

There have been several recent Census Bureau initiatives targeted at understanding data users’ needs. A recently conducted content review was focused primarily on understanding federal agencies’ uses of the questions on the ACS, although other data users were also encouraged to provide input. The ACS Data User Group (ACS DUG) was formed in 2013 with the goal of providing a platform for information exchange related to the data, and the Census Bureau also sponsored several data users’ workshops over the past few years, with both federal and nonfederal data users. However, the ACS DUG is not tasked with providing formal data user input to the Census Bureau and is not set up in a way that could serve that function. Closer collaboration with data users is needed to ensure that, as refinements

Suggested Citation:"1 Vision for the American Community Survey." National Research Council. 2015. Realizing the Potential of the American Community Survey: Challenges, Tradeoffs, and Opportunities. Washington, DC: The National Academies Press. doi: 10.17226/21653.
×

to the survey are considered going forward, the decisions are informed by stakeholder input. A standing group that is available to provide feedback on the survey and the data from stakeholders’ perspectives can provide highly valuable long-term benefits, for very little cost to the Census Bureau. The panel thus offers an overarching, priority recommendation.

RECOMMENDATION 1: As a priority, the Census Bureau should establish a formal, institutionalized, standing group to provide ongoing data user input on decisions related to the American Community Survey, and this standing group should include representation of data user organizations.

CHALLENGES AND OPPORTUNITIES FOR ENHANCING THE ACS

This report discusses features of the ACS that can be improved or for which research should be conducted at this stage in the survey’s evolution. Although the report is not intended to be a comprehensive review of all aspects of the ACS, it addresses a wide range of elements of the survey, including survey design, implementation, and data production and dissemination, because each of these affects the utility of the data. Although the most obvious solution to the frequently voiced concern of the low precision of the small area data would be to allocate more funding to increase the sample size, the focus of this report is on methodological changes that the panel believes could enhance the value of the survey for data users with current funding levels (as stated in our charge).

One important advantage in considering changes to the ACS derives from the survey’s design of continuous data collection that relies on smaller monthly samples, in contrast to the one-time-only census long-form sample. This feature enables the Census Bureau to develop and test improvements to the survey, including new statistical methods, new question wording, or new data collection methods that can then be implemented in relatively short time. The ACS can be more nimble than the long-form survey was, and innovative solutions are possible. One example is the recently implemented imputation methodology to improve the estimates of the group quarters population in small areas. However, year-to-year comparisons are an important use of the survey, and 5 years of consistent data are needed to create data products for small areas, so changes to the survey have to be introduced with great care.

As detailed below, the next four chapters of the report (Chapters 2 through 5) focus on the first part of the panel’s charge, improving the accuracy and usefulness of the information produced from the ACS for small areas. The final chapter (Chapter 6) addresses the second part of the charge, processes to better meet national and local needs for new survey content.

Suggested Citation:"1 Vision for the American Community Survey." National Research Council. 2015. Realizing the Potential of the American Community Survey: Challenges, Tradeoffs, and Opportunities. Washington, DC: The National Academies Press. doi: 10.17226/21653.
×

Sample Design

The primary challenge for the ACS is that the sample sizes and design are inadequate for producing estimates with precision that approaches the precision of the data that users were accustomed to from the census long-form sample. While ACS data are an invaluable resource for meeting analytic needs associated with larger geographic areas, data users are concerned about the precision of the estimates for the smallest geographic areas and groups. Simply put, the low precision of some of the estimates, even with 5 years of aggregation, renders them unusable from the perspective of many data users at the local level. Further aggregation of tracts or categories improves the usability of the data in many cases, but it is not a suitable solution in all areas.

Chapter 2 describes the ACS sample size and design characteristics, which are the fundamental determinants of precision. The chapter also discusses the implications of the recent sample reallocation implemented by the Census Bureau to increase the sampling rates in the smallest geographic areas and the panel’s recommendations for further refinements to the sampling approach, given current funding levels.

Data Collection Methods

The ACS covers many topics, some of them in considerable detail, although it is not any more burdensome for respondents than the census long-form survey was (the time required to answer the questions depends largely on the number of people in the household). Response to the ACS is required by law (Title 13, U.S. Code, Sections 141, 193, and 221), as was completing the long-form survey. The main difference in respondent burden between the two data collections is that the ACS is continuously in the field, so respondent complaints can arise at any time. Consequently, questions about justifications for the survey may arise more frequently in the media or in Congress than they did with the decennial long-form survey.

The Census Bureau takes the issue of respondent burden seriously and is researching ways of addressing it. The recently added Internet option is expected to make responding to the survey more convenient for respondents, as well as to increase its efficiency. The ACS has also been serving as the test survey for several adaptive design ideas that similarly have the potential of reducing respondent burden and increasing efficiency. Chapter 3 discusses the panel’s findings and recommendations for further research and enhancements to the data collection methods, which are indirectly also expected to improve data quality.

Suggested Citation:"1 Vision for the American Community Survey." National Research Council. 2015. Realizing the Potential of the American Community Survey: Challenges, Tradeoffs, and Opportunities. Washington, DC: The National Academies Press. doi: 10.17226/21653.
×

Data Processing and Analytic Issues

Processing of collected data can affect data quality. Estimates from the census long-form sample benefited from weights available from the simultaneous census enumeration. By contrast, the weights used to align characteristics of the ACS sample with the overall population are based on control totals from the postcensal population estimates produced by the Census Bureau’s Population Estimates Program. Consequently, controls are not available for all levels of geography, and any imprecision in the population estimates also affects the ACS estimates.

The magnitude of sampling error in published estimates is represented by a margin of error published with the estimates. This is informative for most estimates, but for estimates of small proportions it can be confusing and not particularly useful. Chapter 4 suggests improvements and further directions for research on these aspects, and others, of ACS data processing. The chapter also discusses the use of administrative records that can be considered for editing, for imputation, or to evaluate bias, as well as small area and domain estimation methodologies.

Data Dissemination Limitations

ACS data products are closely modeled on the census long-form data products. Although the availability of annual estimates is one of the major benefits of the ACS, production of a wide range of data products based on three different datasets (of 1, 3, and 5 years) is very resource intensive. The overlaps in production, review, and dissemination stress the system, increasing the risk of compromising data quality.

The volume of datasets and data products can also be overwhelming for users. Because of the large volume of available data, which can be accessed in many different ways, users sometimes find it difficult to figure out what is the most efficient way of obtaining the data, and some methods are challenging to use without training.

At the same time, for small areas and population groups, the volume of data can be misleading: because of the sample size limitations, many of the estimates cannot be made available to data users. To protect the identities of respondents, the Census Bureau’s Disclosure Review Board prohibits release of some estimates. In addition, the 1- and 3-year data releases are filtered on the basis of data quality (precision) considerations. The filtering rates are particularly high for small geographic areas.

Due to the limitations of the small area data, an option for data users is to aggregate estimates across geographic areas or population subgroups, at least when the geographic and analytic needs lend themselves to such aggregation. Performing the aggregations, however, requires cumbersome

Suggested Citation:"1 Vision for the American Community Survey." National Research Council. 2015. Realizing the Potential of the American Community Survey: Challenges, Tradeoffs, and Opportunities. Washington, DC: The National Academies Press. doi: 10.17226/21653.
×

margin-of-error calculations, which few data users have the expertise or resources to perform. The Census Bureau does not provide a tool to assist data users with the aggregation, only instructions for how to do it. The current range of dissemination methods is also limited in terms of the opportunities it provides for custom analyses. Chapter 5 discusses these issues and presents the panel’s recommendations for improving data dissemination.

Survey Content

The current ACS content is largely based on the census long-form survey, but it is likely that some of the questions are no longer as useful or necessary as they once were. The Census Bureau also has to balance requests for new content with the need to limit the time required to complete the survey, in other words, with respondent burden. A benefit of continuous data collection is that new content can be added to the ACS more quickly than was possible in the decennial census cycle, but the process still takes many years. Chapter 6 of the report discusses how the Census Bureau can make the ACS responsive to users’ needs with the highest value content.

Suggested Citation:"1 Vision for the American Community Survey." National Research Council. 2015. Realizing the Potential of the American Community Survey: Challenges, Tradeoffs, and Opportunities. Washington, DC: The National Academies Press. doi: 10.17226/21653.
×
Page 13
Suggested Citation:"1 Vision for the American Community Survey." National Research Council. 2015. Realizing the Potential of the American Community Survey: Challenges, Tradeoffs, and Opportunities. Washington, DC: The National Academies Press. doi: 10.17226/21653.
×
Page 14
Suggested Citation:"1 Vision for the American Community Survey." National Research Council. 2015. Realizing the Potential of the American Community Survey: Challenges, Tradeoffs, and Opportunities. Washington, DC: The National Academies Press. doi: 10.17226/21653.
×
Page 15
Suggested Citation:"1 Vision for the American Community Survey." National Research Council. 2015. Realizing the Potential of the American Community Survey: Challenges, Tradeoffs, and Opportunities. Washington, DC: The National Academies Press. doi: 10.17226/21653.
×
Page 16
Suggested Citation:"1 Vision for the American Community Survey." National Research Council. 2015. Realizing the Potential of the American Community Survey: Challenges, Tradeoffs, and Opportunities. Washington, DC: The National Academies Press. doi: 10.17226/21653.
×
Page 17
Suggested Citation:"1 Vision for the American Community Survey." National Research Council. 2015. Realizing the Potential of the American Community Survey: Challenges, Tradeoffs, and Opportunities. Washington, DC: The National Academies Press. doi: 10.17226/21653.
×
Page 18
Suggested Citation:"1 Vision for the American Community Survey." National Research Council. 2015. Realizing the Potential of the American Community Survey: Challenges, Tradeoffs, and Opportunities. Washington, DC: The National Academies Press. doi: 10.17226/21653.
×
Page 19
Suggested Citation:"1 Vision for the American Community Survey." National Research Council. 2015. Realizing the Potential of the American Community Survey: Challenges, Tradeoffs, and Opportunities. Washington, DC: The National Academies Press. doi: 10.17226/21653.
×
Page 20
Suggested Citation:"1 Vision for the American Community Survey." National Research Council. 2015. Realizing the Potential of the American Community Survey: Challenges, Tradeoffs, and Opportunities. Washington, DC: The National Academies Press. doi: 10.17226/21653.
×
Page 21
Suggested Citation:"1 Vision for the American Community Survey." National Research Council. 2015. Realizing the Potential of the American Community Survey: Challenges, Tradeoffs, and Opportunities. Washington, DC: The National Academies Press. doi: 10.17226/21653.
×
Page 22
Next: 2 Sample Design and Precision of Estimates »
Realizing the Potential of the American Community Survey: Challenges, Tradeoffs, and Opportunities Get This Book
×
Buy Paperback | $56.00 Buy Ebook | $44.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

The American Community Survey (ACS) was conceptualized as a replacement to the census long form, which collected detailed population and housing data from a sample of the U.S. population, once a decade, as part of the decennial census operations. The long form was traditionally the main source of socio-economic information for areas below the national level. The data provided for small areas, such as counties, municipalities, and neighborhoods is what made the long form unique, and what makes the ACS unique today. Since the successful transition from the decennial long form in 2005, the ACS has become an invaluable resource for many stakeholders, particularly for meeting national and state level data needs. However, due to inadequate sample sizes, a major challenge for the survey is producing reliable estimates for smaller geographic areas, which is a concern because of the unique role fulfilled by the long form, and now the ACS, of providing data with a geographic granularity that no other federal survey could provide. In addition to the primary challenge associated with the reliability of the estimates, this is also a good time to assess other aspects of the survey in order to identify opportunities for refinement based on the experience of the first few years.

Realizing the Potential of the American Community Survey provides input on ways of improving the ACS, focusing on two priority areas: identifying methods that could improve the quality of the data available for small areas, and suggesting changes that would increase the survey's efficiency in responding to new data needs. This report considers changes that the ACS office should consider over the course of the next few years in order to further improve the ACS data. The recommendations of Realizing the Potential of the American Community Survey will help the Census Bureau improve performance in several areas, which may ultimately lead to improved data products as the survey enters its next decade.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!