Realizing the Potential of the American Community Survey: Challenges, Tradeoffs, and Opportunities (2015)

6

Survey Content

The American Community Survey (ACS) replaced the census long-form sample, and the current content of the survey is largely based on the questions that had been part of the long-form survey. Now that the ACS has been in place for several years, it is the right time to consider how the content of the ACS meets small area needs.

Since the ACS was launched, a very large and diverse array of stakeholders have come to depend on the data. The survey is used by federal agencies to inform policy makers, assess programs, and distribute funds; by state and local agencies to evaluate the need for new services, such as roads, schools, and hospitals; and by businesses to understand potential markets, such as a concentration of people who might be interested in their services. Nongovernmental organizations, emergency planners, organizations serving American Indians and Alaska Natives, academic researchers, and journalists are also frequent users of ACS data. Many examples of important uses of the data are well documented (see, e.g., National Research Council, 2013) and are a reflection of the survey’s vital importance to the nation as a whole.

The broad range of uses and nuances in data user needs also raises difficult questions about how to set priorities among the demands for survey content, without further increasing respondent burden. The Paperwork Reduction Act of 1995 (44 U.S.C. 3501) requires federal agencies to obtain approval from the Office of Management and Budget (OMB) prior to collecting information from the public. As part of the decennial census, the ACS is mandatory, and OMB’s role also includes ensuring that only ques-

tions that are “necessary” are asked on the ACS. This assessment, however, is a challenging task, given that the history of the census long-form survey created a tangle of obligations in terms of the items on the ACS. Some of the questions have been included since the first census, while others were added over the years, primarily driven by legislative requirements.

For the 2000 census, the general guideline was that a question could be included on the long-form survey if it met one of three criteria (U.S. Census Bureau, 2014d):

• a current federal law that explicitly called for the use of the decennial census data for a particular federal program;
• a federal law or implementing regulation that required the use of specific data, and the decennial census was the historical or only source of the data; or
• an operational need by the Census Bureau.

After the ACS was implemented, to further refine the criteria, the following additional considerations were added by OMB for evaluating whether the survey is the right vehicle for a question (U.S. Census Bureau, 2014d):

• whether the data were needed with the frequency of the ACS,
• whether the data were needed at a small area level, and
• whether any other source of data is available to meet the need.

Managing the ACS content is a constant balance between federally mandated functions and broader uses of the data, and it appears to be hindered by a lack of systematic, in-depth understanding of the range of uses. Although, as described below, outreach activities are currently under way to document the uses of the data, particularly at the federal level, content management is one of the areas for which close collaboration with the broader community of data users and a continuous feedback loop about how the questions are meeting data needs are essential in order to ensure that the potential benefits of the survey are maximized (see Recommendation 1, in Chapter 1).

CURRENT CONTENT AND USES

Box 6-1 shows the topics currently covered in the survey. The mail version of the 2014 questionnaire is included in Appendix E, and Appendix F shows when each of the questions was added and which agency requested it. Due to the large number and variety of requests, the Census Bureau does

not have systematic documentation of questions that were requested over the years but not added.

The questionnaire collects basic demographic information and additional population and economic characteristics about residents in the housing unit selected into the sample (“living or staying at [the] address for more than 2 months”). The full set of questions are asked of up to five residents, and if there are more than five people living at the address, the name, sex, and age if collected for up to an additional seven people. The questionnaire also includes questions about the physical characteristics of the property and the financial characteristics of the household. Contact information for the person completing the questionnaire is collected for administrative purposes.

Some examples of the intent and main uses of the questions are briefly described below. As discussed in subsequent sections of this chapter, the Census Bureau recently completed a review of the items on the survey to more extensively document the purpose of each of the questions.

Basic Demographic Questions

Basic demographic questions, such as age, sex, and race, are used to inform programs that are targeted at specific groups. For example, age data are used to allocate funds for services to children, working-age adults, and the elderly. Under the Voting Rights Act, an estimate of the population of voting age is required for legislative redistricting. Various programs, such as those targeted at women, require information on sex. Race data are used to promote equal employment opportunity and to assess racial disparities in access to programs and services. The ACS questionnaire also asks about the relationships among those living in the household. This information is used to understand living arrangements and family characteristics, such as the number of people living alone or the number of children living with one parent, which informs the planning of federal programs, such as nutrition and education programs.

Population Questions

In addition to the basic demographic questions, the ACS includes questions aimed at understanding broader social and population characteristics and trends. For example, the citizenship questions are used to understand how immigrant groups are assimilated. The place of birth question not only helps understand immigration patterns, but also provides information on migration among states. The question about a person’s residence 1 year ago sheds further light on population mobility and its effects on various geographic areas.

Ancestry data are required to enforce provisions under the Civil Rights Acts of 1964 and 1968, which prohibit discrimination, and are used to develop services that accommodate cultural differences. The education questions provide further useful socioeconomic data and are also used for program planning and funding allocation. Information on undergraduate field of degree in the case of college graduates provides data for assessing the qualifications of the U.S. workforce and local economic development efforts. Questions about disability are used to inform programs aimed at assisting people with disabilities. The veteran status questions are used to measure the presence and needs of veterans and to evaluate the effects of programs aimed at veterans. The number of children born in the past 12 months is used to project the future size of the population. Questions about marital history and grandparents who have primary responsibility for the care of their grandchildren are used to understand family structure and develop and evaluate programs and services.

Economic Questions

The ACS’s economic questions include labor force status and occupation questions, which are used to understand employment and unemployment patterns, the availability of workers, and to formulate policy. Employment data also factor into defining metropolitan areas, calculating state per capita incomes, and assessing the impact of immigration on the economy and job markets. Income questions are used to measure economic well-being and determine poverty levels, as well as the need for economic assistance. Health insurance questions are used to understand state and local health insurance needs. Questions about commuting patterns are used by planning organizations to identify areas that need transit services, design programs that ease traffic problems, and plan for emergency services.

Housing Questions

The ACS’s housing questions cover both the physical and financial characteristics of the home. This information is used to understand the characteristics of the housing inventory, determine fair market rents, and manage a variety of programs, such as the Low-Income Home Energy Assistance Program. For example, the question about the volume of agricultural sales helps determine whether the property is a farm and factors into estimates of the size of the farm population, which in turn is used to allocate funds to states. The question about the existence of a business on the premises helps understand property values in context. In addition to helping determine fair market rents, questions about plumbing and kitchen facilities are used to identify areas eligible for public assistance programs

and rehabilitation loans. The question about heating fuel provides additional information about the adequacy of housing, as well as data about energy supply and consumption. Questions about Internet access are used to understand the availability of broadband technology and for programs that are aimed at expanding public access. The question about the number of vehicles is used in transportation planning. Financial questions related to housing, such as mortgage, insurance, taxes, and the cost of utilities, are used to measure the cost of home ownership and determine the need for housing assistance programs in different areas.

ACS CONTENT REVIEW

Based on a review of data from the U.S. General Services Administration’s Catalog of Federal Domestic Assistance and the Census Bureau’s Consolidated Federal Funds Report, Reamer (2010) estimated that 184 federal assistance programs relied directly or indirectly on ACS data to help guide the distribution of $416 billion, or 29 percent of all federal assistance, while federal grant funding informed by the ACS accounted for $389.2 billion, or 69 percent of all federal grant funding. The ACS facilitates the distribution of federal funds primarily by providing data that contribute to or complement several other datasets, such as the Bureau of Economic Analysis’ per capita income series and the Census Bureau’s own population estimates.

While overviews such as the Reamer report present a compelling case that information obtained on the ACS is essential for the functioning of a variety of programs, there is no up-to-date inventory of the ways each of the questions are used, even by other federal agencies. To address this gap, the Census Bureau recently conducted a comprehensive review of all of the questions on the ACS to better understand how federal agencies use the data. One aspect of this effort is the updating of the list of laws, statutes, and regulations that underlie the collection of each of the data items and determining at what level of geography are the data needed. Although federal uses represent only a small fraction of the many uses of the ACS data, this is an important first step towards documenting the Census Bureau’s obligations and understanding how much flexibility there is for content changes.

Because responding to the ACS is mandatory for those who are selected into the sample, minimizing respondent burden is particularly important. In addition to the need to limit the time required to complete the survey, the Census Bureau is also concerned about the possibility that the burden placed on respondents by questions that are perceived to be particularly burdensome or sensitive could adversely impact overall data quality, either by increasing nonresponse rates or by decreasing the quality of the responses provided.

Cost-Benefit Analysis

In an effort to develop a framework that would facilitate an evaluation of the questions on the survey in the context of both their usefulness and the difficulty of obtaining quality information, the Census Bureau planned to conduct a cost-benefit analysis. For this analysis, it developed 19 decision criteria, 13 that measure each question’s utility and ability to produce high-quality data and 6 that measure the difficulty of obtaining the information: see Box 6-2. Based on these criteria, a composite score was developed for each question. Because the development of this framework was still in the early stages at the time of writing this report, the panel did not have enough information to assess how well it would function. As emphasized through-

out the report, however, data user input and transparency regarding this process are particularly important.¹

As a first step in the evaluation, federal agencies were asked to provide information for about one-third of the decision criteria, including how the data are used and at what level of geography. The Census Bureau relied on its own records for such information as the time required to complete the survey (available based on the computer-assisted telephone interviews and web paradata) and item nonresponse rates. Questions and complaints received on the Census Bureau’s respondent helpline were also factored in, as well as feedback from field representatives, who were asked to share their perspectives on the difficulties associated with administering the questions.

The Census Bureau’s plan was to first apply the cost-benefit analysis to several data items that are considered particularly difficult to collect, and then proceed to look at all of the items on the questionnaire. The items that are considered particularly difficult to collect on the basis of anecdotal information, primarily because they are perceived as sensitive or burdensome by the public, include plumbing facilities, journey to work, income, and disability. All of these items involve multipart questions on the ACS. Based on prior inventories of the way the questions are used, many of these are also heavily used questions, although the purpose of the content review was to understand how the relative value of these items may have changed over the years. For example, data about plumbing facilities are used as an indicator of housing quality and to identify areas eligible for public assistance programs and rehabilitation loans. The questions are also used to locate areas in danger of ground water contamination and waterborne diseases, especially in rural areas. However, the condition of the housing stock has improved dramatically since questions about plumbing were first introduced in the 1940 census, and the availability of alternative data sources that capture this information, such as administrative records, may have also become more widespread.

Broader Data User Input

The Census Bureau content review was focused on obtaining information from federal agencies about their uses of the data, primarily because the applicable statutes are also at the federal level. The Census Bureau also provided an opportunity to other data users and the general public

______________

¹The Census Bureau issued the results of its first-round cost-benefit analysis of the ACS content in a request for comment on October 31, 2014 (https://www.federalregister.gov/articles/2014/10/31/2014-25912/proposed-information-collection-comment-request-theamerican-community-survey-content-review-results). The next steps in the process are for the Census Bureau to review the comments provided and then send a proposal to OMB with the revisions proposed to the ACS content.

to provide feedback using an online form, but this effort was not intended to be as systematic or comprehensive as the review of the federal uses of the data. Because the use of the ACS data at the subnational level often involves the distribution of funds that are part of federal programs, federal agencies were also asked to indicate if other entities use the data for the purposes listed as justifications by the agency (e.g., if the data are used for the allocation of assistance to states, then states might be the other users of the data). However, the information provided by federal agencies about other data users is likely to be fairly limited and can by no means provide a comprehensive view of the variety of ways the data are used outside of programs associated with the federal government.

To understand how the ACS is meeting its goal of being a useful source of information for small geographic areas and populations, much broader outreach is needed to data users beyond the federal level. This is a priority in the context of content management. The establishment of a standing group to provide ongoing data user input (see Recommendation 1, in Chapter 1) can also help with guidance in accomplishing this task, and the resources invested into broader outreach can lead to the support needed to manage the content of the ACS. As emphasized throughout the report, although national- and state-level ACS data have many important uses, the panel believes that the ability of the ACS to provide data with levels of disaggregation not available from other surveys is what makes the survey unique and argues that this is an important consideration when evaluating the survey’s content.

POSSIBILITIES FOR MODIFICATION

The content review is expected to yield useful information on the uses and justifications for the ACS data, at least in the context of federal programs. The information gathered about the geographic levels at which the data are used will be particularly helpful in understanding how well the ACS is able to meet these needs. As discussed above, it is essential to supplement this research with a solid understanding of other uses of the data, particularly at the small area level. That research can serve as a guide for decisions about possible future revisions or modifications to the questionnaire, including

• dropping questions that are no longer needed,
• adding new questions,
• evaluating the possibility of obtaining some of the data from other sources (e.g., administrative records), and
• redesigning the survey based on what is learned about the importance and geographic needs for the questions.

Dropping Questions

The content review could reveal that some of the questions have become obsolete, are no longer needed for other reasons, or are of relatively little value, while at the same time increasing the burden on the respondents. The challenge for the Census Bureau is that even if the circumstances that led to the initial justification of the questions have changed over the years, it is likely that most of the data that are made available through the ACS are used by some stakeholders for a variety of planning, research, or other purposes, and that these stakeholders would lament the loss of the data. Assembling as much information as possible about the uses and quality of the data, as well as about the difficulties (or lack thereof) associated with collecting those data, is important for deciding which questions can be dropped, in order to ensure that once the decisions are made, they can be implemented without significant unanticipated objections from stakeholders.

Adding New Questions

Under the Paperwork Reduction Act, the practical utility of all federal data collections has to be demonstrated, and respondent burden has to be kept to a minimum. Because participation in the ACS is required by law for those selected into the sample, additional guidelines exist to ensure that only questions that are well justified are included in the survey. The Census Bureau’s current estimate is that it takes approximately 40 minutes for an average household to complete the ACS, and OMB has indicated that the goal is to keep the time required to respond to the survey fixed. In other words, new questions are very unlikely to be added without other questions being dropped.

Due to the long history of the survey—with its roots in the decennial census—the guidelines and practices for adding new questions have evolved over the years. As noted above, historically, questions have generally only been added to the ACS if there was a legislatively based requirement to do so. The frequency and level of geography needed for the data collection, as well as what is known about whether alternative sources exist for the data, are currently also taken into consideration. The criterion of whether

the data are required at the small area level is especially important, given the objective of the ACS to meet small area needs. The evolution of the guidelines appears to have contributed to some confusion, in particular around the meaning of the term “programmatic” data need. For the long-form survey content, this was defined as data necessary for Census Bureau operations (U.S. Census Bureau, 2006). However, “programmatic” in the context of the ACS has sometimes been used to refer to data needs for program planning, implementation, or evaluation by other agencies that are not otherwise classified as mandatory or required needs. For example, the content review asked agencies to describe these types of programmatic needs for ACS data.

Regardless, the Census Bureau’s content review was an effort to ensure that the ACS remains the appropriate vehicle for the collection of the data already on the survey and it was not intended to provide insights into the possible need for new questions. Naturally, the potential need and wishes for additional questions far exceed the ability of the survey to accommodate new content, which underscores the importance of having clear guidelines for what can be added and a consistent and transparent process for accommodating the needs that emerge (see below).

Obtaining Data from Other Sources

An important question is whether new data sources, such as administrative records, that may have become available over the years, could provide an alternative to collecting the data from respondents to the ACS. As part of the content review, federal agency representatives were asked to describe how they would address their need for the ACS data they use if the data were not available through the survey. It is unclear, however, to what extent agencies were able to provide substantive responses, especially those without an existing research program actively exploring such alternatives.

The Census Bureau’s current research on coverage rates (see Chapter 4) represents the foundations for evaluating whether administrative records could also be used to replace some items on the questionnaire or possibly enhance the ACS content in other ways. Another important step is to evaluate the accuracy of the data available from administrative records and whether there are systematic differences between these records and the responses provided by respondents to the survey. To date, few Census Bureau studies on administrative records have examined these issues. One interesting exception is an exploratory study comparing self-reported housing values to housing values available from the CoreLogic tax roll database (Kingkade, 2013). The researchers were able to match 80 percent of the single-family owned homes from the 2009 ACS to the records in the CoreLogic database. Their preliminary analysis indicated differences

between the home values in the two databases, systematically related to the characteristics of the householder, the household, and the local area in which the property was located.

Conducting research on the potential use of administrative records to replace items on the questionnaire is not a priority area among the administrative records research projects currently being pursued by the Census Bureau. However, some of the items that have been considered as suitable candidates for investigation based on the data sources available include other housing items, such as real estate taxes, year built, and number of rooms, as well as veteran status, place of birth, public health insurance coverage, and income. The content review could shed further light on whether there might be promising research projects to pursue.

Survey Redesign

If the content review indicates that only a subset of the current ACS questions are heavily used at the smallest geographic levels, then it may be possible to reconceptualize the survey as a set of “core questions” that are administered using the current schedule and sample size and other questions that could be administered less often or to only a portion of the overall ACS sample. Approaches such as subsampling, matrix sampling, or special modules combined with a set of core questions administered to the full sample could provide a mechanism for increasing content while at the same time reducing respondent burden.

A change such as this would unavoidably increase the complexity of an already complex survey, both in terms of survey operations and the analysis of the data. However, if for some of the questions a much smaller sample is sufficient, then this approach could greatly improve the ability of the ACS to satisfy a larger number of stakeholders and possibly address concerns related to respondent burden. The impact on the availability of data for not only small geographic areas, but also small populations, would have to be carefully considered on the basis of what is learned about how the data are used. In addition to relying on a thorough understanding of how the data

are used, considering what the ACS would look like in terms of content if it was being developed today would also help in informing the assessment of whether a redesign of this nature might be worthwhile.

A possible framework for a reconceptualization of the way the ACS content is collected could focus on the following priorities: (1) which questions are most needed, (2) what level of disaggregation is needed for the data, and (3) what is the optimal survey design to meet these requirements. To determine the best way of collecting data, it will be important to consider not only the full ACS sample, subsamples, and supplements, but other sources as well, such as administrative records and other surveys.

THE ACS CONTENT CHANGE PROCESS

As noted at the beginning of this chapter, OMB plays a major role in determining the content of the ACS. To advise the OMB on policies and practices for the ACS, in 2012 the Interagency Council of Statistical Policy Subcommittee for the ACS (ICSP-SACS) was formed. According to the charter of the ICSP-SACS, the unique scope of the ACS makes it “a national resource for which the Federal statistical system is a steward” (Interagency Council of Statistical Policy Subcommittee for the American Community Survey, 2012, p. 1). The ICSP-SACS is cochaired by OMB’s chief statistician and the Census Bureau director, and membership includes three heads of federal statistical agencies on a rotating basis.

Other groups assisting in the management of ACS content include the ACS Content Council at the Census Bureau and the OMB Interagency Committee for the ACS. The ACS Content Council is an internal Census Bureau group that reviews proposed content changes and provides input on their impact. The OMB Interagency Committee for the ACS is composed of representatives of federal agencies that have an interest in ACS data and is the primary source of requests for content changes, as well as a vehicle for communicating proposed changes to federal stakeholders.

The ICSP-SACS charter for the ACS includes guidelines for content changes (Interagency Council of Statistical Policy Subcommittee for the American Community Survey, 2012):

• Federal agencies consult with the Census Bureau’s ACS Office and the Statistical and Science Policy Office at OMB and then submit a request for change to the ICSP-SACS.
• ICSP-SACS makes a recommendation to OMB regarding the requested changes.
• If OMB approves the changes, then an interagency committee is formed to draft new questions or revise existing questions.

• The Census Bureau conducts cognitive testing in English and Spanish, followed by field testing in all modes of data collection.
• The results of the testing are submitted along with a formal request and justification from the agency to make the change.
• The ICSP-SACS reviews the request and makes a recommendation to the OMB chief statistician and the Census Bureau director.
• A formal submission is made to OMB, in accordance with requirements of the Paperwork Reduction Act, including a Federal Register notice and public comment period.

According to a timetable provided to the panel, the typical content change process can take 5-1/2 years from the time of the agency’s initial submission for the proposed changes to the time the Census Bureau begins collecting data using the new questions. To introduce some predictability in the process, the Census Bureau has historically scheduled content tests on a 5-year cycle, and the testing itself takes close to a year. Moreover, funding cuts have further extended this extremely lengthy schedule.

In terms of dropping questions, the charter of the ICSP-SACS states that this should be considered if questions are no longer needed because of a legal, regulatory, or administrative change or because there is not enough evidence of “regular use of estimates at small areas by any Federal Government program or by other users” (Interagency Council of Statistical Policy Subcommittee for the American Community Survey, 2012, p. 4). The guidelines do not provide quite as much detail on the process for dropping questions as they do on adding questions.

The Census Bureau does not have systematic records on the details of how each of the questions was added to the ACS over the years and how decisions were made about questions that were dropped. However, based on the experience of the past few years and discussions with stakeholders, it appears that stated processes are not always followed. When new questions have been added in the past, there were often special circumstances that resulted in exceptions and a need to deviate from the prescribed process. The process for dropping questions can be especially difficult because pressures from data users and other stakeholders can be significant. To a great extent, this reflects the challenges associated with the task of managing a survey that has a complex role in the statistical system and a wide range of stakeholders.

However, despite the difficulties, following a clearly defined, systematic, and transparent process is the best way to maximize the utility of the survey to the greatest number of users and reduce the risk of controversies. These low-cost investments can have substantial payoffs in the long run. Some exceptions to a predefined process for content modifications are unavoidable, especially if the process cannot be shortened, but they need

to be exceptions, not the norm. There may also be ways of shortening the entire process or customizing it, depending on whether the changes can be described as minor (such as small wording changes) or major (such as adding new questions). For example, separate guidelines could be developed for pretesting with a much shorter process, for minor changes.