Nonresponse in Social Science Surveys A Research Agenda (2013) / Chapter Skim
Currently Skimming:
1 The Growing Problem of Nonresponse
1 The Growing Problem of Nonresponse
Pages 7-39
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 7... ...
Why is response declining and how does it increase the possibility of inaccurate survey results? Most important, with the advancement of reliable social science research at stake, what are effective methods for increasing the response rate for household surveys?
|
From page 8... ...
Finally, we discuss the current state of knowledge about some of the practical implications of growing nonresponse rates for survey costs and survey management. The evidence presented in this chapter leads to several recommendations for a research agenda aimed at filling some of the current gaps in our knowledge concerning trends in nonresponse as well as the reasons for those trends.
|
From page 9... ...
This volume has been subsequently amended and updated, most recently in 2011.2 The survey research community has, under the leadership of AAPOR, 1 he T conceptual framework that is presented here was drawn largely from the survey methodology field, with the understanding that every field of social science has concepts and frameworks that might shed light on the problem of growing nonresponse, its causes, and its consequences. 2 he AAPOR work in this field was preceded by a project to develop standard definitions T by the Council of American Survey Research Organizations.
|
From page 10... ...
The process of reaching a consensus definition of nonresponse rates is illustrated by the process within the federal government, which spon 3 Break-offs may occur at several points in the interview. As a rule of thumb, an interview is considered a break-off refusal when less than 50 percent of applicable questions are answered; those with 50–80 percent answered are considered partial responses; and those with 80 percent or more answered are considered completed responses (American Association for Public Opinion Research, 2011, p.
|
From page 11... ...
. These studies found little consistency in how the federal surveys that they reviewed measured and reported nonresponse rates.
|
From page 12... ...
OMB has subsequently issued policy guidance to standardize nonresponse rate definitions and to set targets for acceptable survey response rates -- no easy task since government surveys vary widely in their mandate, sample design, content, interview period, mode, respondent rules, and periodicity. The main source of this guidance is the OMB Statistical and Science Policy Office release titled Guidance on Agency Survey and Statistical Information Collections -- Questions and Answers When Designing Surveys for Information Collections (Office of Management and Budget, 2006)
|
From page 13... ...
cited multiple sources to support the consensus view that nonresponse rates in U.S. household surveys have increased over time.
|
From page 14... ...
. We supplement the work of Brick and Williams with a discussion of results of analysis performed by the Census Bureau, the Bureau of Labor Statistics, and the Federal Reserve Board on response rates in the surveys they conduct.
|
From page 15... ...
General Social Survey The GSS is a social survey that collects data on demographic characteristics and attitudes. It is conducted by NORC at the University of Chicago using face-to-face interviews of a randomly selected sample of non- TABLE 1-1 National Health Interview Survey Unconditional Response Rates, 1997–2011 (in percentage)
|
From page 16... ...
41) concluded that NHES has experienced a much greater increase in nonresponse rates, with an annual increase of 2.3 percentage points per year over the period, than
|
From page 17... ...
NIS response rates have fallen fairly consistently over a period of 16 years (see Table 1-3)
|
From page 18... ...
The Council of American Survey Research Organizations (CASRO) response rate equals the product of the resolution rate, the
|
From page 19... ...
The mail survey approach had the added advantage of including households that did not have landlines, which led to better responses in 5 he T description of resolution rate, screener completion rate, interview completion rate, and CASRO response rate is taken from Centers for Disease Control and Prevention, National Center for Immunization and Respiratory Diseases, and National Center for Health Statistics (2011, p.
|
From page 20... ...
. The CASRO response rate assumes that the unresolved numbers contain the same percentage of eligible households as those records whose eligibility or ineligibility is determined.
|
From page 21... ...
The authors did not find evidence that lower unemployment or the increased use of call-screening devices in the 1990s contributed to the rise in refusals and non-contacts. They conjectured that the rapid growth in sales phone calls 7 he response rate shown here uses every sampled phone number with the exception of T those known to be ineligible in the denominator and uses partial interviews in the numerator (AAPOR response rate RR4)
|
From page 22... ...
Table 1-4 shows the initial interview nonresponse rates for the years 1990 and 2009. Initial nonresponse rates are those for the first interview in multi-interview surveys.
|
From page 23... ...
. 90 85 80 75 70 CPI-Housing Percentage 65 CE Interview CE Diary 60 TPOPS 55 ATUS 50 45 40 Dec-00 May-01 Jan-03 Jul-05 Dec-05 May-06 Jan-08 Jul-10 Dec-10 Oct-01 Sep-04 Feb-05 Oct-06 Sep-09 Feb-10 Jun-03 Jun-08 Mar-02 Nov-03 Mar-07 Nov-08 Aug-02 Apr-04 Aug-07 Apr-09 Year FIGURE 1-3 Response rate trends for major Bureau of Labor Statistics household surveys, December 2000–December 2010.
|
From page 24... ...
. Response Rate Trends by Survey Type The experience of this illustrative set of surveys provides evidence that nonresponse rates continue to increase in all types of cross-sectional surveys, with little to suggest that the trend has plateaued.
|
From page 25... ...
individuals over a long period of time and thus tend to suffer from attrition as well as initial nonresponse, the dropout rates from panel surveys, after the initial baseline entry interviews, tend to be smaller than the nonresponse rates in cross-sectional surveys. Of course, nonresponse rates in the baseline interview of panel surveys are often comparable to those in cross-sectional surveys.
|
From page 26... ...
. There are many different response rates that can be calculated for panel surveys, and each has its separate purpose (Cheshire et al., 2011)
|
From page 27... ...
Table 1-7 documents the decreasing response and TABLE 1-6 Type of Survey and Completion Rate for NLSY79 by Survey Year Completed Interviews Mode Not Not Completion Year Personal Telephone Available Interviewed Rate (%) 1979 11,863 548 275 -- -- 1980 11,493 648 0 545 95.7 1981 11,541 654 0 491 96.1 1982 11,066 1,054 3 563 95.6 1983 11,897 324 0 465 96.3 1984 11,422 646 1 617 95.1 1985 9,941 953 0 713 93.9 1986 9,726 929 0 952 91.8 1987 1,126 8,998 362 1,122 90.3 1988 9,494 920 51 1,142 90.2 1989 PAPI: 8,832 1,469 3 1,002 91.4 CAPI: 252 49 1990 PAPI: 6,972 1,032 2 1,171 89.9 CAPI: 2,145 285 1991 7,773 1,241 4 946 90.5 1992 7,848 1,164 4 948 90.5 1993 7,917 1,081 13 953 90.4 1994 7,948 933 10 1,073 89.2 1996 7,594 1,042 0 1,328 86.7 1998 6,330 2,069 0 1,565 84.3 2000 5,420 2,613 0 1,931 80.6 2002 2,317 5,407 0 2,240 77.5 NOTES: PAPI interviews are those conducted with paper survey instruments and pencilentered responses; CAPI interviews are administered using a laptop computer and an electronic questionnaire that captures respondent-, interviewer-, and machine-generated data.
|
From page 28... ...
The baseline response rates for AHEAD, WB, CODA, and EBB were 80.4 percent, 70 percent, 72.5 percent,
|
From page 29... ...
The baseline (first wave) response rate is based on completed interviews of all individuals deemed eligible for HRS.
|
From page 30... ...
These include incentive payments, payments for updating locating information during non-survey years, the use of experienced interviewers, persuasion letters sent to reluctant respondents, informing respondents about upcoming data collection waves, sending thank-you letters to responding households, and training interviewers to handle refusals. Even though response rates are exemplary in the case of the PSID relative to other panel surveys, it does suffer from high cumulative nonresponse rates due to an attrition rate of 50 percent (Cellini et al., 2008)
|
From page 31... ...
TABLE 1-9 PSID Response Rates and Sample Sizes, 2003–2005 Non-Split-Offs Split-Offs Response Rate Number of Response Rate Number of Total Number Sample (%) Interviews (%)
|
From page 32... ...
. When the authors examined the relationship between survey nonresponse rates and nine selected characteristics that might be expected to influence response rates by affecting accessibility or cooperation, they were able to identify four variables that were highly correlated with nonresponse rates for the four surveys they studied: the percentage of families with children under the age of six; the percentage of single-person households; the violent crime rate; and travel time to work.
|
From page 33... ...
These ideas are captured to one degree or another in three main theories -- social capital theory, leverage–saliency theory, and social exchange theory -- which are summarized below. Social Capital Theory Social capital theory may help to explain the social and psychological underpinnings of the interpersonal relationships that promote trust and cooperation and thus promote the willingness to respond to surveys.
|
From page 34... ...
Brick and Williams state that they would find "a rigorous investigation of the relationship between social capital and nonresponse rates to be extremely helpful" (Brick and Williams, 2013, p.
|
From page 35... ...
To the extent that distrust of mainstream institutions is intensifying, rising nonresponse rates may be the result. "Saliency" -- or salience -- refers to the prominence of different attributes of the survey request for a sample person who is deciding whether to participate.
|
From page 36... ...
As the cost of dualframe and ABS designs rose -- and driven in part by the lack of timeliness of the latter -- survey researchers were drawn to Web survey designs. It is possible that a significant portion of the downward trend in response rates is attributable to (a)
|
From page 37... ...
In an attempt to rectify this knowledge gap, in conjunction with survey sponsors, the U.S. Census Bureau has launched a major effort to identify costs associated with survey activities.12 Statistics Canada is exploring how responsive design initiatives can be employed to yield cost information that is useful for analysis and decision making.13 Both of these initiatives are in their developmental stages, but they do constitute a serious attempt on the part of these agencies to understand the costs associated with response levels, and both are aided by the collection of data during the normal survey process.
|
From page 38... ...
When merged with information about nonresponse rates, missing data rates, and key indicator values, costs and data quality can be evaluated simultaneously, and data collection efforts may be managed more effectively. Despite these pioneering efforts, there is no common framework for assessing the relationship between cost and response rates and no quantitative model of the relationship.
|
From page 39... ...
THE GROWING PROBLEM OF NONRESPONSE 39 gated and otherwise restricted communities in the hopes of gaining an interview with a sampled unit. Recommendation 1-7: Research is needed on the cost implications of nonresponse and on how to capture cost data in a standardized way.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.