Reengineering the Survey of Income and Program Participation (2009) / Chapter Skim
Currently Skimming:
Appendix A: SIPP Data Quality
Appendix A: SIPP Data Quality
Pages 127-158
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 127... ...
in areas that are central to the survey's principal purposes or major uses. Topics include the following: • Income • Program Participation • Income Receipt from Multiple Sources • Wealth • Health Insurance Coverage Transitions • Attrition • Representation of the Population Over Time • Seam Bias • Imputation • Wave 1 Bias These topics are discussed in the order that they are listed.
|
From page 128... ...
SIPP estimates of aggregate pension dollars by type were between 95 and 103 percent of independent estimates. However, SIPP's estimate of total earnings, the largest component of total income by far, was 1.8 percentage points below the CPS.
|
From page 129... ...
If that were the case, however, the lower aggregate income obtained in SIPP would have been due entirely to workers reporting lower income from their employment than workers responding to the CPS. SIPP Income Over Time Between 1984 and 1990, the SIPP estimate of total income slipped below 98 percent of the CPS aggregate according to analyses reported by Coder and Scoon-Rogers (1996)
|
From page 130... ...
One element of the switch from a paper and pencil instrument was clearly related to the increased amount of income collected: the maximum amount of wage and salary income that could be reported was increased from $499,997 to $2,099,999. Roemer determined that this change alone added 2 percentage points to the CPS income total relative to the NIPA total. The combined impact of the SIPP and CPS changes over this period was to reduce the ratio of SIPP to CPS total income to 92.5 percent (see Table A-3)
|
From page 131... ...
SOURCE: Roemer (2000:Table 3b) ; data from the 1990, 1991, 1993, and 1996 SIPP panels.
|
From page 132... ...
; data from the 1991 through 1997 ASEC supplements to the CPS.
|
From page 133... ...
TABLE A-3 SIPP Aggregate Income as a Percentage of March CPS Aggregate Income, 1990 to 1996 Survey Reference Year Income Source 1990 1991 1992 1993 1994 1995 1996 Total Income 97.5 98.3 96.5 94.8 91.3 92.0 92.5 Earnings 96.3 97.7 95.2 92.2 89.6 91.2 92.0 Wages and salaries 94.0 93.9 92.2 89.3 86.8 87.1 89.3 Self-employment 124.2 144.9 132.6 129.4 128.6 154.6 131.4 Property Income 104.0 94.7 95.7 110.3 91.5 80.8 80.9 Interest 84.5 82.9 83.6 77.9 71.0 61.1 59.9 Dividends 160.9 116.6 102.6 176.6 114.5 105.1 85.9 Rent and royalties 133.1 122.4 130.1 139.9 125.0 117.9 139.9 Transfers 105.0 104.3 106.5 104.4 98.1 97.5 97.7 Social Security and Railroad Retirement 107.2 107.2 107.5 105.6 98.4 98.8 95.9 Supplemental Security Income 105.3 104.7 112.5 98.5 110.3 111.8 120.4 Family assistance 101.6 102.7 96.8 116.6 119.4 121.7 112.7 Other cash welfare 95.7 130.2 99.6 95.4 75.3 100.1 141.6 Unemployment compensation 97.0 101.2 113.2 111.2 93.7 82.9 85.0 Workers' compensation 75.8 69.0 83.2 76.9 74.4 73.9 114.4 Veterans' payments 112.4 95.1 102.3 90.6 89.3 76.6 81.4 Pensions 95.2 102.8 102.2 103.9 102.0 108.4 111.9 Private pensions 93.4 89.0 89.9 98.1 101.1 106.0 105.4 Federal employee pensions 91.8 108.7 100.1 104.4 110.0 113.6 93.6 Military retirement 102.1 108.7 112.2 121.8 114.0 121.0 174.6 State and local employee pensions 97.6 122.9 124.8 114.8 129.2 125.9 118.3 SOURCE: Tables A-1 and A-2.
|
From page 134... ...
The SIPP instrument was revised again for the 2004 panel to collect the amount of the Medicare premium as a separate quantity, which the Census Bureau could then add to the reported net payment to obtain the gross amount. Finally, SIPP pension income increased from 95.2 to 111.9 percent of the CPS estimate due to the decline in pension dollars collected in the CPS.
|
From page 135... ...
Speculation about possible reasons for SIPP's underreporting of wage and salary income has focused on the possibility that the short reference period may lead SIPP respondents to report their take-home rather than gross pay despite the specificity of the questions. The short reference period, which is clearly helpful in capturing earnings from people with irregular employment, may also contribute to omissions of earned income.
|
From page 136... ...
To show how SIPP and CPS income estimates compare in different parts of the income distribution, Table A-4 presents estimates of aggregate income, by source, for quintiles of the population based on total family income, prepared for the panel. Estimates are presented for 3 calendar years: 1993, 1997, and 2002. The SIPP estimates are from the 1992, 1996, and 2001 panels and, for consistency, are derived from the second year of data in each panel. The CPS estimates are from the 1994, 1998, and 2003 supplements.
|
From page 137... ...
For all other sources, the differences in their shares change little or in an inconsistent way when the top income quintile is excluded. Turning to the results by income quintile, one finds, first, that the ratio of SIPP to CPS total income declines progressively from the bottom to the top quintile and does so in every year.
|
From page 138... ...
TABLE A-4 SIPP Aggregate Income as a Percentage of CPS Aggregate Income by Source of Income and Family 138 Income Quintile: 1993, 1997, and 2002 Quintile of Family Income Bottom Income Source Bottom Second Third Fourth Top Total Four 1993 Calendar Year Total Income 119.5 101.5 95.9 93.0 88.9 94.5 98.2 Wages and salaries 124.8 100.1 92.6 88.7 81.8 88.9 94.0 Self-employment 257.4 133.3 128.8 139.3 178.0 160.3 139.0 Property income 121.7 112.4 101.5 99.0 67.3 83.5 104.2 Social Security and Railroad Retirement 107.2 97.4 98.6 105.7 107.4 102.2 101.8 Supplemental Security Income 111.6 77.8 75.6 89.5 91.5 98.2 98.5 Welfare 99.0 72.4 114.6 180.2 141.0 97.6 97.0 Other transfers 123.5 89.5 84.6 81.2 72.9 87.9 92.3 Pensions 143.6 116.9 110.6 115.9 107.3 113.9 116.3 1997 Calendar Year Total Income 112.2 96.0 93.8 91.8 80.7 89.0 95.3 Wages and salaries 111.7 91.7 89.7 87.9 77.2 84.6 90.6 Self-employment 214.7 152.2 153.0 160.5 169.3 165.6 159.5 Property income 112.6 90.6 74.7 64.2 32.4 49.0 75.9 Social Security and Railroad Retirement 100.0 89.3 97.4 99.1 75.5 93.6 95.7 Supplemental Security Income 122.0 126.1 121.7 166.7 149.0 126.5 125.7 Welfare 122.4 143.5 267.2 420.1 509.9 145.0 140.4 Other transfers 71.6 58.7 57.7 52.5 41.4 54.7 59.3 Pensions 229.0 150.5 136.9 142.9 102.6 134.5 148.6
|
From page 139... ...
2002 Calendar Year Total Income 105.7 97.3 92.8 90.7 83.0 89.4 94.2 Wages and salaries 109.4 93.8 86.0 85.3 75.2 82.4 88.6 Self-employment 219.5 162.4 160.5 158.1 208.6 188.3 163.2 Property income 104.7 90.0 70.7 56.0 38.6 52.3 70.5 Social Security and Railroad Retirement 87.6 90.4 104.4 115.9 94.5 95.4 95.4 Supplemental Security Income 126.6 121.3 139.4 152.6 222.9 131.5 128.5 Welfare 119.5 109.4 176.4 246.6 1,415.5 146.7 128.4 Other transfers 70.9 58.8 64.2 57.4 42.8 58.0 62.4 Pensions 198.4 140.2 155.7 153.5 128.3 147.0 153.9 SOURCE: 1992, 1996, and 2001 SIPP panels and 1994, 1998, and 2003 CPS ASEC supplements.
|
From page 140... ...
In both quintiles, SIPP captured 50 percent more pension income in 2002 than did the CPS. In the top quintile, the SIPP estimate of aggregate Social Security and Railroad Retirement dropped relative to the CPS, as it did in the first and second quintiles, and the relative gain in the capture of pension income was more modest than in the lower quintiles.
|
From page 141... ...
• Supplemental Security Income (SSI) TABLE A-5 SIPP and CPS Estimates of Program Participants and Aggregate Benefits as a Percentage of Administrative Benchmarks, Selected Years 1987 1996 2005 Program SIPP CPS SIPP CPS SIPP CPS Survey Estimate of Average Monthly Participants as a Percentage of Administrative Benchmark FSP 88.1 73.2 84.2 66.3 82.9 56.5 AFDC/TANF 76.4 80.5 79.5 67.0 80.9 63.0 OASI 94.5 88.0 94.2 84.3 97.1 82.9 SSDI 101.3 101.6 90.6 89.7 93.8 78.8 SSI 90.3 72.7 94.4 65.8 102.7 58.0 NSLP 113.7 54.4 111.6 64.2 112.5 48.4 WIC 66.0 56.1 58.2 60.1 Survey Estimate of Aggregate Benefits as a Percentage of Administrative Benchmark FSP 85.9 74.2 79.0 63.1 76.4 54.6 AFDC/TANF 73.0 74.4 77.0 66.8 62.2 48.7 OASI 95.0 89.0 88.7 90.5 97.4 89.7 SSDI 95.4 100.1 79.3 91.9 84.8 81.8 SSI 89.6 76.7 93.4 77.6 110.0 79.4 NOTE: FSP = Food Stamp Program; AFDC/TANF = Aid to Families with Dependent Children/ Temporary Assistance for Needy Families; OASI = Old-Age and Survivors Insurance (Social Security)
|
From page 142... ...
Whether because of poor recall or because respondents sometimes answer on the basis of their current situation, CPS estimates of persons who ever participated in a program sometimes line up with SIPP estimates of average monthly participants. Medicaid provides a good example.
|
From page 143... ...
reports that for 1996 the survey estimates of full reliance on Social Security benefits were 17.9 percent for the CPS and 8.5 percent for SIPP, so the difference between the two surveys appears to have grown between 1996 and 2001. The 1996 difference was reduced only slightly when matched administrative records were substituted for survey data and used to assign beneficiary status.
|
From page 144... ...
Finally, the SIPP estimate of other interest earning assets was only 33 percent of the SCF amount. In contrast to Wolff's findings from the mid-1980s and early 1990s that SIPP matched the SCF in capturing the asset holdings of the bottom two income quintiles, the estimates from the 1996 panel showed that SIPP did not fare appreciably better with the assets of low-income families than with higher income families.
|
From page 145... ...
HEALTH INSURANCE COVERAGE TRANSITIONS Average monthly estimates of health insurance coverage from SIPP compare closely to estimates of health insurance coverage obtained in the National Health Interview Survey (NHIS) , which measures coverage at the time of the interview and therefore is free of recall bias (Czajka and D enmead, 2008; Davern et al., 2007)
|
From page 146... ...
The bias that attrition may introduce into survey estimates makes the level of attrition a serious concern. Sample Loss Table A-6 documents both incremental and cumulative sample loss due to nonresponse by wave in each of the four SIPP panels that started Such data sources include Medicaid administrative files from the Medicaid Statistical Information System, which have been linked to the CPS and NHIS but not SIPP, and Internal R evenue Service Forms 5500 filed by employers and processed by the U.S.
|
From page 147... ...
The estimates of sample loss apply to eligible households. If at least one member of an eligible household responds during a given wave, the Census Bureau collects or imputes data for every other household member.
|
From page 148... ...
Over this same period, nonresponse to the ASEC supplement among respondents to the labor force survey ranged between 8 and 9 percent, with no distinct trend, yielding a combined sample loss that varied between 14 and 16 percent of the eligible households. In other words, the initial nonresponse to the 2001 SIPP panel is still 2 to 3 percentage points lower than the nonresponse to the ASEC supplement.
|
From page 149... ...
The 2001 panel maintained a lower cumulative sample loss through the remaining two waves. Interestingly, the incremental sample loss rates between Waves 8 and 9 were essentially identical across the four panels at about 1.5 percent.
|
From page 150... ...
used Social Security Administration Summary Earnings Records matched to SIPP panel data to compare attriters and continuers with respect to earnings and program benefits over time.12 Even after removing those who left the survey universe, they found that attriters and nonattriters differed markedly with respect to earnings and receipt of program benefits at the beginning of a panel -- that is, before any attrition had occurred. Over time, however, these differences attenuated.
|
From page 151... ...
The implication is that attrition bias in these characteristics is being addressed in the longitudinal weights. It is not possible to evaluate the Census Bureau's adjustments to the cross-sectional weights in the same manner as the longitudinal weights, as there is no attrition-free crosss ectional sample after the first wave.
|
From page 152... ...
To facilitate cross-sectional uses of SIPP data, the Census Bureau provides monthly cross-sectional weights. These weights include an adjustment for differential attrition and a separate "mover adjustment," which offsets the weights assigned to persons who join SIPP households.
|
From page 153... ...
The Census Bureau has tried two alternative approaches to dealing with seam bias: (1) collecting selected data for the interview month as a fifth reference month, which will overlap the first reference month of the next wave, and (2)
|
From page 154... ...
Imputation rates in the CPS were higher -- in large part because the Census Bureau imputes the entire ASEC supplement for respondents who complete only the brief monthly labor force survey that precedes the supplement. In March 1985, 20.1 percent of total CPS ASEC income for 1984 was imputed -- including 17.9 percent of wage and salary income.
|
From page 155... ...
Yet there was no increase in the already high imputation rate for property income, and the imputation rates for wages and salaries and self-employment income increased by only 3 percentage points. Between 1997 and 2002, the imputation rate for pension income grew another 10 percentage points, taking it very near the imputation rate for property income, which grew by 7 percentage points to nearly 50 percent.
|
From page 156... ...
demonstrated that by not taking into account reported Food Stamp Program benefits when imputing major components of income or by not taking account of income eligibility limits when imputing FSP benefits, the Census Bureau was imputing FSP benefits to households with incomes well beyond the eligibility limits or imputing high incomes to households that reported the receipt of FSP benefits. In response, the Census Bureau made improvements to address this particular problem as well as other related problems.
|
From page 157... ...
They may also become bored or learn how to avoid lengthy segments of the interview. Prior to the 1996 redesign, the Census Bureau compared data from overlapping waves in successive panels in a search for evidence of a time-in-sample bias in the reporting of income and benefit receipt in the SIPP.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.