Reengineering the Survey of Income and Program Participation (2009) / Chapter Skim
Currently Skimming:
4 Innovation in Design and Data Collection
4 Innovation in Design and Data Collection
Pages 97-126
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 97... ...
Regularly scheduled topical modules will no longer be included in the redesigned SIPP, although some prior topical module content will be incorporated into the primary survey instrument, and federal agencies may pay for supplemental questions to be asked between annual interviews.
|
From page 98... ...
Monthly data on incomes, employment, program participation, health insurance coverage, and demographic characteristics of the household allow analysts to study transitions into marriage and divorce, transitions into and out of poverty, and transitions in health insurance coverage, at a monthly frequency. Monthly data also make SIPP particularly well suited for assessing eligibility for major transfer programs, since program rules typically depend on economic and demographic characteristics in the month or months prior to application.
|
From page 99... ...
Annual income for the first calendar year of the conventionally collected SIPP panel requires data through Wave 4, which will be affected by three waves of attrition beyond the initial sample loss at Wave 1. In the 2004 SIPP panel, the cumulative sample loss after four waves was 28 percent compared with a Wave 1 nonresponse rate of 15 percent (from information provided by Census Bureau staff; see also Table 2-1 in Chapter 2)
|
From page 100... ...
The Census Bureau and Panel Study of Income Dynamics conference highlighted many of the reasons the Census Bureau is envisioning that an event history methodology may play a key role in the reengineered SIPP's efforts to reduce burden on respondents, reduce program costs, improve accuracy, and improve timeliness and accessibility. Belli (2007)
|
From page 101... ...
Not only must the bureau design an effective pretesting program for the EHC methodology, but it must also make its survey reengineering plans for SIPP sufficiently flexible so that it can modify its plans if the pretesting reveals unanticipated, negative evidence on the likely success of the proposed methodology. Paper EHC Test The Census Bureau administered a paper test of the EHC approach that was completed in June 2008.
|
From page 102... ...
Moreover, it will provide the first available information on the ability of households to recall spells of program participation and amounts of monthly income. Nevertheless, an extensive program of design and research must be conducted to assess the EHC approach.
|
From page 103... ...
This can be done by making appropriate power calculations and then drawing appropriately sized test samples, perhaps augmented by oversamples of program recipients drawn from administrative records. If too few program participants are in an EHC test sample, it will be extremely difficult for the Census Bureau to assess whether the EHC can provide accurate month-to-month information on program participation for sampled individuals.
|
From page 104... ...
The panel does not have enough detail on the 2010 one- or two-wave electronic prototype test to fully assess its ability to resolve questions about whether the EHC approach can adequately replace the traditional SIPP interview structure. Our understanding is that the Census Bureau will not use respondents to the 2008 traditional SIPP panel as the sample for the electronic EHC because of a concern that doing so could compromise responses to the traditional interviews for some or all waves following the 2010 EHC test.
|
From page 105... ...
Overlap of Traditional and Reengineered SIPP Panels While the panel thinks the Census Bureau's EHC electronic prototype plans are promising, it is clear that the knowledge base for EHC methods is not yet sufficiently well developed to have confidence that the approach can be used to generate data of equal or better quality than found in the traditionally collected SIPP. The paper test prototype provides only limited information on data quality for the reasons given above.
|
From page 106... ...
If the expense of having two SIPP surveys in the field is prohibitive, cost savings could be achieved by making the 2012 traditional SIPP panel smaller than prior panels. There is another reason why it is critical to field overlapping traditional and reengineered SIPP panels.
|
From page 107... ...
In turn, respondent burden may contribute to item nonresponse, poor quality responses, and attrition from the survey. It is essential, as the Census Bureau evaluates its electronic EHC prototype and implements the overlapping redesigned and traditional SIPP panels, that it not only carefully examine the ability of the EHC approach to generate accurate month-by-month transitions in employment, earnings, household structure, and program participation, but also determine whether the burden on respondents from the redesigned questionnaire is not so taxing as to degrade the overall q uality of responses.
|
From page 108... ...
The Census Bureau should study the trade-offs in survey quality between longer versus more frequent interviews as part of its research and development program for the reengineered SIPP. Seam Bias Concerns The phenomenon of seam bias, in which a large fraction of transitions in employment status, insurance coverage, or program participation occurs between SIPP waves (see Chapter 2)
|
From page 109... ...
With the proposed annual interviews and a fixed, calendar-year reference period, the Census Bureau runs the risk that three moderately sized seams per year, which are invisible crosssectionally, will be replaced by one very large seam between December of one year and January of the next. LENGTH AND OVERLAP OF PANELS Length SIPP panels are currently 4 years in length; for the period 1984-1993, they were generally 2-3 years in length.
|
From page 110... ...
. Arguing against very short panels for SIPP, such as 1 or 2 years, is that SIPP users often want to look at changes in income and program participation before and after a major change in program rules or a major event, such as a recession, which requires information on the same respondents over a longer period than just 1 or 2 years.
|
From page 111... ...
Unless the Census Bureau can find an alternative way to achieve this same result, it is important to conduct at least one pair of overlapping panels relatively early in the history of the reengineered SIPP. CONTENT EVALUATION A key element of reengineering SIPP and keeping it relevant to user needs concerns the survey content.
|
From page 112... ...
. First, the panel recommends that SIPP management staff and the advisory group make a periodic top-to-bottom review of SIPP survey content, ensuring that questions are consistent with SIPP's core mission and that each question is worth the cost -- both the dollar cost and the opportunity cost in terms of questions forgone.
|
From page 113... ...
When Aid to Families with Dependent Children was abolished, for example, or Medicare Part D was enacted, the SIPP questionnaire had to evolve. We expect that future changes in health insurance coverage and in the treatment of immigration will have implications for SIPP survey content.
|
From page 114... ...
-- should scrutinize and conduct, on an annual or semiannual basis, evaluations of SIPP survey content. These evaluations should focus on improving survey questions, cognitive understanding of questions, response rates, benchmarking survey responses against external, reliable sources, and imputation and editing procedures.
|
From page 115... ...
, and citizenship status have been collected from all adults ages 15 and older in a topical module, usually as part of Wave 2. Starting with the 1996 panel, limited information on visa status was added, including information about whether the respondent was originally admitted as a legal permanent resident, and whether and when the person subsequently converted to this status.
|
From page 116... ...
For example, among Mexican-born adults in the 2004 SIPP panel who reported on immigration status, 33 percent (weighted) said they were not admitted as a legal permanent resident, had not naturalized, and had not converted to this
|
From page 117... ...
If SIPP foreign-born respondents were successfully matched to OIS admission and naturalization records, the information in the administrative records could be used to improve the quality of SIPP data on citizenship and immigration status. For example, matched data could be used to evaluate the accuracy of responses generated by alternative survey methodologies (e.g., in-person interviews versus self-administered questionnaires, or the random response method versus standard questions)
|
From page 118... ...
The Census Bureau is capable of timely dissemination of data, as evidenced by the efficiency of the processing of the CPS ASEC supplement and occasional past SIPP panels. The bureau needs to ensure that the same type of management attention and coordination is applied to ensure timely delivery of future SIPP panels, particularly in years when the survey instrument or processing procedures are being updated, which occurs periodically.
|
From page 119... ...
The panel thinks that 1 year between the end of a survey and data release should be an achievable target. Enhancing Access to SIPP One common complaint from current and prospective SIPP data users is the difficulty associated with working with SIPP files.
|
From page 120... ...
SIPP Management and Budget As we recounted in Chapter 2, SIPP has a unique position among the Census Bureau's data creation activities for the household sector. Unlike other surveys of people and households that the Census Bureau conducts, SIPP does not have a government client outside the Census Bureau or a federally mandated set of reports that are based on the survey.
|
From page 121... ...
Not having an external client, such as the Bureau of Labor Statistics (which has a collaborative and financial stake in the monthly CPS) , or a set of regular reporting requirements, as with the decennial census and the American Community
|
From page 122... ...
CONCLUSIONS AND RECOMMENDATIONS Event History Calendar Approach Conclusion 4-1: The Survey of Income and Program Participation (SIPP) is the only national survey that provides information on the shortterm dynamics of employment, income, program participation, and other family characteristics, and its monthly time frame is essential for many applications.
|
From page 123... ...
If the experimental results indicate that the quality of data on income and program dynamics is significantly worse under the event history calendar approach than in the traditional SIPP, the Census Bureau should return to a more frequent interview schedule, say, every 6 months, devise other methods to improve data on short-run d ynamics, or revert to the traditional SIPP with 4-month interviews using standard questionnaires. Recommendation 4-2: To ensure not only adequate evaluation of a reengineered Survey of Income and Program Participation (SIPP)
|
From page 124... ...
Recommendation 4-4: The Census Bureau should study the tradeoffs in survey quality and respondent burden in comparison to survey costs between longer but less frequent event history-based interviews in a r eengineered Survey of Income and Program Participation (SIPP) and more frequent interviews in the traditional SIPP.
|
From page 125... ...
The group would provide a sounding board for the Census Bureau's plans to develop appropriate survey content in a reengineered SIPP and advise the bureau on appropriate modifications to survey content as policy developments occur, such as health care and immigration reform Timeliness Conclusion 4-7: The release of Survey of Income and Program Participation (SIPP) data is often not timely.
|
From page 126... ...
The budget should always include adequate research and development funding, since SIPP is a major ongoing survey that requires regular evaluation and improvement.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.