Appendix K

Study Design Strategies for Reducing the Effects of Selection Bias in Studies Comparing WIC Participants to Others

In Chapter 4, the committee reviewed study designs that could to some degree ameliorate the challenges to evaluating the effects of the Special Supplemental Nutrition Program for Women, Infants, and Children (WIC). In particular, specific types of study designs that could address the problem of selection bias, or other biases due to reverse causation or measurement error, were reviewed. In this appendix, the committee defines these types of study designs, and proposes ways that the U.S. Department of Agriculture’s Food and Nutrition Service (USDA-FNS) could apply these designs to the evaluation of the WIC program.

REGRESSION DISCONTINUITY

A regression discontinuity (RD) design compares people who are just below an eligibility threshold for a program or treatment with those who are just above it. Because the likelihood of an individual falling just above or below a threshold is not affected by self-selection, this method yields high internal validation,¹ provided that these individuals do not manipulate their presence below or above the threshold (the method typically includes tests for this). The types of thresholds that are relevant to WIC and can potentially be used in RD designs include (1) income eligibility, (2) age after which one can no longer participate in the program, and (3) age at which the value of the program changes (i.e., the value of the food package

__________________

¹ Internal validation can be indicative of causality regarding the relationship between WIC participation and measured outcomes.

changes). Versions of this design, which rely on comparing individuals with incomes just below and just above the eligibility threshold for WIC participation, require researchers to use the measure of income that is applied by the program. Similarly, versions of the design which rely on age thresholds require researchers to identify age thresholds relevant to the program (e.g. age 5 when children are no longer eligible for WIC). RD designs yield unbiased estimates of the impact of WIC, provided that the identifying assumptions are satisfied for individuals that are right at the eligibility thresholds.² However, RD designs are not necessarily applicable to the WIC participants with the most need, who might have lower income-to-poverty ratios and may be affected differently by WIC.

Potential USDA-FNS Applications for WIC

Research studies could target comparisons at 185 percent of poverty, at age 5 when eligibility ends, and at eligibility/takeup at 1 year when formula is no longer provided. Research examining comparisons at the income threshold would compare WIC participants near but below the top of the income threshold (170–185 percent) with non-WIC participants just above the income threshold (185–200 percent), while controlling for the income-to-poverty ratio.³ Comparisons of the two age thresholds would involve what is known as a “fuzzy RD.” At age 5, when children that meet the income criteria are no longer eligible, they would be compared to themselves at periods before and after program participation, while controlling for age. At age 1, some families may choose not to participate (particularly formula-feeding families) because the value of the package is reduced with the loss of formula. Thus, if one could follow families who use formula throughout the first year and into the second year, comparisons between families who fail to recertify before and after age 1 might be useful. The RD approach could be made more feasible if USDA-FNS encouraged states to make administrative data on health outcomes and precise income-to-poverty and age (in months or exact days) available for large numbers of potential WIC participants (including participants and nonparticipants). Another strategy could be to collect data on all Medicaid participants near

__________________

² Other assumptions for RD are either that the potential outcomes are continuous and have a number of continuous derivatives or that within some window close to the income eligibility threshold, the side of the threshold is randomly assigned. This is easy to test for income (i.e., are there more families with income levels just below the 185 percent of poverty threshold) and less necessary for age (i.e., age is generally well-captured in administrative data).

³ The income-to-poverty ratio is the ratio of family income to the federal poverty guideline. It is this ratio which must be below 1.85 for families to be eligible for WIC.

these same WIC age and income thresholds as all Medicaid recipients are adjunctively eligible for WIC. These data will provide estimates that have high internal validity for those near the age or income thresholds (and can be indicative of causality) but may not be externally valid for those far from the age or income thresholds.

COMPARISON GROUP

In a comparison group (CG) design, WIC participants are usually compared with income-eligible nonparticipants. In contrast to RD studies, these designs can be applied to groups of the lowest-income WIC participants. However, a major drawback of CG designs is that it is not possible to know if the analysis has accounted for all unobservable differences between the two groups. The analysis can be made more robust by including an additional comparison between the treatment and control groups when program changes are taking place in what is known as a differences-in-differences or interrupted-time-series design (i.e., a series of observations over time interrupted by introduction of an intervention). Thus, studies in which program changes are implemented randomly across locations are best suited for this design.

Potential USDA-FNS Applications for WIC

Application of this approach could be optimized if the timing and scope of rule changes in WIC and other programs affecting WIC use were made available. The period of new food package implementation offers one such opportunity. Larger sample sizes and access to linked WIC administrative data on WIC eligibility and participation would allow for useful outcomes as long as nonparticipants are somehow included. It is also key to have data on individuals who are potentially eligible but not enrolling in the program. This approach relies on assumptions of common trends and the absence of changes in the composition of the comparison and treatment groups after treatment.

INSTRUMENTAL VARIABLES

The instrumental variables (IV) approach considers a population of participants and nonparticipants whose decision to participate was based on a factor that is: strongly correlated with participation, cannot be directly related to the outcome, and only affects the outcome through participation. An example is distance of residence from the WIC clinic (ideally combined with variation in locations of clinics driven by clinics closing and opening). Distance (particularly driven by openings and closings) plausibly affects

use of WIC but should not otherwise have direct effects on dietary intake. Although this is a sound method for obtaining causal estimates, it can have low external validity (i.e., results will not be generalizable). Some researchers who have tried to use IV to estimate the impact of WIC have concluded that they were unable to identify an instrument that works well to predict WIC use.

Potential USDA-FNS Applications for WIC

As an example of how IV has been applied to evaluate WIC, Rossin-Slater (2013) used, as the instrument, whether a mother resided close to a WIC clinic at the timing of conception of her oldest child. Controlling for mothers’ fixed characteristics, this instrument proved to be a strong predictor of WIC participation in Texas during a period when many clinics were opening or closing. This type of instrument (relying on distance to a location where eligibility for a program is assessed) is common in the evaluation of the effects of other programs. Another IV application would rely on state differences in assessment of eligibility by income (e.g., disregards family units, the way that eligibility is assessed for the other programs such as Temporary Assistance for Needy Families, Supplemental Nutrition Assistance Program [SNAP] and Medicaid, which confer automatic WIC eligibility). For example, Ziliak (2016) demonstrated that states where SNAP uses broad-based categorical eligibility (which raises the gross income threshold for eligibility) have higher SNAP participation. This might permit use of these SNAP rules for IV estimation.

FIXED EFFECTS

The fixed effects (FE) approach uses variation in WIC participation across time, time and place, siblings within a family, or other variables, to estimate effects. This approach has some disadvantages, including lack of control for unobserved factors associated with participation in WIC that change over time (e.g., shocks to the economy) and lack of sufficiently detailed longitudinal data on participation over time and on outcomes.

Potential USDA-FNS Applications for WIC

This approach would be made more useful by encouraging more sharing of WIC administrative data and WIC data from other data systems, including data on families with not all siblings enrolled, as well as data from locations where program changes affect eligibility and enrollment.

REFERENCES

This page intentionally left blank.