Proceedings of a Workshop
|
July 2016 |
IN BRIEF |
Using Linked Census, Survey, and Administrative Data to Assess Longer-Term Effects of Policy
Proceedings of a Workshop—in Brief
“IF WE BUILD IT, WILL THEY COME?”
Standing committee chair Michael Hout (Department of Sociology, New York University) began the workshop by stating its goal: to better understand the potential impact of the AOS on future research, program evaluation, and policy analysis across the social sciences. The work envisioned—developing the necessary linkages, at the person level, between decennial censuses, important survey and program evaluation data, and relevant administrative data—is foundational to future social research. These data would be linked in the Census Bureau’s secure data environment to provide researchers and policy analysts a robust, longitudinal national data infrastructure for the study of social welfare and the evaluation of public policy. Hout suggested that the workshop theme could well be: “If we build it, will they come?”
Responding to the question, Timothy Smeeding (School of Public Affairs, University of Wisconsin—Madison; chair, workshop steering committee) said that much policy relies on assessing longitudinal processes and long-run ef-
fects of programs, treatments, and life experiences. “We need to move beyond the one-off studies to systematically allow limited, orderly, and secure access to data, which we already have,” he said. Smeeding was followed by Amy O’Hara (Center for Administrative Records Research & Applications, U.S. Census Bureau; member of the workshop organizing committee), who talked about the infrastructure within the Census Bureau that can enable the kind of data linkages envisioned. Much of the linking methodology is in place, and the 2000 and 2010 censuses have been linked to each other and to some survey and administrative data. A big gap is the 1990 census. “We have built an infrastructure and have pulled in tax data, Social Security data, housing data, and data from other sources, . . . and we are trying to create the most robust linkages. We are now interested in opening that data infrastructure up in a controlled fashion . . . to enable more studies,” O’Hara said.
Smeeding reemphasized that the goal of the workshop is to provide “proof of usefulness,” moving beyond the core concept of social mobility to using the AOS for evidence-based policy making. Explaining the organization of the workshop, he said that the invited speakers represent a broad array of social science research that the workshop steering committee grouped into four categories: social and economic mobility; health, education and the life course; program evaluation and life experiences; and using the AOS cooperatively with other sources of data. The speakers were asked to discuss how the AOS, when fully actualized, could contribute to their own research and, more broadly, to their field of research.
THE AOS IN THE STUDY OF SOCIAL AND ECONOMIC MOBILITY
The study of social mobility speaks to issues of class formation, equal opportunity, and lifetime inequality. The speakers in this session addressed this area of social research with emphases on neighborhood effects and family structure.
David Grusky (Department of Sociology, Stanford University; member, standing committee) spoke about how the AOS would improve the country’s capacity to monitor and understand social mobility. Grusky said there are growing worries about a decline in social mobility in the United States, but because existing data cannot provide reliable information, everyone has been left with clever attempts to “make do.” He listed seven specific advantages the AOS would bring to the study of mobility: (1) improved measurement of occupational mobility; (2) multidimensional measurement of mobility; (3) better measurement of racial and ethnic heterogeneity; (4) an annual measurement protocol; (5) neighborhood heterogeneity (see the next presentation, by Nathaniel Hendren); (6) improved representation of family effects (see Laura Tach’s presentation); and (7) better measurement of the intergenerational reproduction of poverty.
Nathaniel Hendren (Department of Economics, Harvard University) discussed how social mobility might be rooted in neighborhood effects. His research (joint with several colleagues) aggregates data from U.S. income tax returns and tax information returns (such as W-2s) to study geographical variation in intergenerational mobility. He provided an example study involving 5 million families that moved among areas in the country: it showed that the earlier in a child’s life a family moves to a better neighborhood, the better that child does in adulthood. These results prompted Hendren and his colleagues to reexamine a 1990s-era social policy experiment program known as the “moving to opportunity experiment,” it provided families living in high-poverty housing projects with vouchers to move to lower-poverty neighborhoods. The initial evaluations of this experiment showed few effects on economic outcomes, but the reexamination—which focused on younger children—showed different results as the children reached adulthood: a higher percentage going to college and a 30 percent increase in earnings. Hendren said that an intergenerational panel as proposed by the AOS would be of enormous value for such research. He also identified some potential challenges, including legal constraints, privacy concerns, and logistical constraints.
Laura Tach (College of Human Ecology, Cornell University) addressed the topic of family structure and family demography in the context of social mobility. Three specific aspects of family demography—family formation, family disruptions, and family complexity—are important to study, and such work is difficult to do with administrative and survey data. She said that an AOS-type data structure might help to overcome many of the difficulties.
Tach described her research using the Fragile Families and Child Wellbeing Study, a prospective cohort study of children born in the late 1990s that tracks the children over time. A 15-year follow-up is now in the field. The study follows the child, the child’s mother, and the child’s father. She said that the chief limitations of the study are its restriction to an urban sample and its coverage for only the 1990s and forward. She indicated that AOS might have the potential to help with both limitations, as well as to help recover important information about unmarried fathers. She concluded: “A panel like the AOS, with its intergenerational linkages, is probably the only data source I can think of that
would allow us to be able to look at how growing family instability and complexity affects the processes of intergenerational mobility and the reproduction of inequality.”
THE AOS IN THE STUDY OF HEALTH, EDUCATION, AND THE LIFE COURSE
Speakers in this session addressed the early life effects of environmental factors, health conditions, poverty-related stressors, and education on people’s long-term health and opportunities.
Janet Currie (Woodrow Wilson School, Princeton University) spoke on early life experiences as a root of inequality. Currie’s main research focuses on two questions: Which environmental factors affect early life health? How does early life health affect later life outcomes? In pursuing these questions, Currie merges birth and death records from vital statistics with hospital discharge and emergency room visit records. She discussed the main advantages and disadvantages of each source, along with examples. Linking these data sources to census data would add considerable value, she said: “It is extremely helpful just to know where people were born.”
For an example, she cited forthcoming research showing that children who were born in counties that were subject to the U.S. Clean Air Act have higher employment and earnings and lower levels of disability as adults than children born in counties not subject to the act.1 In discussing barriers, Currie pointed out that most health data are “owned” by states, which vary in their openness to share data and in their technical capabilities to construct usable data structures. She said that it might be possible to build on existing state collaborations with the Agency for Healthcare Research and Quality through the Healthcare Cost and Utilization Project.
Ken Smith (Huntsman Cancer Institute, University of Utah), who works extensively with state data, focused his presentation on work with the Utah Population Database (UPDB). He said he thinks the UPDB includes the kind of breadth for Utah that the AOS is hoping to achieve nationwide. Smith described the Early Life Conditions, Survival, and Health Project, which is looking at how early life conditions affect later life health. One goal of this project is to try to identify at-risk populations and—by improving their understanding of health disparities that may arise early in life—to help guide interventions and surveillance strategies. Smith enumerated some of the administrative sources of data that have been brought together in the UPDB and noted that there is also a large ongoing record-linkage operation. “We are able to link not only parent to child, but we are able to link tenth cousin to tenth cousin. Any kind of any family connection that you can imagine—polygamy, consanguinity, all of the complexities that you would expect to see in any population—we are able to capture,” he emphasized. The goal is to establish family histories of health and medical conditions, some of which could have effects on social mobility, but also on individual health outcomes. Smith concluded by noting that the AOS as envisioned could help close some gaps in the UPDB, such as following people who migrate to different states, and working with administrative data from cancer registries in other states.
C. Cybele Raver (Vice Provost for Research and Faculty Affairs, New York University) described her research on the role of poverty-related stressors (such as scarcity, threat, and turbulence) for children’s difficulty in school. The work is based on a longitudinal follow-up of a randomized trial in 2003-2004 in very low-income neighborhoods in Chicago. In the study, teachers intervened in specified ways to help children regulate their emotions and their cognitive functions. Raver reported: “We saw significant benefit . . . and those interventions worked for young children. What was really challenging is that those benefits were quickly eroded as children transitioned into both low-quality schools and continued to live in very dangerous, unsupported, environmentally less stable, and less resourced environments.” She said that she sees environmental insult and environmental repair as fundamentally intergenerational, and thus could benefit from the type of intergenerational data structure in the proposed AOS, with stronger administrative, census-level, and population-level data. Raver concluded: “The extent to which [research on] long-range educational outcomes is a possibility from these kind of data is phenomenal . . . It would be great to be able to estimate two-generational effects.”
Susan Dynarski (School of Public Policy, University of Michigan) spoke on the linkage between education and economic mobility. She described several examples of her research on the effect of education policies on educational attainment, achievement, and adult well-being. Her work documents levels, trends, and inequality in educational outcomes by parental income and by race and gender. Dynarski uses data from both traditional surveys and administrative sources obtained through research partnerships with state education agencies. In cooperation with state agencies in Massachusetts and Michigan, she is building a longitudinal data system that covers students starting in 2003. Dynarski stated that grants from the Institute of Education Sciences is encouraging these types of partner-
___________________
1Adam Isen, Maya Rossin-Slater, and Reed Walker. (In press). Every breath you take—every dollar you’ll make: The long-term consequences of the Clean Air Act of 1970. Journal of Political Economy.
ships between academics and the state agencies that hold educational data, creating usable data systems. She added: “These state data systems are where educational research is currently being done—they are huge, they are comprehensive, they are cheap.” It is pretty important to tie these partnerships and data systems into the AOS vision for linking existing data and data systems, she said. Better data are needed to help understand the effects of educational interventions on adult well-being—the long-term effects. Important to this understanding is knowing parental background in a more detailed way to understand variability and the effects of policies.
Dynarski identified several barriers in this work. On postsecondary intergenerational mobility, a “big constraint is that there are no comprehensive national data on attendance, attainment, or degrees.” She said that variability among state agencies in interpreting federal laws, such as the Family Educational Rights and Privacy Act (FERPA), is also a barrier. Lastly, Dynarski said there is an organized constituency that opposes the linkage of student data to anything else, presumably for reasons of privacy: she warns that this opposition will be a barrier to including more educational data in the AOS infrastructure.
THE AOS IN PROGRAM EVALUATION AND LIFE EXPERIENCES
Speakers in this session discussed the issues involved in evidence-based program evaluation and the importance and difficulty in studying the impact of two different life-course connections: military service and criminal involvement.
Gordon Berlin (President, MDRC) talked about the work of his organization, a nonprofit, nonpartisan education and social policy research organization. MDRC performs evaluation studies of the effects of services for low-income families, typically using randomized controlled trials with mostly short-term (5 years or less) follow-up. The studies span the life cycle and many fields, including families and children, education (all levels), and the economic outcomes for low-wage workers and the hard-to-employ. To conduct these studies, MDRC uses primary data collected from study participants, state and federal administrative data, and data from program management information systems—which are usually linked by MDRC.
Berlin talked about the limitations of the current approach to program evaluation. State administrative data are hard to access and difficult to use. There is considerable variability across agencies in terms of regulations (which may limit access), consent requirements, interpretations of federal policies such as the Health Insurance Portability and Accountability Act (HIPAA), and data security requirements. State agencies have limited resources to assist. Similarly, federal data also have limitations for program evaluation purposes. Often the data are designed for limited types of research; some are “de-identified,” which makes them hard to link to other data; and there is often a substantial lag in their availability. Berlin said one of the potential benefits of the proposed intergenerational data in the AOS would be to greatly reduce the effort of collecting data so that MDRC could possibly examine longer-term impacts. For MDRC’s work, the intergenerational panel would need identified data that could be linked to a study sample, have a short lag for timely assessment of outcomes, and are consistent with the study’s consent requirements. Other potential uses of the AOS data would be to help in the study design, to use in predictive modeling, and to help generalize results from the study sample to a larger population.
John Laub (Department of Criminology and Criminal Justice, University of Maryland) and William Sabol (Westat) jointly discussed the potential of the AOS for research and program evaluation on crime, punishment, and human development. To illustrate the power of using robust, longitudinal data in criminology, Laub described the Glueck Project, which followed 1,000 juvenile males (500 delinquents and 500 non-delinquents) from 1940 to about 1963. In a 50-year update of the Glueck men’s lives,2 researchers found enormous variability in criminal activity among the men over the full life course. They concluded that trajectories of crime are influenced but not determined by prior childhood differences: desistance (from criminal activity) is a process and occurs even for those at highest risk for continued offending. Interventions may be potential turning points in the life course.
Sabol talked about the effects of growing up in an era of mass incarceration. Questions continue about the effects of this phenomenon on labor market outcomes, physical and mental health outcomes, family formation and dissolution, family outcomes, victimization risk, and persistent criminal involvement. Sabol noted a key question in the research: “What happens to life-course changes of people who have been through the system in the 1970s versus the 1990s? I think the AOS clearly provides opportunities for addressing that question.” Sabol said the major challenge for him is “how to bring into this mix of information on earnings, employment, and so forth some measure of criminal history.”
Alair MacLean (Department of Sociology, Washington State University) talked about her research exploring the life-course trajectories of veterans who served in
___________________
2John H. Laub and Robert J. Sampson. (2003). Shared Beginnings, Divergent Lives: Delinquent Boys to Age 70. Cambridge, MA: Harvard University Press.
the U.S. armed forces, focusing on the effects of military service and combat exposure on work and health. She looks at what happens to veterans immediately after they get out of the military, as their postservice careers unfold, and when they retire. MacLean examines factors, such as earnings, income, unemployment, health, and disability across different cohorts (such as World War II, Vietnam). “They come home to very different societies, so I look at how that might differ,” MacLean said. Addressing the data that she uses for research, she mentioned the Wisconsin Longitudinal Study, the Health and Retirement Study, the National Longitudinal Study of Youth, and the Educational Longitudinal Study. MacLean identified three limitations to these data: lack of information about family background, small sample sizes (rates of military service in the general population are quite low), and limited coverage of eras. Regarding the potential usefulness of the AOS, MacLean said: “I was very excited to read the proposal because the idea of being able to have these very large population-based samples that link people to their families would be extremely helpful for military research.” She believes that it would be important to link Department of Defense data to the AOS because it would enable studies of the long-term effectiveness of veteran’s benefits (such as health care and the GI bill), which are currently a large part of the federal budget.
THE AOS USED COOPERATIVELY WITH OTHER SOURCES OF DATA AND INFORMATION
In this session, speakers discussed examples of existing work being done to provide or link data for longitudinal research in social science and how these efforts might benefit from and contribute to the AOS.
Katherine Harris (Carolina Population Center, University of North Carolina at Chapel Hill) spoke about the National Longitudinal Study of Adolescent to Adult Health (Add Health). Add Health started with a nationally representative cohort of adolescents in grades 7 through 12 in 1995 and now follows them into adulthood. The study’s fifth wave is about to be fielded. Harris said the study has an integrative design to understand social, environmental, behavioral, biological, and genetic linkages to health across the life course. Add Health is primarily a health study with a focus on the role of social context. The researchers have linked to school records and military records; they also asked respondents to consent to linking to their Social Security Administration (SSA) records, and about 50 percent consented.
One example of how the AOS could enhance the research work being done through Add Health would be to confirm the linkages between the sampled children and their parents and to obtain more information about the parents. A second example is that it could be used to link to important administrative data that would provide annual income and unemployment data on the parents over time. This linkage would enable research on the effects of early life socioeconomic status (SES) that is now missing. Linkages to tax data would provide residential information on an annual basis that would be incredibly valuable for the study of environmental threats to health. With such data, the researchers could then model residential selection processes among parents and also among children after they leave home. Harris talked about the peer networks that the children have in adolescence. Linkage with the AOS would provide an opportunity to track these peer networks as the adolescents disperse later in life. All of this research would advance the goal of understanding how adolescent experience and exposures affect adult health.
David Johnson (Survey Research Center, University of Michigan; member, standing committee) discussed his work as codirector of the Panel Study of Income Dynamics (PSID). The PSID began in 1968 with 4,800 families, with oversampling of low-SES families; it was designed to evaluate causes and consequences of poverty. It has now been running for almost 50 years. Researchers follow the original families, their children, and the new families that have been formed among them. The PSID added immigrant families in 1997. Johnson said that the best feature of the PSID is that it is multigenerational, containing five generations of families: “We have the base for looking at mobility.” The PSID survey data are linked to administrative data, such as Medicare claims data, college data from the Integrated Postsecondary Education Data System, and data on assisted housing from the Department of Housing and Urban Development. The researchers now plan to link to data from the real estate Website Zillow and the SSA.
Johnson briefly described two examples of how the PSID and AOS could be used in concert with each other to evaluate long-term policy. He said that the biggest complaints researchers have with the PSID is that the sample is too small, and it is no longer representative of the U.S. population. The AOS would allow the PSID to efficiently supplement its data and obtain more recent information about respondents who dropped out of the survey and historical information about the new people in the PSID sample, such as those who married into the PSID families. He also suggested that the PSID could assist the AOS. The AOS intends to link 1990 census data to data from subsequent censuses at the person level. Johnson suggested that since PSID participants are closely tracked, it could be used to find the PSID participants in the 1990, 2000, and 2010 censuses, and
evaluate whether the AOS procedures matched those individuals correctly across censuses.
Martha Bailey (Department of Economics, University of Michigan) discussed the Longitudinal Intergenerational Family Electronic Micro-Database (LIFE-M). Just coming out of a pilot phase, LIFE-M is an administrative database that is going to use vital records (birth, marriage, death) as a basis for a long-term family network created with linkages of individuals from birth to death and across generations. The assembled data will further longitudinal analyses of large samples and subsamples of individuals and families over recent U.S. history. There are plans to link these vital records to census and other data. This database is going to span the late 19th century and most of the 20th century for a subset of states. Bailey said they would like to harness a variety of data relating to birth-family characteristics and to obtain economic and demographic outcomes from census data. The researchers are also interested in geographic information, specifically at the time of vital events. In discussing limitations, Bailey said that the coverage of vital records by state and period varies, and not every state has its records available online. Furthermore, if parents had children and then moved to a different state, it will be difficult to continue to link these individuals. In addition, the study will be limited by the periods that are covered. Lastly, she noted, clerical review turns out to be really important for linking, and such review is very expensive. The AOS has the potential to overcome or mitigate some of these limitations, Bailey said.
Jennifer Noyes (Institute for Research on Poverty, University of Wisconsin–Madison) discussed the collaboration between the Institute for Research on Poverty (IRP) and Wisconsin state agencies to inform policy and practice. Figure 1 graphically displays components of that collaboration. Through this collaboration, IRP has created a single record for each individual by matching and merging from the primary data sources, using identifying variables: these matches are complex and time-consuming to program. Noyes said the database is very flexible for different uses, such as measuring program participation, understanding the relationships between programs, and measuring the effect of policies. For example, one study addressed the reasons that program participants become “disconnected” from programs. Noyes described three key lessons from this work: (1) collaboration with data custodians has been and will continue to be essential; (2) involvement by an outside partner who does not “own” any of the programs or the associated data can provide objectivity to the entire process; and (3) infrastructure requires sustained commitment by all parties and significant funding. In thinking about linkage with the AOS, Noyes said that it could help fill in the gaps in the IRP’s information about income, education, and occu-
pation. It could help track people outside of Wisconsin and could help the IRP improve its matches.
A RELATED ACTIVITY IN CONGRESS
Bruce Meyer (School of Public Policy, University of Chicago) briefly described the Evidence-Based Policymaking Commission, recently created by Congress. Meyer, a commission member, said the commission, which is to report to Congress in 15 months, will play a critical role in facilitating policy evaluation and research. The commission will have 15 members, 3 each appointed by the President and by the leaders of both houses of Congress. The commission’s mandate is first to conduct a comprehensive study of the inventory, infrastructure, and data security of administrative and survey data and then to make recommendations on an optimal arrangement for using these data together. The commission is to make recommendations on the incorporation of outcome measures, randomized controlled trials, and impact analysis in program design. It might recommend the creation of a data clearinghouse. Meyer noted some opportunities emerging from the commission’s work: it could help show the value of administrative data; push agencies to share data while giving them cover; build a consensus for evidence-based legislation; and reassure the public that data are secure. “If there is a statement by a congressionally appointed commission that sharing administrative data is good for the country, good for policy making, and good for research, then I think it does help,” Meyer said.
STATISTICAL CHALLENGES
Stephen Fienberg (Department of Statistics, Carnegie Mellon University; member, standing committee) cautioned that enhancements in data matching and estimation methodology will be needed to fulfill the promise of the AOS. “Everyone has been talking about matching as if it is a simple thing to do and you can do exact matching. It just ain’t so,” he said. The most common approach to data linkage with multiple files today, and the one that is used at the Census Bureau, is to match each file pair-wise with the Social Security Numident file (containing names and information from applications for Social Security cards). Fienberg recommends an approach he calls “multiple record linkage” in which all files are analyzed together in a systematic way to make efficient use of all information across all files and avoid the transitivity problems that can arise from sequential steps. He said that, once linked, users will have to consider how to incorporate uncertainty about matching into the statistical estimates based on those matches. Since there is error in matching no matter how well it is done statistically, both bias and uncertainty need to be measured. “We need to be able to ‘propagate uncertainty’ from the record linkage as an added component of uncertainty into the statistical estimation based on linked files.” Fienberg noted that research into matching and estimation methodology is part of the agenda before the AOS standing committee.
NEXT STEPS
Three members of the workshop steering committee answered the questions: What did we hear? What are the big questions? What else do we need to do?
Hout spoke first: “The big lesson from today, and I was really heartened, is that we are not inventing the wheel here. There is a lot of experience out there that we can start to draw on and collect and pool together. The American Opportunity Study is the next big step.” Hout noted that the need to build relationships with state-level agencies was repeated by several presenters. “There is a quid pro quo in dealing with administrative data. We have to offer back to those who have the data the products and insight they could not otherwise get. We are not just bringing together data or assembling data but enhancing and passing back better, more usable information to those who have provided data to us.”
Smeeding said he has concluded that this meeting provided extensive proof of usefulness. “We heard many times today that ‘longitudinal’ is important, and the AOS can and will lead to big improvement in current longitudinal surveys and vice versa. A second point, he said, is that there is a lot of work going on using state data on health and education. The AOS will not get comparable data from all the states, but these state-based studies are all asking the same questions: How do kids do as adults as a result of their SES and experiences in early life? Smeeding said that there is considerable value added that the AOS can give back to the states on this subject once the state data are aligned to longer-term national data on education, health, social, and economic outcomes that will be available in the AOS.
Bhashkar Mazumder (Federal Reserve Bank of Chicago) offered several points from the day’s speakers. There is a strong need for both administrative and survey data, he noted. Even the presenters whose primary work was with administrative data talked about how results could be enhanced by linking them to survey data. He commented that several people touched on the issue of data access and whether the Federal Statistical Research Data Centers might be a good vehicle to provide that access. He noted that there was a lot of discussion about the importance of linking various survey and other data to vital statistics from state agencies. He said he agrees with Currie’s comment that the AOS work should start with low-hanging fruit—the
most populous states or states that make their vital statistics reasonably easy to access.
Mazumder also summarized several issues that might be important for the new congressional commission to address. One is the in-house expertise, or lack thereof, of federal and state agencies to develop memoranda of understanding to make effective state-federal partnerships. A second issue for consideration is the apparent variability in interpretation of federal regulations, such as FERPA and HIPAA. Some federal help will be needed on this issue. Mazumder also noted the timeliness issue: 4-5 years may be fine for a research project, but program evaluation requires a more rapid turnaround. “We need to remember the need for timeliness in the procedures to access data for these purposes,” he concluded.
DISCLAIMER: This Proceedings of a Workshop—in Brief was prepared by Carol House, on the staff of the Committee on National Statistics, as a factual summary of what occurred at the meeting. The statements made are those of the author or individual meeting participants and do not necessarily represent the views of all meeting participants; the planning committee; or the National Academies of Sciences, Engineering, and Medicine.
REVIEWERS: To ensure that it meets institutional standards for quality and objectivity, this Proceedings of a Workshop—in Brief was reviewed by Thomas A. Kecskemethy, Office of Executive Director, American Academy of Political and Social Sciences, Philadelphia, PA; Jared S. Murray, Department of Statistics, Carnegie Mellon University; and C. Matthew Snipp, Department of Sociology, Center for Comparative Studies in Race and Ethnicity, and Stanford Secure Data Center, Stanford University. Patricia Morison, National Academies of Sciences, Engineering and Medicine, served as review coordinator.
SPONSOR: This workshop was supported by the Carnegie Corporation of New York.
For additional information regarding the meeting, visit
www.sites.nationalacademies.org/dbasse/cnstat/dbasse_172506.
Suggested citation: National Academies of Sciences, Engineering, and Medicine. 2016. Using Linked Census, Survey, and Administrative Data to Assess the Longer-Term Effects of Policy: Proceedings of a Workshop—in Brief. Washington, DC: The National Academies Press. doi: 10.17226/23583.
Division of Behavioral and Social Sciences and Education
Copyright 2016 by the National Academy of Sciences. All rights reserved.
