7

Alternative Approaches to Criminal Justice Measurement

John Laub (University of Maryland) began the workshop session on looking at alternative approaches by summarizing the strategies proposed earlier in the workshop for collecting information on the health effects of criminal justice involvement. Those strategies included linking criminal justice involvement to ongoing general social surveys, gaining more knowledge of the needs of the incarcerated population through criminal justice and inmate surveys, and using administrative records to make linkages to past incarceration experience. He then turned to the designated speakers to discuss their experiences with the Fragile Families Study, community-based surveys, the National Longitudinal Survey of Youth, the Panel Survey of Income Dynamics, and the National Crime Victimization Survey.

MEASURING INCARCERATION IN THE FRAGILE FAMILIES STUDY

Amanda Geller (New York University) described measures of incarceration and criminal justice involvement collected in the Fragile Families and Child Well-Being studies and what has been learned from a pilot linkage project with administrative data provided by the New York State Division of Criminal Justice Services.¹

___________________

¹ Administrative data were provided by the New York State Division of Criminal Justice Services (DCJS) in the interest of information exchange. The opinions, findings, and conclusions are not necessarily those of DCJS. Neither New York State nor DCJS assumes liability for these findings or the use thereof.

Fragile Families is a birth cohort study of 4,898 families with children born in 20 large cities (populations of 200,000+) between 1998 and 2000. The study was originally designed to primarily address four questions of great interest to researchers and policy makers: (1) What are the conditions and capabilities of unmarried parents, especially fathers? (2) What is the nature of the relationships between unmarried parents? (3) How do children born into these families fare? and (4) How do policies and environmental conditions affect families and children? The study has since expanded to look at broader social disadvantages faced by children (e.g., paternal incarceration) and is currently in its sixth wave of data collection, surveying children 15 years of age. As nonmarital births are systematically oversampled in the study, about 75 percent of the children in the study were born to unmarried parents, which resulted in a sample dominated by children belonging to minority racial and ethnic groups of lower socioeconomic status and high rates of paternal incarceration (about 50 percent). The core study consists of interviews with both mothers and fathers at birth and again when children are 1, 3, 5, and 9 years old. At the time of writing, the year 15 survey focuses on the children and their primary caregivers, and does not interview fathers.

As a population-based survey rather than a survey of inmates, Fragile Families collects information on incarcerated and formerly incarcerated parents, as well as on parents without histories of incarceration experience. The systematic oversample of unmarried parents in the Fragile Families sampling plan produces a sample that is highly socioeconomically disadvantaged, with high rates of incarceration experience, particularly among the fathers. This makes the study of parental incarceration more feasible than many other samples. Fragile Families also provides ways to compare fathers who have an incarceration history to those who do not, but who nonetheless experience very similar socioeconomic disadvantages. Incarceration can be reported in self-reports or in partner reports, providing some information about incarcerated parents lost from the survey due to attrition. Incarcerated fathers are often non-custodial parents and not the primary caregivers, which makes it hard to identify father–child relationships in other administrative datasets (such as incarceration surveys). Fragile Families provides an opportunity to link currently and formerly incarcerated fathers to their children.

Geller noted that both parents are asked questions about their own and their partner’s involvement with the criminal justice system. Generally, fathers are asked more detailed questions than mothers. They are first asked if they have been charged with a crime in a particular time period; if so, they are asked if they were convicted of a crime; and if so, they are asked if they have spent time in prison or jail. Mothers, on the other hand, are simply asked if their partner has ever been incarcerated. Mothers’ questions

are therefore more inclusive than fathers’: If a father was charged with a crime, held in central booking or jail for a period of time, but charges were ultimately dismissed, mothers would have an opportunity to report the incident, but fathers would not: they would report the charge, but with no conviction, they would not be asked about incarceration. In addition there are questions that gauge criminal justice involvement indirectly, such as asking why the mother and father are separated, or why one parent has been unable to search for employment, providing incarceration as an answer choice. Geller said that her analysis shows that the majority of incarceration experiences are identified through partner reports rather than self-reports, since response rates are higher for mothers than fathers (Geller et al., 2016). She noted that most incarceration analyses based on Fragile Families assume that a self-report or partner-report of an incarceration incident is correct, since research shows that potentially stigmatizing behavior is unlikely to be overreported (see Chapter 5 on how social desirability influences answers to sensitive questions).

Geller and colleagues (2016) explored the possibility of improving Fragile Families data quality by supplementing it with administrative records on criminal histories. Official criminal history information may identify and provide more detail on incarceration experiences that are unreported or underreported in survey data. Conversely, a data linkage exercise such as this can also reveal to what extent official records miss criminal history experiences that are reported in surveys. As part of the data-matching pilot study, information on 333 fathers from the Fragile Families New York City sample was submitted to New York State’s Department of Criminal Justice Services after working to design a data collection protocol that adequately addresses human subject protection and privacy protection. Records were matched on the basis of name, Social Security number, and date of birth. Analysis of the data showed a 23 percent increase in fathers identified as incarcerated, and a 30 percent increase in fathers identified as criminal justice involved (Geller et al., 2016).

Geller stressed that findings from the data linkage exercise underscore the challenges in identifying criminal justice involvement in surveys and the extent of criminal histories that do not show up in administrative records. For example, criminal history records may be sealed under a variety of circumstances, making it difficult to collect full information on criminal histories. The exercise also makes clear that administrative records are not a panacea for underreporting in surveys because of their aforementioned shortcomings and access restrictions.

Geller provided directions for future research in the area of survey methodology and data linkage, drawing on experience from the data matching pilot:

1. Collecting administrative data prospectively alongside survey waves could make matching more manageable.
   • Respondents could provide consent for comprehensive administrative record searches during the survey itself, so that human subjects concerns are addressed prospectively.
   • Survey questions could be used to identify states to search records, increasing the probability that administrative records fully reflect respondents’ criminal histories.
   • Collecting data at every wave can help ascertain timing.
   • This prospective collection still would not, however, include sealed records in states that cannot release them.
2. Details provided in administrative data raise questions about how respondents might interpret survey questions, which may be addressed through testing or qualitative research.
   • Do respondents count “convictions after plea bargains” as convictions?
   • How about adjournments in contemplation of dismissal and pleas to violations (e.g., disorderly conduct)?
   • Do respondents count pretrial detention as incarceration?
3. Future surveys might include cognitive tests in which respondents randomly receive alternate question wordings, to see if more explicit explanation leads to higher reporting rates.

COMMUNITY-BASED SURVEYS

John Boyle (ICF International) drew on his experience conducting more than 3,000 surveys to discuss how household surveys can serve as a vehicle for collecting information on such topics as incarceration experience and criminal justice involvement.

Boyle suggested that a cross-sectional survey may not be the best way to measure the variables that are of interest: instead, workshop participants might consider an age cohort study in which individuals are followed over time. He cited the example of the Longitudinal Harlem Adolescent Health Study, a community-based sociomedical study focused on the health problems of teenagers. It was a community-based study fielded in the late 1960s in which participants aged 12-17 were selected from one in six households in the central Harlem health district. The study participants were followed for 25 years, with their information collected in five waves. A general household survey may not have sufficient sample size to answer questions on the health

effects of incarceration, but a community-based study such as this one, with its focus on a high-risk community, will have enough study participants who have gone through an incarceration experience or had some other level of contact with the criminal justice system to serve this purpose.

The second advantage of an age cohort study, Boyle said, is twofold; it controls for age, and it also includes individuals living under the same conditions who did not have such an experience, thereby facilitating the definition and determination of counterfactual cases. Such an approach requires careful selection of households so that study participants remain engaged with the study for a long period of time. A community-based cohort study allows for logistical support to collect biological specimens because the location is fixed, and interviewers are going back to the same household and location in every wave. Although longitudinal surveys are excellent for studying health outcomes of criminal justice involvement, Boyle acknowledged that they are costly and require enormous amounts of time.

Boyle said that the next best alternative is a retrospective health study for which the sampling frame is available and the intervention has already taken place. He described the design of a multi-community-based retrospective cohort study in which choosing schools was the first stage of sample selection, followed by selection of particular grades. The approach would be to select schools at a given time, identify a cohort of students who attended the selected schools, and then locate and collect information on the selected cohort. One of the challenges would be proactively identifying prospective study participants, as administrative records may be sealed for various reasons.

Boyle noted the example of the retrospective twin registry study. The Vietnam-era twin registry was a resource of the U.S. Department of Veteran Affairs, composed of 7,369 middle-aged male-male twin pairs, both of whom served in the military during the time of the Vietnam conflict (1965-1975). It was used for genetic epidemiologic studies of mental and physical health conditions. Boyle urged workshop participants to consider twin registries as a vehicle to collect information on criminal justice involvement and said he thinks that they are a great resource for studying counterfactual cases. He concluded his talk by stressing the importance of reducing/minimizing nonresponse bias, cognitive testing of criminal justice questions, and framing the questions in a double-anchored fashion to get accurate responses.

THE NATIONAL LONGITUDINAL SURVEY OF YOUTH

Elizabeth Cooksey (Ohio State University) discussed measures of criminal justice involvement collected on the 1979 National Longitudinal Survey of Youth (NLSY79), a nationally representative sample of 12,686 young

men and women who were aged 14-21 when they were first surveyed in 1979. They were interviewed annually through 1994 and are currently interviewed on a biennial basis. The response rate in the latest round of the survey stands at 78 percent.

Cooksey said the NLSY79 is focused primarily on labor market involvement, but the instrument does have a few criminal justice and health questions. Variables on criminal justice include the earliest arrest date, total number of arrests, earliest incarceration date, total number of incarcerations, age at first incarceration, length of first incarceration, length of longest incarceration, total months incarcerated and current incarceration. The study does attempt to locate and contact survey respondents who are serving time. She suggested that three subsamples of the survey—a supplemental sample designed to oversample minority groups; the NLSY79 child sample consisting of children born to female respondents; and the NLSY79 young adults sample consisting of these same children aged 15 and older—would be of primary interest to the workshop participants. The first sample is useful for understanding the prevalence of criminal justice involvement and health outcomes in minority groups, and the child and young adult samples can provide information on collateral consequences of parental incarceration. Cooksey told the workshop participants that the NLSY97 included about 750 individuals who reported being incarcerated at some point in their lives. The NLSY79 and NLSY97 have an extended health module for respondents of a certain age that contains information on mental and physical health, health conditions, and health care coverage. Beginning in 2006, NLSY79 respondents aged 48 and 49 were given a battery of exercises designed to capture cognitive capabilities.

THE PANEL STUDY OF INCOME DYNAMICS

David Johnson (University of Michigan) focused on health data collected on the Panel Survey of Income Dynamics (PSID), as well as linkage opportunities. The PSID is a household longitudinal survey that started in 1968 with 5,000 families. The unique feature of the survey is that it covers households that developed from the original households—the family units started by the children of the original 5,000 families. Consequently, there are data available on approximately 1,200 three-generation families, making it the longest running household longitudinal survey in the world. Collecting the survey’s detailed health data and biological specimens requires consent from survey respondents and is an expensive process.

The PSID consists of a core interview, which collects self-reported health, health care, and economic data, as well as rotating supplementary modules, focusing on other social and demographic content. The survey gathered information on household members’ jail or prison experiences

from 1985 to 1992. PSID also conducted a special Childhood Retrospective Circumstance Survey (PSID-CRCS) in 2014 funded by the National Institute on Aging in which they asked participants aged 19 and older about incarceration experience before age 26. Of the over 8,000 responses received, Johnson said about 188 adults in the sample reported that they had been in jail at least once before age 26 (Institute for Social Research, n.d.-b). In another PSID supplement conducted in 2007 called the Transition to Adulthood Study, 254 of the approximately 1,260 18-year-olds surveyed reported that they had been to jail (Institute for Social Research, n.d.-a). Johnson said that, while the data can prove insightful, the sample sizes for the PSID core and supplementary modules often provide challenges when trying to generate valid estimates of health outcomes. Johnson explained that PSID data can be linked to other sources, such as Medicare claims, Housing and Urban Development records, census data, and real estate data. It would be possible to link PSID data to criminal justice and inmate surveys; the challenge is getting consent from survey respondents regarding linking their data to other sources.

THE NATIONAL CRIME VICTIMIZATION SURVEY

Jim Lynch (University of Maryland), former head of the Bureau of Justice Statistics (BJS), discussed the challenges in capturing a relatively rare phenomenon in household surveys. Referring to the surveys described in this workshop session, he said that most of them do not have a large enough sample size to generate a worthwhile subsample of people who are incarcerated. The traditional approach has been to study these types of phenomena through a small longitudinal or repeated cross-section survey. Often, however, the lack of relevant responses results in expanding the definition of dependent variable (from “did you” statements to “have you ever” statements) in an effort to obtain a sufficient number of observations to conduct an empirical analysis. Consequently, Lynch said, the National Crime Victimization Survey (NCVS) is a useful tool to study criminal justice involvement because of its large sample size of 90,000 households. Even if the single-year sample for incarceration is small, larger samples can be achieved by combining sample sizes across years. The disadvantage of the NCVS is that it does not collect data on health outcomes or health care access or utilization.

Lynch pointed out, however, that there are attributes of the NCVS that make it a good vehicle for fielding questions on criminal justice involvement. The NCVS has a rotating panel design in which a household is interviewed twice in 6 months, making certain to capture every member of the household between those two interviews (or using proxy interviews if that is the only option). Because the survey asks about people who have

entered or exited the household, one can sometimes crudely deduce which households have been touched by incarceration (although “moved to an institution” could mean a transition to a nursing home or other non-penal setting). The NCVS does offer the opportunity for externally funded supplements, which could be a potential vehicle for more specific criminal justice involvement questions.

Lynch next described the administrative data linkage initiatives undertaken by BJS and the Census Bureau that can improve the amount of information available on criminal justice involvement and health outcomes. The Census Bureau started the Center for Administrative Record Research and Application (CARRA) to harness the potential of administrative data from federal, state, and third-party providers through record linkage and statistical matching while ensuring confidentiality. He said that CARRA is a terrific model because participating data providers can use linked datasets and analyze data linkages from sources in the criminal justice system.

BJS’s correctional unit has undertaken efforts to link data from the National Prisoner Statistics Program and the National Corrections Reporting Program to other data sources, including those with health data. He relayed the experience of BJS in using Social Security numbers as the identifier variable: they are generally a good linking variable for the nonprison population but not for those who have a criminal history record. With that limitation, the agency has been successful in matching certain prison and corrections data cases and observations to administrative data on health. Other strategies tested for multiple years when the Social Security numbers were not available had relatively high success rates. Another potential source of data that allows investigation and differentiation of correctional experience, as well as an expansion into health data, is the National Inmate Survey. BJS is currently in the process of securing consent from inmates so that their data can be linked to other sources: he noted that some inmates have refused to give consent.

Lynch stressed that linking survey data with administrative data is fraught with challenges as there are errors in both data sources, and definitional differences can hamper connecting similar pieces of information. Moreover, he said, accessing administrative data and securing permission to match them with other sources requires a good relationship with the data provider, be it federal, state, or some other third party. Such initiatives are undertaken on a case-by-case basis, and the full potential of administrative data has not been realized.

Lynch advised the workshop participants to strongly consider investigating the linking opportunities that are available in the Federal Bureau of Investigation’s Computerized Criminal History (CCH) data system. The CCH data have two major strengths—information on arrests and information on corrections. The utility of criminal history data has not been

explored for the purposes of studying health effects of incarceration; it has been mostly used to study recidivism. Lynch concluded his presentation by stressing the importance of building a statistical infrastructure of record linkage that is easily accessible to researchers.

DISCUSSION

To begin the discussion, Laub invited comments and questions from the presenters in earlier sessions. Candace Kruttschnitt (University of Toronto) said she was struck by the lack of coverage of incarcerated populations in general and social surveys. Marcie Cynamon (National Center for Health Statistics) said that national household health surveys could cover institutionalized populations if their mandates were modified and their budgets considerably expanded in order to meet the objective of understanding health outcomes in a relatively rare and small population. She also noted that the population is very heterogeneous, which will require a larger than normal sample size to conduct empirical analysis for subgroups of interest.

Allen Beck (Bureau of Justice Statistics) informed the workshop participants that the health data captured on inmate surveys conducted by the agency are collected through self-reporting. The data cover information on inmates’ health conditions prior to their being incarcerated and the medical conditions they developed while in prison. Beck noted that mortality rates of prison inmates are lower than comparable general non-institutionalized populations (except for liver mortality and sepsis mortality), which he said can be attributed to access to health care services in prisons.

Cynamon raised the issue of the reference period being restricted to only 1 year: limiting a household survey question to events that transpired in the previous year ignores many people who had contact with the criminal justice system early in their lives or at any point before the past year and the effects of that contact. Laub responded by saying that restricting the reference period to 1 year provides an anchor to the respondent that helps in recall, but there is the danger of telescoping, bringing forward the events that happened much earlier. On this point, Anjani Chandra (National Center for Health Statistics) talked about the purpose and focus of surveys: some, like the National Survey of Family Growth, are aimed at producing annual aggregate statistics, and therefore a reference period of 1 year is justified. Life trajectories can be deduced from lifetime experience questions that ask respondents about when they first experienced a particular event. Boyle responded to the point about life trajectories by pointing out that there had been no mention of covariates associated with traumatic experiences, such as resilience and coping. He said this implies that adding a couple of criminal justice questions on a national health survey will not provide that type of information.

Johnson suggested that the American Community Survey (ACS) could be another data source on the health outcomes of the incarcerated population because the instrument contains six questions on disability and is administered to institutionalized populations. There is precedent for using the sampling frame of the ACS for a follow-on survey.

Emily Wang (Yale University) said she surmises that household surveys set national priorities from the standpoint of government and the general public. The reason that incarcerated populations are not on their radar is due to a lack of understanding about the collateral consequences of incarceration among the general public. She pushed for inclusion of measures on criminal justice involvement in national health surveys as the value of such information can further the understanding of the effects of incarceration and criminal justice contact on individuals, families, and communities.