Review of Four CARA Programs and Preparing for Future Evaluations (2023)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Review of Four CARA Programs and Preparing for Future Evaluations Kenneth B. Wells, Hortensia Amaro, M. Kelly McHugh, Kathleen Stratton, Editors Committee on the Review of Specific Programs in the Comprehensive Addiction and Recovery Act Board on Population Health and Public Health Practice Health and Medicine Division A Consensus Study Report of PREPUBLICATION COPY: UNCORRECTED PROOFS 

NATIONAL ACADEMIES PRESS 500 Fifth Street, NW, Washington, DC 20001 This activity was supported by a contract between the National Academy of Sciences and the Department of Health and Human Services. Any opinions, findings, conclusions, or recommendations expressed in this publication do not necessarily reflect the views of any organization or agency that provided support for the project. International Standard Book Number-13: 978-0-309-XXXXX-X International Standard Book Number-10: 0-309-XXXXX-X Digital Object Identifier: https://doi.org/10.17226/26831 This publication is available from the National Academies Press, 500 Fifth Street, NW, Keck 360, Washington, DC 20001; (800) 624-6242 or (202) 334-3313; http://www.nap.edu. Copyright 2023 by the National Academy of Sciences. National Academies of Sciences, Engineering, and Medicine and National Academies Press and the graphical logos for each are all trademarks of the National Academy of Sciences. All rights reserved. Printed in the United States of America. Suggested citation: National Academies of Sciences, Engineering, and Medicine. 2023. Review of Four CARA Programs and Guidance for Future Evaluations. Washington, DC: The National Academies Press. https://doi.org/10.17226/26831. PREPUBLICATION COPY: UNCORRECTED PROOFS 

The National Academy of Sciences was established in 1863 by an Act of Congress, signed by President Lincoln, as a private, nongovernmental institution to advise the nation on issues related to science and technology. Members are elected by their peers for outstanding contributions to research. Dr. Marcia McNutt is president. The National Academy of Engineering was established in 1964 under the charter of the National Academy of Sciences to bring the practices of engineering to advising the nation. Members are elected by their peers for extraordinary contributions to engineering. Dr. John L. Anderson is president. The National Academy of Medicine (formerly the Institute of Medicine) was established in 1970 under the charter of the National Academy of Sciences to advise the nation on medical and health issues. Members are elected by their peers for distinguished contributions to medicine and health. Dr. Victor J. Dzau is president. The three Academies work together as the National Academies of Sciences, Engineering, and Medicine to provide independent, objective analysis and advice to the nation and conduct other activities to solve complex problems and inform public policy decisions. The National Academies also encourage education and research, recognize outstanding contributions to knowledge, and increase public understanding in matters of science, engineering, and medicine. Learn more about the National Academies of Sciences, Engineering, and Medicine at www.nationalacademies.org. PREPUBLICATION COPY: UNCORRECTED PROOFS 

Consensus Study Reports published by the National Academies of Sciences, Engineering, and Medicine document the evidence-based consensus on the study’s statement of task by an authoring committee of experts. Reports typically include findings, conclusions, and recommendations based on information gathered by the committee and the committee’s deliberations. Each report has been subjected to a rigorous and independent peer-review process and it represents the position of the National Academies on the statement of task. Proceedings published by the National Academies of Sciences, Engineering, and Medicine chronicle the presentations and discussions at a workshop, symposium, or other event convened by the National Academies. The statements and opinions contained in proceedings are those of the participants and are not endorsed by other participants, the planning committee, or the National Academies. Rapid Expert Consultations published by the National Academies of Sciences, Engineering, and Medicine are authored by subject-matter experts on narrowly focused topics that can be supported by a body of evidence. The discussions contained in rapid expert consultations are considered those of the authors and do not contain policy recommendations. Rapid expert consultations are reviewed by the institution before release. For information about other products and activities of the National Academies, please visit www.nationalacademies.org/about/whatwedo. PREPUBLICATION COPY: UNCORRECTED PROOFS 

COMMITTEE ON THE REVIEW OF SPECIFIC PROGRAMS IN THE COMPREHENSIVE ADDICTION AND RECOVERY ACT KENNETH B. WELLS (Chair), Director, Center for Health Services and Society, Semel Institute, University of California, Los Angeles HORTENSIA AMARO (Vice Chair), Distinguished University Professor and Senior Scholar on Community Health, Florida International University GINA BRYAN, Director and Clinical Professor, Post Graduate Psych-Mental Health Program, University of Wisconsin–Madison KAREN CROPSEY, Conaster Turner Endowed Professor of Psychiatry, The University of Alabama at Birmingham MARCELA HORVITZ-LENNON, Senior Physician Scientist, RAND Corporation SANDEEP KAPOOR, Assistant Professor, Medicine, Emergency Medicine, and Science Education, Donald and Barbara Zucker School of Medicine at Hofstra/Northwell RAYMOND C. LOVE, Professor and Director, Mental Health Program, School of Pharmacy, University of Maryland ROSALIE PACULA, Professor and Elizabeth Garrett Chair in Health Policy, Economics, and Law in the Sol Price School of Public Policy, University of Southern California JOSÉ A. PAGÁN, Chair and Professor, Department of Public Health Policy and Management, New York University School of Global Public Health SHARON REIF, Professor, Heller School for Social Policy and Management, Brandeis University GERY RYAN, Professor, Health System Science, Kaiser Permanente Bernard J. Tyson School of Medicine Study Staff KATHLEEN STRATTON, Study Director M. KELLY MCHUGH, Associate Program Officer CRYSTI PARK, Administrative Assistant MISRAK DABI, Senior Financial Business Partner ROSE MARIE MARTINEZ, Senior Board Director NAM Fellow RUCHI FITZGERALD, Service Chief, Inpatient Addiction Medicine at PCC Community Wellness PREPUBLICATION COPY: UNCORRECTED PROOFS v 

PREPUBLICATION COPY: UNCORRECTED PROOFS vi 

ACRONYMS AND ABBREVIATIONS BCOR Building Communities of Recovery BRSS TACS Bringing Recovery Supports to Scale Technical Assistance Center Strategy CAPTA Child Abuse Prevention Treatment Act CARA Comprehensive Addiction and Recovery Act CBA cost-benefit analysis CDC Centers for Disease Control and Prevention CEA cost-effective analysis CFIR Consolidated Framework for Implementation Research CSAP Center for Substance Abuse Prevention CSAT Center for Substance Abuse Treatment DSP-MRT Division of State Programs Management Reporting Tool EMS Emergency Medical Services EPIS Exploration, Preparation, Implementation, Sustainment FDA U.S. Food and Drug Administration FOA Funding Opportunity Announcement FQHC federally qualified health center FR first responder FR-CARA First Responder Training FY fiscal year PREPUBLICATION COPY: UNCORRECTED PROOFS vii 

GAO U.S. Government Accountability Office GPRA Government Performance and Results Act HHS U.S. Department of Health and Human Services MOUD medication for opioid use disorder OD Treatment Access Improving Access to Overdose Treatment OIG Office of Inspector General OMB Office of Management and Budget ORH Ohio Recovery Housing ORR Overdose Risk Registry OUD opioid use disorder PAF Public Access File PPW pregnant and postpartum women PPW-PLT State Pilot Grant Program for Treatment for Pregnant and Postpartum Women QRT Quick Response Team RCO recovery community organization RSS recovery support services SAMHSA Substance Abuse and Mental Health Services Administration SBIRT Screening, Brief Intervention, and Referral to Treatment SSA State Agency for Substance Abuse SUD substance use disorder PREPUBLICATION COPY: UNCORRECTED PROOFS viii 

TA technical assistance PREPUBLICATION COPY: UNCORRECTED PROOFS ix 

PREPUBLICATION COPY: UNCORRECTED PROOFS x 

REVIEWERS This Consensus Study Report was reviewed in draft form by individuals chosen for their diverse perspectives and technical expertise. The purpose of this independent review is to provide candid and critical comments that will assist the National Academies of Sciences, Engineering, and Medicine in making each published report as sound as possible and to ensure that it meets the institutional standards for quality, objectivity, evidence, and responsiveness to the study charge. The review comments and draft manuscript remain confidential to protect the integrity of the deliberative process. We thank the following individuals for their review of this report: MOLLY BROWN, DePaul University H. WESTLEY CLARK, Santa Clara University (retired) CHIN HWA (GINA) DAHLEM, University of Michigan School of Nursing BETHANY DiPAULA, University of Maryland PETER D. FRIEDMANN, Baystate Health KIRSTEN JOHNSON, City of Milwaukee Health Department LAWRENCE A. PALINKAS, University of Southern California CORI SHEEDY, Abt Associates JOSH TRENT, Leavitt Partners Although the reviewers listed above provided many constructive comments and suggestions, they were not asked to endorse the conclusions or recommendations of this report nor did they see the final draft before its release. The review of this report was overseen by JACK C. EBELER, member of NASEM Health and Medicine Division committee, and JOSIAH (JODY) D. RICH, Brown University. They were responsible for making certain that an independent examination of this report was carried out in accordance with the standards of the National Academies and that all review comments were carefully considered. Responsibility for the final content rests entirely with the authoring committee and the National Academies. PREPUBLICATION COPY: UNCORRECTED PROOFS xi 

PREPUBLICATION COPY: UNCORRECTED PROOFS xii 

Acknowledgments The committee of this study thanks both the Substance Abuse and Mental Health Services Administration and the Comprehensive Addiction and Recovery Act grantees for their work and for providing information to the National Academies for this report series. PREPUBLICATION COPY: UNCORRECTED PROOFS xiii 

PREPUBLICATION COPY: UNCORRECTED PROOFS xiv 

Contents SUMMARY S-1 1 INTRODUCTION 1-1 Comprehensive Addiction and Recovery Act, 1-1 The Four Programs, 1-5 Report Series Statement of Task, 1-13 Interpretation of the Statement of Task, 1-14 Organization of the Report, 1-17 2 METHODS AND APPROACH 2-1 Assessing Effectiveness: Approach, 2-1 Assessing CARA Programs: Streams of information, 2-4 Future Evaluation Mandates: Approach, 2-14 3 BCOR Findings 3-1 Program Purpose, 3-1 Information Sources and Limitations, 3-2 Findings: Grantee Activities, 3-3 Findings: Partners and Network, 3-10 Findings: Structural and Environmental Change, 3-13 Findings: Individual-Level Outcomes, 3-16 Findings: Community-Level Outcomes, 3-19 4 PPW-PLT Findings 4-1 Program Purpose, 4-1 Information Sources and Limitations, 4-3 Findings: Grantee Activities, 4-3 Findings: Partners and Network, 4-10 Findings: Structural and Environmental Change, 4-11 Findings: Individual-Level Outcomes, 4-15 Findings: Community-Level Outcomes, 4-17 5 OD Treatment Access Findings 5-1 Program Purpose, 5-1 Information Sources and Limitations, 5-2 Accomplishments: Grantee and Partner Activities, 5-3 Accomplishments: Structural and Environmental Change, 5-5 Barriers to Project Activities and Structural Change, 5-7 Findings: Individual-Level Outcomes, 5-8 Findings: Community-Level Outcomes, 5-8 Summary, 5-9 PREPUBLICATION COPY: UNCORRECTED PROOFS xv 

6 FR-CARA Findings 6-1 Program Purpose, 6-1 Information Sources and Limitations, 6-2 Findings: Grantee and Partner Activities, 6-3 Findings: Outcomes, 6-9 Findings: Structural and Environmental Change, 6-13 7 CONCLUSIONS ABOUT THE CARA PROGRAMS 7-1 Limits on Inferences About Program Effectiveness, 7-2 Assessing Grantee Activities, 7-3 Significant Obstacles, 7-8 8 PREPARING FOR FUTURE EVALUATIONS 8-1 Introduction, 8-1 Recommendations, 8-2 Implications for the future of the four CARA programs, 8-9 9 CONCLUDING OBSERVATIONS 9-1 REFERENCES R-1 APPENDIXES A Supplementary Program Information A-1 B Contracted NORC Report B-1 C Evaluation Types and Data Requirements C-1 D Committee Member Biographies D-1 PREPUBLICATION COPY: UNCORRECTED PROOFS xvi 

PREPUBLICATION COPY: UNCORRECTED PROOFS xvii 

Preface This is the third report on the evaluation of the congressionally mandated Comprehensive Addiction and Recovery Act (CARA) programs, supported by the Substance Abuse and Mental Health Service Administration (SAMHSA), to address key aspects of the opioid and substance use disorder crisis facing the United States. The four mandated programs address prevention of overdose deaths, support for treatment and recovery across diverse populations, and piloting efforts to address opioid use disorder in perinatal populations. The committee’s work has spanned 5 years, during which it was tasked to comment on the programs’ design and measures, offer evaluation of data and reports for funded programs provided by SAMHSA, and then provide a more formal analysis of effectiveness and cost-effectiveness. The issues addressed across these reports are important; they are timely given the increase in opioid use disorder and consequences, including overdose and death, and overlap in timing with the progression of the COVID-19 pandemic, which was associated with increases in opioid use disorder and other addictions. Such evaluation efforts involve interaction of congressional goals, SAMHSA and funder goals and policies, measures and approaches to interventions and evaluation, meaning of the initiative to funded programs, partners, providers, and client participants. Given the multiple agencies involved, as well as complexity of evaluation in this context, the committee sought to include a range of academic and policy leaders as members based on experience across this range of perspectives. As a result, the committee included experts in cost-effectiveness, implementation of substance misuse programs, behavioral health, emergency response, health policy, SAMHSA projects, and implementation models including qualitative and quantitative approaches. One of the main congressional requests was to evaluate cost-effectiveness, which could inform future initiatives and funding, beyond these four programs. We noted in prior reports and in this one that the structure of the programs and data provided was not consistent with the ability to conduct a formal, data-based evaluation of effectiveness or cost-effectiveness. Rather, the data PREPUBLICATION COPY: UNCORRECTED PROOFS xviii 

largely inform the process of describing the implementation of programs—challenges encountered, strategies tried, and examples of potential impact. As a result, the committee worked to expand the focus on the development of program implementation, including by contracting to have formal qualitative interviews of members of various grantees to better understand their experiences. In addition, the committee sampled from reports to clarify process, context, and when available, examples of impact. Further, to provide “lessons learned” for future efforts in important health policy areas, the committee included recommendations for future evaluations and invited committee members with relevant expertise (qualitative and quantitative) to inform that effort as well as the main activities. Despite limitations in the data described in this and prior reports, the chair, co-chair, and committee members thought that the opportunity to understand what was accomplished by the CARA programs was an important effort to track how policy initiatives implemented through funding by federal agencies may move forward—what can be learned even with data limitations noted—to consider how in the future the nation may learn to have more “front-end” structuring of goals, designs, and data to assure that policy lessons can be learned, in addition to “stories of progress” in implementation, which was the main learning from the data available to us. Kenneth B. Wells (Chair) Hortensia Amaro (Vice Chair) PREPUBLICATION COPY: UNCORRECTED PROOFS xix