Summary
The United States is experiencing high numbers of overdose deaths and a lack of access to treatment. To address these issues, the Comprehensive Addiction and Recovery Act (CARA) (P.L. 114-198) became law on July 22, 2016.1 CARA is extensive legislation designed to address many components of the opioid epidemic, including prevention, treatment, recovery, law enforcement, criminal justice reform, and overdose reversal. It authorizes more than $181 million each year in new federal funding to overcome the opioid epidemic. It mandates the implementation of programs and services across the United States to address substance use disorders (SUDs) and recovery from SUDs.2 While opioid use disorder (OUD) is a significant cause of harm in the United States, it is important to note that many individuals with OUD also have other SUDs, as SUDs tend to co-occur (NIDA, 2020; Seth et al., 2018). The CARA program grants described in this report, while focused primarily on opioids, may also occasionally include treatment and recovery services for co-occurring SUDs. After CARA was signed into law, the Consolidated Appropriations Act, 2018, included appropriations requesting a study of the Substance Abuse and Mental Health Services Administration (SAMHSA) programs in CARA, to be conducted by the National Academies of Sciences, Engineering, and Medicine (the National Academies).3
STATEMENT OF TASK AND REPORT METHODOLOGY
The Statement of Task to the National Academies calls for an ad hoc committee to conduct a review of outcomes achieved by four specific programs funded by SAMHSA, as authorized in CARA. The specific CARA programs are Building Communities of Recovery (BCOR), State Pilot Grant Program for Treatment for Pregnant and Postpartum Women (PPW-PLT), First Responders (FR-CARA), and Improving Access to Overdose Treatment (OD Treatment Access). The committee’s review will result in three reports over 5 years.
- The first report will recommend outcomes and metrics for each of the four programs that grantees should collect and report to SAMHSA.
___________________
1 See https://www.congress.gov/bill/114th-congress/senate-bill/524/text (accessed March 13, 2020).
2 See https://www.cadca.org/comprehensive-addiction-and-recovery-act-cara (accessed March 13, 2020).
3 See https://www.govinfo.gov/content/pkg/CPRT-115HPRT29457/pdf/CPRT-115HPRT29457.pdf (accessed January 5, 2021).
- The second report will review reported outcomes and metrics to assess progress toward achieving program goals.
- The third and final report will review the specified programs for their effectiveness in achieving their respective goals, as measured by the metrics identified, and will provide recommendations to Congress concerning the appropriate allocation of resources to such programs to ensure cost-effectiveness in the federal government’s response to the opioid epidemic.
The present report is the second of the three. The first was released on May 14, 2020, and a summary of its recommendations can be found in Appendix C of this report.4
To address its charge for the second report, the committee gathered data from SAMHSA about each of the grant programs. The committee’s specific methodological approach is outlined in Chapters 2 and 3. In general, however, the committee requested information on work plans, evaluation plans, and progress reports from the principal investigator of each grant and from SAMHSA.5 Varying amounts of information were received from 56 of 87 total grantees.6 For the first report, SAMHSA provided the funding opportunity announcements (FOAs) and mandatory reporting tools for each of the four programs. For this present report, SAMHSA has additionally provided data summaries for the PPW-PLT and BCOR programs in the form of data outputs from the Center for Substance Abuse Treatment (CSAT) Government Performance and Results Act (GPRA) tool, as well as redacted grantee annual progress reports. For the OD Treatment Access and FR-CARA programs, SAMHSA provided summaries generated from text-based answers to the questions in the Center for Substance Abuse Prevention (CSAP) Division of State Programs-Management Reporting Tool (DSP-MRT) and from an additional program-specific reporting tool for OD Treatment Access.
Individual grantee progress reports received from SAMHSA were heavily redacted, resulting in much contextual information not being available to the committee. SAMHSA explained that their agreements with grantees prohibited SAMHSA from sending the committee unredacted program progress reports. To fill in some of these gaps, the committee reached out to individual grantees to request additional information; a limited number of unredacted reports were received. After an initial review of all of the materials received, the committee contacted SAMHSA with clarifying questions about the data, and the agency’s responses were used in the creation of this report.
The committee’s approach to assessing progress toward achieving program objectives was broadly informed by several conceptual methodological approaches applied within the constraints of the available data. Given the limitations of the information and the distinct nature of the four grant programs, the committee assessed progress by reviewing and matching information received to each of the required and allowable activities specified in the FOAs (see Tables S-2 and S-3). The committee considered whether there was evidence regarding progress in terms of planning and implementation steps (e.g., hiring, training staff, relationship building) and outcomes (e.g., client-based substance use outcomes, naloxone distribution and use, public education, trainings conducted). Importantly, the committee notes that the grantees of all four CARA programs have continued implementing the programs since the point in time at which they submitted these required reporting materials to SAMHSA. As such, this report does not reflect grantees’ most recent progress.
The committee encountered many limitations in the data sources it received from SAMHSA. Table S-1 presents a summary of these limitations and their impact on the committee’s analysis.
___________________
4 For the full report, see NASEM, 2020b.
5 Public Access File available by request via the National Academies. See https://www.nationalacademies.org/our-work/review-of-specific-programs-in-the-comprehensive-addiction-and-recovery-act (accessed March 10, 2021).
6 The committee received information from 6 of 6 PPW-PLT grantees, 25 of 27 BCOR grantees, 4 of 6 OD Treatment Access grantees, and 21 of 48 FR-CARA grantees.
TABLE S-1 Limitations of Data Sources Used by the Committee
| Cross-Cutting Limitations | Impact on Analysis | ||
|---|---|---|---|
| Reporting tools not clearly linked to the requirements of the grant programs, per funding opportunity announcements | Difficult to organize information received around program requirements | ||
| Absence of pre-program data | Difficult to interpret data, particularly on program impact | ||
| Lack of information about each grantee’s local context | Cannot account for policy context, cannot compare outcomes to expected outcomes/status in a given community | ||
| Lack of information on other programs in same area with similar features (whether funded by SAMHSA or others) | Difficult to assign impact to SAMHSA programs | ||
| Sources of data, reliability, meaning of data in context not always clear | Difficult to interpret data, particularly on program impact | ||
| Evidence Source Used by Committee | Program with This Source Type | Limitations of Evidence Source | Impact on Analysis |
| CSAT Government Performance and Results Act data reports | PPW-PLT, BCOR | Not all grantees submitted (25/27 BCOR grantees) | Results only speak to progress of reporting grantees |
| Individual-level data | Does not speak to community-level or systems-level change | ||
| Unclear whether clients were (a) enrolled in other programs simultaneously or (b) participating in the grantee programs prior to the reporting period | Cannot isolate effect of SAMHSA grant funding on client outcomes | ||
| Potential bias in client outcomes data, due to (a) low follow-up rates or (b) characteristics of patients with follow-up data | Client progress, in aggregate, may be overstated | ||
| Data aggregated across grantees (without pre–post data or comparison groups) | Difficult to interpret effects of SAMHSA grant funding on client outcomes; redactions prevent interpretation of the context for client outcomes, either in terms of program features or community/population context or data | ||
| Evidence Source Used by Committee | Program with This Source Type | Limitations of Evidence Source | Impact on Analysis |
| Grantee progress reports | PPW-PLT, BCOR | Not all grantees submitted (25/27 BCOR grantees) | Results only speak to progress of reporting grantees |
| Redacted information (applies to 2/6 PPW-PLT grantees; 24/25 reporting BCOR grantees) Variation in the reporting periods covered by individual grantee reports Variation in clarity, quality, completeness of information provided |
Relevant activities undertaken may be misunderstood, misrepresented, or not included Difficult to compare across grantees; some grantee activities may not be included Difficult to compare across grantees |
||
| Summary Report from the CSAP Division of State Programs Management Reporting Tool (DSP-MRT) | OD Treatment Access, FR-CARA | Not all grantees submitted (4/6 for OD Treatment Access, 21/48 for FR-CARA), therefore not all grantees are reflected in the summaries | Results only speak to progress of reporting grantees |
| Unclear reporting period (OD Treatment Access) | Unclear how much progress may have been accomplished before/after that may not be reflected in summary | ||
| Report from the OD Treatment Access Reporting Tool | OD Treatment Access | Material summarized across grantees, redacted (some subsections of OD Treatment Access Reporting tool included disaggregated, verbatim responses; limitation does not apply for these few responses) | Unclear how many grantees reported a given activity; aggregated quantitative data prevents analysis |
| Summaries created by funder (rather than independent evaluator) | Difficult to confirm interpretation of data; unclear how SAMHSA determined which information would be included | ||
| Variation in grantee responses to prompts (likely due to unclear prompts) | Inconsistent reporting of activities | ||
| No explanations are provided as to why some grantees left questions blank | Unclear whether unanswered questions indicated no progress, some progress, or simply a skipped question | ||
| Some material in the two report types for OD Treatment Access directly contradict one another | Unclear which is correct; cannot interpret these data | ||
| SAMHSA Responses to National Academies Questions | PPW-PLT, BCOR, OD Treatment Access, FR-CARA |
At times, the committee requested clarification on a point (e.g., reporting period for another reporting tool), and gaps remained in SAMHSA’s responses | Did not always provide additional clarifications |
NOTE: BCOR = Building Communities of Recovery; CSAP = Center for Substance Abuse Prevention; CSAT = Center for Substance Abuse Treatment; FR-CARA = First Responders-Comprehensive Addiction and Recovery Act; PPW-PLT = State Pilot Grant Program for Treatment for Pregnant and Postpartum Women; SAMHSA = Substance Abuse and Mental Health Services Administration.
PPW-PLT AND BCOR PROGRAMS
Two of the CARA grant programs—PPW-PLT and BCOR—are administered by CSAT. Table S-2 summarizes key characteristics of these programs.
TABLE S-2 Summary of the PPW-PLT and BCOR Programs
| Program | Eligible Applicants | Awards and Project Period | Funding Restrictions | Program Purpose | Program Activities |
|---|---|---|---|---|---|
| PPW-PLT | Single state agencies for substance abuse | 2017 awards: 3 ($1,100,000 each) 2018 awards: 3 (up to $1,100,000 each) Up to 3 years |
No more than 15 percent of the federal funding may be allocated to data collection, performance measurement, and performance assessment. A minimum of 75 percent of federal funds must be directed toward service provision; no more than 25 percent may go toward infrastructure or capacity building. |
The purpose of this program is to “enhance flexibility in the use of funds designed to
Through this program, SAMHSA seeks to “reduce the misuse of alcohol and other drugs, increase engagement in treatment services, increase retention in the appropriate level and duration of services, and increase access to the use of medications approved by the Food and Drug Administration (FDA) in combination with counseling to treat drug addiction.”a |
Required Activities
Allowable Activities
|
| Program | Eligible Applicants | Awards and Project Period | Funding Restrictions | Program Purpose | Program Activities |
|---|---|---|---|---|---|
| BCOR | Recovery community organizations (RCOs) that are domestic private nonprofit entities in states, territories, or tribes (must be controlled and managed by members of the addiction recovery community) | 2017 awards: 8 (up to $200,000 each) 2018 awards: 18 (up to $200,000 each) Up to 3 years |
No more than 20 percent of the federal funding may be allocated to data collection, performance measurement, and performance assessment. Must receive at least 50 percent of funds from nonfederal sources with a 100 percent matching requirement. |
The purpose of this program is to “mobilize resources within and outside of the recovery community to increase the prevalence and quality of long-term recovery support from substance abuse and addiction. These grants are intended to support the development, enhancement, expansion, and delivery of recovery support services (RSS) as well as promotion of and education about recovery.” |
Required Activity
Allowable Activities
|
a The committee prefers the use of the terms substance use disorder (SUD) and opioid use disorder (OUD); “addiction” will be used when directly quoting grantees or SAMHSA.
SOURCES: The information in this table is pulled directly from the initial funding opportunity announcements for PPW-PLT (SAMHSA, 2017b) and BCOR (SAMHSA, 2017a). The information in the Awards and Projects column comes from the Individual Grant Awards page for PPW-PLT (SAMHSA, 2017h, 2018g) and BCOR (SAMHSA, 2017g, 2018f).
The committee found evidence from both the CSAT GPRA data and from the progress reports that grantees in both programs were operational in a number of activities, including providing services to clients, conducting important community outreach, providing community education, and, to some extent, building systems and infrastructure. For example, the GPRA data indicate that, although they were below planned goals, intake interviews have been conducted for 839 clients in the PPW-PLT program and 2,022 in the BCOR program.
All PPW-PLT grantees were functioning, recruiting, and serving clients, but at varying levels. All programs lagged behind in the recruitment goals they had set, and most programs had lower follow-up interview rates than the 80 percent expected by SAMHSA. Some programs cited greatly delayed approval for first-year budgets as an early setback, as well as a lack of clarification on certain questions regarding client eligibility and permitted uses of SAMHSA funds. It seems that they would have benefited from greater support, technical assistance, and training on aspects of programming and evaluation. The information on integrated treatment and system change–related goals was sparse, and these aspects were not evaluated by any quantitative data but rather came from descriptions in the progress reports, when grantees provided them.
The BCOR program was funded by SAMHSA primarily to support the “development, expansion, enhancement, and delivery of” community and statewide recovery support services (RSS). Based on a review of redacted progress reports and GPRA data (which were not designed for activities beyond direct client service provision), the committee finds that the BCOR programs as a whole made progress toward this goal. The progress reports indicate a wide range of ways in which RSS were developed, expanded, and enhanced, and the GPRA data indicate that many clients directly received RSS, although the data do not allow for assessment of how effective those services were. Individual grantee programs, based on the progress reports, were more or less successful. Some of this depends on how well developed a grantee’s program had already been prior to receiving this funding, its ability
to hire staff and implement infrastructure needs, and the specifics of its own goals. Many programs addressed the additional allowable activities: “(1) build connections through infrastructure building between recovery networks, between recovery care organizations, and with other RSS; (2) reduce the stigma associated with drug and alcohol addiction; and (3) conduct public education and outreach on issues relating to drug and alcohol addiction and recovery” (SAMHSA, 2017a). Each of these is assessed only via the progress reports.
OD TREATMENT ACCESS AND FR-CARA PROGRAMS
Two of the CARA grant programs—OD Treatment Access and FR-CARA—are focused on preventing overdose and are administered by the CSAP. The programs are summarized in Table S-3.
TABLE S-3 Summary of the OD Treatment Access and FR-CARA Programs
| Program | Eligible Applicants | Awards and Project Period | Funding Restrictions | Program Purpose | Program Activities |
|---|---|---|---|---|---|
| OD Treatment Access | Federally qualified health centers (FQHCs); opioid treatment programs; practitioners dispensing narcotic drugs | 2017 awards: 1 (up to $1,000,000 each) 2018 awards: 5 (up to $200,000 each) Up to 5 years |
No more than 20 percent of total award may be used for (1) purchasing or distributing drugs or devices to treat overdose and (2) offsetting copayments or other cost sharing for such drugs or devices. No more than 20 percent of the total award can be used for data collection, performance measurement, and performance assessment. |
The purpose of the program is to “expand access to FDA-approved drugs or devices for emergency treatment of known or suspected opioid overdose.” |
Required Activities
Allowable Activities
|
| Program | Eligible Applicants | Awards and Project Period | Funding Restrictions | Program Purpose | Program Activities |
|---|---|---|---|---|---|
| FR-CARA | States; tribes and tribal organizations; local governmental entities | 2017 Awards: 21 (between $250,000 and $800,000 each) 2018 awards: 27c (between $250,000 and $800,000 each) Up to 4 years |
No more than 20 percent of the federal funding may be used for data collection, performance measurement, and performance assessment expenses. No more than 10 percent may be used for administrative costs. | The purpose of this program is to “allow first responders and members of other key community sectors to administer a drug or device approved or cleared under the Federal Food, Drug, and Cosmetic Act for emergency treatment of a known or suspected opioid overdose.” |
Required Activities
Allowable Activities
|
a See the toolkit at https://store.samhsa.gov/product/Opioid-Overdose-Prevention-Toolkit/SMA18-4742 (accessed March 13, 2020).
b The committee prefers to use the term “survivors”; however, “victims” will be used when directly quoting SAMHSA.
c SAMHSA initially awarded funding to 28 grantees in this year. One of the original grantees rescinded (see Chapter 3 for further discussion).
SOURCES: The information in this table is pulled directly from the funding opportunity announcements for OD Treatment Access (SAMHSA, 2017d) and FR-CARA (SAMHSA, 2017c). The information in the Awards and Projects column comes from the Individual Grant Awards pages for OD Treatment Access (SAMHSA, 2017f, 2018e) and FR-CARA (SAMHSA, 2017e, 2018d).
For both programs, the committee found evidence that some grantees had taken steps to address required activities. For the OD Treatment Access program, two of four grantees clearly established prescribing or co-prescribing services at their sites. All of the program’s grantees created curricula and delivered training on the prescribing of overdose drugs and devices, used SAMHSA’s Overdose Prevention toolkit as a guide, and described lessons learned and best practices. However, scant information was provided about the outcome of the trainings, and the quantitative data left ambiguity about the types of audiences that were reached. Additionally, the four grantees responding described their protocols for connecting overdose survivors to treatment, and reported that a total of 139 overdose survivors were connected to treatment and that 76 initiated treatment. While all grantees described their mechanisms for referrals to treatment, only in the case of one grantee was the committee able to determine that the survivors were connected to treatments that were appropriate or evidence-based. All four grantees directly distributed overdose reversal drugs and devices and formed partnerships across a range of sectors. The grantees identified major obstacles toward sustaining their programs after federal support ends. There was little available
information about whether grantees had engaged in the three allowable activities to collaborate with health care professionals and pharmacies, and to assist in the provision of public education about Good Samaritan laws, though some pieces of the reports suggested that small steps had been taken.
The information received from SAMHSA about the FR-CARA program indicates that with regard to the primary goal of allowing first responders to administer a drug or device for emergency treatment of known or suspected opioid overdose, grantees distributed 14,509 nasal spray kits and 1,925 other kits. The information from SAMHSA suggests that 21 percent of the naloxone kits distributed were administered and 1,815 overdoses were reversed. Training sessions (n = 624) were conducted, with 15,581 individuals involved in these trainings. SAMHSA reported that, overall, grantees had developed 17 new strategies for referring overdose survivors to treatment and recovery services; at the time of reporting, grantees had distributed information about treatment options to 570 overdose survivors and families, and many reported that overdose survivors were receiving various types of treatment, including medications for OUD, counseling, and more. Twenty grantees had held meetings with advisory councils that they had joined or that were newly established during the reporting period. Some, but not all, grantees engaged to varying degrees in the allowable activities to collaborate with health care professionals and to assist in the provision of public education about applicable Good Samaritan laws. No information was provided on the effectiveness of these trainings or communication campaigns.
IMPACT OF COVID-19
The committee understands that the COVID-19 pandemic inevitably has affected grantees’ ability to function as envisioned in their grant applications. A few grantees mentioned COVID-19 in the material received by the committee. Evidence shows that since the start of the pandemic, substance use and overdoses have both increased (Ahmad et al., 2020; CDC, 2020a; Czeisler et al., 2020; Slavova et al., 2020; Stack et al., 2020). The committee acknowledges that the pandemic might slow progress, especially in implementation. SAMHSA has COVID19-related guidance for their grantees.7 The committee looks forward to hearing from SAMHSA and the grantees about innovative steps taken to reach those with SUD, for example, by providing services through telehealth.
CONCLUSIONS
The committee concludes, based on information provided by SAMHSA, that grantees providing data have shown at least some progress in planning and implementing the four Comprehensive Addiction and Recovery Act programs under review. The degree of implementation and progress and the nature of supporting data vary across programs, grantees, and specific activities; data were not provided on all grantees.
The committee concludes that it cannot determine whether these programs have had specific, identifiable impacts on people with substance use disorders. It is very possible that these programs have had a positive impact on people’s substance use and on their health and well-being, but the limitations of the information provided do not allow for confidence in such a determination.
The committee concludes that it cannot determine whether the Comprehensive Addiction and Recovery Act programs have had a positive impact on advancing systems change in substance use prevention and treatment or in advancing systematic interagency collaboration. The lack of systematic, quantifiable, or descriptive data does not allow for such a determination.
In light of the limitations identified above and as reflected in the conclusions, the report continues with a discussion of several approaches, including information needs, the committee can take in preparation of the third and final report in the series.
___________________
7 See https://www.samhsa.gov/coronavirus (accessed December 31, 2020).
THE FORTHCOMING FINAL REPORT
The purpose of the third and final report by this committee, due in March 2023, is “to review the specified programs for their effectiveness in achieving their respective goals as measured by the metrics identified and to provide recommendations to Congress concerning the appropriate allocation of resources to such programs to ensure cost-effectiveness in the federal government’s response to the opioid addiction epidemic.” The committee addresses the feasibility of a more robust third and final report by discussing separately the two parts of the mandate: assessing effectiveness in achieving goals and offering recommendations to Congress.
As illustrated in this second report, and discussed extensively in Chapters 2 and 3 and briefly above, there are serious limitations with the material provided by SAMHSA, which might or might not be overcome in the next 2 years. The committee is concerned that its ability to draw firm conclusions on the specified CARA programs will likely continue to be limited.
The committee uses these identified limitations, described in the previous sections of this chapter and throughout the report, to lay out a set of information needs that would allow it to design and perform a more rigorous evaluation of program effectiveness, recognizing that these programs are fully under way, and some are soon to be completed. The committee also lays out three potential approaches to addressing cost-effectiveness in the federal government’s response to the opioid crisis. The committee looks forward to conversations about how it can be most helpful to SAMHSA and to Congress in anticipation of the third and final report.
