7

Conclusions About the CARA Programs

In its second report, the committee’s conclusions about progress to date were limited. As discussed in previous chapters, for this third and final report the committee reviewed final data and progress reports from most grantees and, for the overdose prevention programs (Improving Access to Overdose Treatment [OD Treatment Access] and First Responder Training [FR-CARA]), reviewed much more information than was available previously. The committee makes no conclusions about individual grantees or specific Comprehensive Addiction and Recovery Act (CARA) grant programs, nor does the committee make conclusions about “effectiveness in achieving their goals.” As discussed in the second report in this series, none of the data the Substance Abuse and Mental Health Services Administration (SAMHSA) collected and gave to the committee is suitable for an assessment of “effectiveness,” let alone cost-effectiveness. This is discussed in more detail in the next chapter.

Nevertheless, as illustrated in the four previous chapters, the materials the committee reviewed included a great deal of description about the efforts and activities that grantees undertook using CARA funding, which hold promise as a step toward addressing the opioid use disorder epidemic. It is the committee’s hope that the evidence it has highlighted and described in this report, though not a formal effectiveness evaluation, still highlights the hard work of SAMHSA and its grantees to confront an important public health challenge.

The committee’s conclusions apply to all four programs and are based on the information found in previous chapters. The committee’s seven conclusions are organized around three themes: limits on inferences about

program effectiveness; grantee activities; and barriers, including those that were outside of the control of grantees or SAMHSA.

LIMITS ON INFERENCES ABOUT PROGRAM EFFECTIVENESS

The charge to the committee was to assess grantee effectiveness in achieving program goals. As discussed in the second report, throughout the current report, and as will be discussed in greater length in Chapter 8, the committee could not make conclusions about program effectiveness. The committee offers one conclusion regarding this issue, which pervades most of the other conclusions.

SAMHSA’s off-the-shelf, mandatory reporting tools—the Government Performance and Results Act (GPRA) reporting tool for Center for Substance Abuse Treatment (CSAT) programs and the Division of State Programs Management Reporting Tool (DSP-MRT) for Center for Substance Abuse Prevention (CSAP) programs—required grantees to track and submit a great deal of information, but not all of that information aligned with the specific goals of the four CARA programs. For example, none of the CSAT reporting tools asked grantees to share information about partnerships developed, which was a central goal of both the Building Communities of Recovery (BCOR) and State Pilot Grant Program for Treatment for Pregnant and Postpartum Women (PPW-PLT) grants according to their Funding Opportunity Announcements (FOAs). The outcomes data that the committee did receive were also at times not aligned with the goals of the program (e.g., the GPRA tool tracked long-term treatment outcomes that were appropriate for individuals who were in treatment, but not for the people being trained as recovery coaches for the BCOR program).

As a result of this misalignment, grantees were not prompted to share information that might have been more pertinent to the committee’s evaluation. In its first report, the committee made suggestions for improving SAMHSA’s mandatory reporting tools, but it was not until they began work on the second report that the extent of the mismatch became obvious. No amount of improvement in the existing reporting tools can overcome that they are the wrong tool for the stated purpose of understanding the reach, scope, and impact of the CARA programs.

Additionally, the information that the committee did receive that related to program goals and targeted outcomes often was not suitable for a

formal evaluation. First, the committee did not receive information from all grantees for the BCOR, OD Treatment Access, and FR-CARA programs, leading to potential selection bias in the information that it did receive. There was a lack of data on direct and indirect outcomes identified using a quasi-experimental or experimental design (i.e., using comparable control groups or pre-post design). There was a lack of information on broader community trends relevant to the grantees (e.g., other funding, local policy, population demographics, or expected outcomes in a given catchment area), and the aggregation and redaction of grantee identities made it impossible for the committee to identify relevant secondary data. The lack of specific prompts in grantee progress reports also meant that the information provided in response varied across grantees in terms of detail, thoroughness, and focus.

Given these challenges and that the grantees operated in a complex milieu of health care, public health, criminal justice, law enforcement, child welfare, employer, and community programs, often across broad geographic areas, the committee cannot attribute some reported outcomes specifically to CARA programs.

Finally, other considerations—such as timing of the evaluation, details of the data sharing agreements between SAMHSA and grantees, and level of SAMHSA support to grantees for evaluation purposes—from the outset precluded a rigorous evaluation of program effectiveness. These issues will be discussed in greater detail in the next chapter. As discussed in previous chapters, a contract to perform this evaluation was not formally executed until after the grantees had already begun their work; to perform a rigorous evaluation, an evaluator should begin its work prior to the beginning of program evaluation. Additionally, the FOAs for grantees did not clearly set expectations for data sharing agreements with grantees; without such agreements, SAMHSA stated that it was not able to share information in an unredacted, disaggregated format. Finally, grantees may have had insufficient funding, resources, access, and communication with their SAMHSA Government Program Officers or technical assistance (TA) from SAMHSA to collect or analyze data that might have supported an effectiveness evaluation.

ASSESSING GRANTEE ACTIVITIES

As described in Chapter 2–6, the committee reviewed grantee-generated information submitted to SAMHSA. The structure of the reporting tools is described in Chapter 2. The committee took at face value what grantees reported they did; the committee did not independently verify the material submitted. Grantee activities varied across the four CARA programs, according to the structure of the grant programs expectations. Nevertheless, the committee makes four conclusions based on this reporting.

CONCLUSION 2: CARA grantees and their partners engaged in a range of activities to address the worsening substance use disorder epidemic. Based on the information provided, the committee is unable to say whether the programs as a whole were effective.

Despite the limitations noted above, the materials the committee reviewed highlighted evidence that SAMHSA’s grantees worked diligently and used the CARA funding to stand up or expand activities intended to address the needs of individuals with substance use disorders (SUDs), and to address the worsening SUD epidemic. Grantees reported that CARA funding likely facilitated the implementation of these programs even if the exact outcomes are not quantifiable. The committee emphasizes that a lack of evidence about effectiveness does not amount to evidence of ineffectiveness. The committee also emphasizes that the data issues it identified are not related to grantee underperformance, rather to issues around the reporting requirements and agency expectations. This is discussed further in Chapter 8.

Though there were insufficient data available to draw conclusions about the effectiveness of these activities, the committee has attempted to note, throughout the findings chapters, relevant literature about the effectiveness of interventions similar to those that grantees implemented. These might provide context for why the committee has reason to believe that grantees’ activities could have been beneficial. However, because the committee did not have enough information, it also cannot directly compare grantee efforts to this outside literature—it could not, for example, assess for the suitability of an otherwise effective intervention to the specific populations in the catchment areas of the anonymized grantees, or otherwise analyze implementation. Nevertheless, the following paragraphs summarize some of the grantee activity accomplishments described for each of the four programs in the previous chapters.

BCOR grantees primarily provided two main types of direct services: recovery support services and wraparound to individuals with SUDs, and training to individuals seeking to work as peer recovery coaches. Such recovery and wraparound services have been demonstrated to support recovery (Eddie et al., 2019; McLellan et al., 1999; Scott and Dennis, 2009; Stewart et al., 2021), as has peer recovery coaching (Eddie et al., 2019; White et al., 2007). Their work extended far beyond these direct services, however. They developed partnerships across a wide variety of sectors, invested in internal capacity building efforts such as technology systems improvement and physical infrastructure development, and facilitated workforce development. In an effort to spread knowledge about SUD and reduce stigma, they engaged with the broader community through public

events, advocacy efforts and speakers bureaus, public events, resources distribution, and other programming.

Grantees in the PPW-PLT program provided outpatient treatment services to pregnant and postpartum women; several grantees intended to expand the availability of these services to rural areas where they were not previously available. Several grantees used CARA funds to enhance their services by engaging peer support workers, or by training staff in evidence-based practices and particularly trauma-informed care relevant to the PPW patient population (Henry et al., 2021; Sperlich et al., 2017). Like BCOR grantees, they connected with a wide variety of partners to train other service providers, to advocate for policy changes, and to facilitate treatment and ancillary services for services like housing that impact recovery for this population (Frazer et al., 2019; Petzold et al., 2022).

OD Treatment Access grantees created and disseminated training materials, toolkits, and presentations about overdose reversal. In addition to distributing naloxone, they engaged in a broader range of overdose prevention activities as well, including efforts to expand access to medications for opioid use disorder (MOUD) and distribution of fentanyl test strips, both of which are evidence-based (Fairley et al., 2021; Karamouzian et al., 2018; Krieger et al., 2018, NASEM, 2019; Peiper et al., 2019). They undertook steps to support these efforts by increasing staffing, developing their informational technology systems, encouraging co-prescribing in their communities, formulating assessments and approaches to direct interventions to high-risk settings, and strengthening community partnerships with relevant stakeholders.

The FR-CARA grantees also used funds to perform naloxone distribution and training, which have been shown to effective and cost-effective in reducing overdose events (CDC, 2018b; Chao and Loshak, 2019; Chimbar and Moleta, 2018; Coffin and Sullivan, 2013; McClellan et al., 2018; Naumann et al., 2019; NIDA, 2017; Townsend et al., 2020). They took steps to tailor their trainings to a variety of audiences, create and distribute toolkits, and evaluate their efforts. They also focused efforts on partnership formation and collaboration in an effort to extend the reach and potential sustainability of their programming.

The committee does note that some grantees appeared to engage in more activities than others, which the committee suspects is because some grantees had a long history in SUD work and others were newly engaged by the SAMHSA grants.

CONCLUSION 3: CARA funding supported grantees in enhancing or expanding treatment and recovery support services, as well as naloxone delivery. Grantees facilitated the education and training of community

members and professionals about substance use disorder, stigma reduction, and overdose reversals.

Grantees across the four programs undertook efforts to impact the individuals their program activities targeted—whether individuals in treatment or recovery or those in need of naloxone administration. By engaging in educational efforts with community members and with a wide variety of professionals, they also undertook efforts to improve knowledge about SUD treatment, recovery, and overdose reversal and to reduce stigma.

Many grantees noted that the CARA funding allowed them to expand their services to additional clients or even additional population types (e.g., to expand services into rural areas).

Though the committee was not able to identify the precise individual-level or community-level outcomes of these efforts, and therefore cannot comment on their effectiveness, these types of efforts have been demonstrated to impact communities and individuals in other settings; as such, the committee has reason to believe that these efforts have potential to be beneficial.

CONCLUSION 4: Partnerships were a key feature of all four grant programs. Grantees varied in the success of partnership building and the impacts these partnerships have on reach and structural change.

Grantees across all four programs emphasized their efforts to build and sustain partnerships with a wide variety of community partners. Grantees engaged with SUD providers including clinicians and other organizations providing treatment and recovery services; they connected with ancillary support, social service providers, and employers as well; and they partnered with advocacy groups and initiated relationships with the criminal justice system, child welfare services, local and state governments. The committee underscores that cross-sector collaboration such as this is crucial to approaching the SUD epidemic, as both its causes and solutions demand multidisciplinary approaches.

There was some variability in grantees’ success in partnership building. To some extent, this seemed to reflect grantees’ capacity to build partnerships at the beginning of grant funding—that is, some grantees were initiating partnerships and others were strengthening or expanding existing partnerships. The COVID-19 pandemic exacerbated the challenges of partnership building.

The importance of partnership building to these four CARA programs was underscored by the FOAs that SAMHSA released—all four include either required or allowable activities that explicitly instruct grantees to engage in connection, partnership, and network building.

But beyond just being a requirement of the grant, grantees underscored the value of those partnerships to both their short-term goals, such as providing treatment and recovery services and training others in naloxone prescribing and use, and long-term goals. Relevant long-term goals involve building grantee capacity, improving community education about SUD and overdose reversal, influencing local systems and policies, and assuring funding for work after the SAMHSA grant ended.

CSAP grantees highlighted immediate benefits of partnerships such as connection to and buy-in from new audiences for their naloxone and co-prescribing trainings, and facilitation of referrals of overdose survivors to supportive services or medical treatment; in the longer term, they expressed that these partnerships could increase communication among agencies and result in permanent and sustainable changes in systems-level protocols and processes related to the care of people with SUD. CSAT grantees highlighted similar immediate benefits to their partnerships—they felt they increased referrals to their services from their partners and enabled the grantees to connect their clients more easily to a range of helpful services; in the longer term, they felt that their efforts to educate partners or build relationships with new entities likely spread community knowledge about SUD and reduced stigma. Finally, many grantees expected that these partnerships could increase sustainability of the efforts undertaken by grantees of the four programs.

There was variability in the information provided to the committee about partnerships, particularly among the CSAT grantees. The GPRA tool used by the CSAT programs does not ask grantees about partnerships. The progress reports SAMHSA requested also did not include specific prompts about partnerships; though essentially all grantees elected to highlight this important activity, the information provided varied and was not systematic. On the other hand, the DSP-MRT tool used by the CSAP programs did ask questions about partnerships and sustainability efforts, so all grantees were asked to comment on them. As with most responses to the DSP-MRT, responses varied in their directness and completeness, as such assessments of their effectiveness are not possible.

The committee identified many activities that grantees participated in that have the potential to develop or change existing systems and partnerships in such a way that could impact individual and community outcomes in the long run. Activities intended to impact systems and policies take time, and certainly more time than the short funding periods of the CARA grants; as such, it is not surprising that the extent to which these activities produced their desired effects on individuals or communities may not be visible yet. Nevertheless, the committee notes that grantee activities have potential to create such change.

Partnerships and network-building efforts, as previously noted, have the potential to improve coordination among multiple sectors and facilitate

service delivery, strengthen care infrastructure, spread knowledge about SUD, and reduce stigma among partners and public members that may not have previously been bought in. Grantee efforts to create resources, spread evidence-based practices through trainings, and create protocols could increase the sustainability of their efforts. Many grantees participated in efforts to secure funding that would sustain CARA-related services past the end of the grant periods—examples included changing organizational legal status to fundraise, identifying other grants, or meeting qualifications to bill their services through Medicaid. Finally, many grantees engaged in advocacy and policy efforts to create long-term change that, if successful, could facilitate future efforts to address SUD or its determinants.

SIGNIFICANT OBSTACLES

The committee appreciates that grantees and SAMHSA staff operated throughout at least some of the programs’ lifecycle under the extraordinary stress of the COVID-19 pandemic, which was concurrent with a deepening in the nation’s crisis surrounding the opioid epidemic. The committee further appreciates that the four CARA programs, broad in scope as they were, were limited in what grantees and SAMHSA could address. The first of this set of conclusions pertains to general barriers to implementation that were cross-cutting across the four programs; the next two highlight individually the two biggest barriers.

CONCLUSION 5: Grantees identified several barriers across all four programs that inhibited their ability to achieve some of their goals, including but not limited to grant management, data collection, and staffing.

Several of the challenges that grantees highlighted related to communication and management challenges with SAMHSA.¹ In the NORC report, grantees referred to difficulty achieving program goals in light of the low funding levels and short implementation time frames. They also noted that in some cases there were delays in budget approval, and a lack of communication with Grant Program Officers (in part due to turnover on SAMHSA’s end during the grant period). They sometimes cited a desire for additional TA or the opportunity to share strategies with peer grantees. For the CSAT grantees in particular, many highlighted the high administrative burden of the SAMHSA-mandated reporting tools, the misalignment of the

___________________

¹ See Chapter 8 for a discussion of resource needs for any agency involved in program evaluation.

GPRA tool in particular with program goals, and what a few described as unrealistic follow-up target rates set by SAMHSA.

Data collection was also frequently cited as an obstacle, and one for which grantees might have benefited from additional TA from SAMHSA. As previously mentioned, many of the challenges in data collection cited by CSAT program grantees revolved around the difficulties with the GPRA tool and what they described as unrealistic follow-up target rates set by SAMHSA. CSAP grantees cited difficulties with non-response on follow-up surveys from training participants. They also described encountering difficulties tracking outcomes across disconnected systems, such as on naloxone administration, hospital admissions, connection to treatment or other social services; they cited underdeveloped infrastructure in their areas for collecting tracking that would help inform their efforts.

Grantees across the four programs cited maintaining adequate staffing to implement their programming as a major challenge. They described general shortages in their areas of particular types of staff and high turnover across the board. BCOR grantees described a particular challenge in retaining peer recovery coaches.

Finally, grantees reported challenges in identifying options to complement CARA funding or to sustain their work past the end of the SAMHSA funding. Grantees reported challenges brought on by insufficient funding of their efforts and sought additional funding sources or to qualify for reimbursement through Medicaid for some of their services.

CONCLUSION 6: The confluence of the substance use disorder epidemic and the COVID-19 pandemic posed significant challenges to all grantees, but programs continued. COVID-19 prompted some grantees to engage in novel activities in pursuit of program goals.

As noted in both the scientific literature and the media, SUD increased significantly during the pandemic. On December 17, 2020, the Centers for Disease Control and Prevention (CDC) alerted the nation that fatal drug overdoses in the United States had worsened significantly during the COVID-19 pandemic, and that there were unprecedented rises in overdose rates (CDC, 2020a). The past several years have seen increased rates in SUD and relapses² or return to use for those in recovery (Abramson, 2021; UNODC, 2022). These trends simultaneously made the work of the CARA grantees even more important, increasing the demand and burden on the organizations, and created obstacles to implementing their activities.

___________________

² The citation provided uses the term “relapse,” which is sometimes considered to be stigmatizing. For a fuller discussion on the committee’s choice in terminology, refer to Chapter 1 of this report.

The implementation periods of all CARA grantees overlapped at least somewhat with the COVID-19 pandemic.³ Since the grantees began their work in September 2017 and 2018, the pandemic struck at a time when they were perhaps even more likely to be focused on implementation, having transitioned away from planning phases in earlier years.

Every grantee across the four programs highlighted the effects of the COVID-19 pandemic on their activities during the grant period. The impacts were diverse. Grantees noted general delays in planning and implementation of service delivery. Perhaps most obviously, many in-person services—trainings for CSAP grantees and BCOR, and treatment and recovery services provision for BCOR and PPW-PLT—were halted due to stay-at-home orders. BCOR grantees noted that social distancing was especially troubling for recovery programs, which are heavily focused on interpersonal connection. CSAT grantees noted declines in participation in treatment and recovery programs and difficulty following up with clients as day-to-day life shifted.

Some grantees noted that partnership development became more complicated as priorities shifted, and especially with emergency medical services (EMS) and medical providers who were overwhelmed with COVID-19 cases. Many grantees also attributed increased staffing turnover in part to the challenges of the COVID-19 pandemic and, in some cases, the increased caseloads it brought about.

Of note, some grantees across the four programs highlighted creative strategies they undertook to continue with their program activities. In some cases, they even highlighted unexpected benefits—for example, the transition to online programming at times brought cost savings for services provision or allowed them to reach new populations. However, grantees still noted that certain populations were harder to reach during this time.

CONCLUSION 7: Structural and policy barriers may have limited the ability of grantees to impact the substance use disorder epidemic. The CARA grant programs were not intended to address most of these barriers.

Grantees across the four programs highlighted the effects of systemic and policy barriers on their ability to carry out program activities and achieve program goals.

Grantees repeatedly cited the lack of availability or inequitable access in their local environments to medical and social services that are crucial

___________________

³ For the specific funding time frames of the grantees and a discussion of overlap with the COVID-19 pandemic, see the sections on COVID-19 in each of the four findings chapters.

to supporting survivors of overdose and other individuals with SUD. Such resources included recovery or transitional housing, transportation services, MOUD treatment, childcare needs, and access to internet. The inadequate supply of or differential access to such resources limited who was able to participate in their programs, or negatively impacted recovery after discharge or after an overdose event. The high cost and limited availability of naloxone was also cited by many CSAP grantees as a hinderance to their efforts to distribute naloxone through the community.

The criminalization of SUDs was also frequently cited as a barrier to grantee activities and achievement of program goals. CSAT grantees cited risk of arrest, incarceration, or punitive action by the child welfare system as obstacles to recruiting and maintaining clients in their programs. These factors also likely contributed to difficulties connecting overdose survivors to treatment for the CSAP programs. Policies and structures that work to increase stigma and punishment of individuals and populations using substances are a challenge to improving public health.

Stigma, more generally, was also cited as a broad barrier to a number of activities. Grantees speculated that it caused hesitation from some potential partners, prevented engagement by individuals who might benefit from program participation, and contributed to a general lack of support for the programs.

Finally, broader trends in funding availability and sustainability were a challenge for grantees. Grantees noted challenges brought about by a lack of funding sources and opportunities for SUD work, or the short time frames of soft money opportunities that are often the most available to organizations doing this type of work. This is a larger scale issue in the funding and reimbursement opportunities available in the field at large.

As outlined in this text, it is clear that these obstacles impacted grantees, but that changing them requires systemic change that falls beyond the scope of work of CARA grantees. CARA grantees were not tasked with addressing these structural and policy barriers. Nevertheless, some did engage in exemplary efforts around advocacy and policy change. The committee notes that for programs such as these to reach their full potential to positively impact individuals and communities, they must be accompanied by structural and systems change through policy change at local, state, and national levels.

This page intentionally left blank.