8
Preparing for Future Evaluations
INTRODUCTION
Over the past 5 years, the committee has reviewed the four Comprehensive Addiction and Recovery Act (CARA) programs and the information that the Substance Abuse and Mental Health Services Administration (SAMHSA) and its grantees have supplied. As illustrated in previous chapters, these data did not lend themselves to answering all of the questions that Congress requested the committee consider. The limitations of those data have been outlined in this report as well as in the second report of this series.
The first chapter of this report describes in detail the committee’s tasks, and how the committee interpreted its task in light of those limitations. The original language for its task is included in Box 8-1.
In light of the data limitations described earlier in this report and as proposed in its second report, the committee addresses the second part of its task, on “recommendations to Congress” with forward-looking commentary on “how, in the future, Congress can specify required evaluation methods and processes to assess program effectiveness and cost-benefit analysis, and how SAMHSA can improve program evaluation capacity and efforts in similar programs,” which “would result in more robust evaluation assessment and information for decision makers responsible for guarding the public’s health and related expenditure decisions” (NASEM, 2021).
This chapter lays out the committee’s two recommendations for future evaluations. The committee chose to propose recommendations that could be applied to the future program development and evaluation efforts of federal programs broadly, rather than just to those conducted by SAMHSA. So, while the earlier chapters of this report focus on the four CARA programs and the evaluation effort conducted by the National Academies in partnership with SAMHSA, this chapter looks beyond these specific entities or this particular evaluation effort. As such, the recommendations provided will refer to “the evaluator,” “the implementing agency,” “the program,” and “the grantees,” more generally, as these recommendations would apply to any evaluator, federal program, or implementing agency of the federal program. These are recommendations for what these types of entities, in general, require in order for a successful evaluation to be conducted. However, the committee does include specific illustrative examples drawn from the CARA evaluation experience, where relevant, to help explain the recommendations.1
On a final note, these recommendations are offered with the recognition that Congress is sometimes responding to a crisis when it passes laws to create new programs and initiatives. For example, when writing the CARA legislation, the country was facing the burgeoning opioid epidemic. When responding to a crisis, there are inevitably tradeoffs between speed, quality or effectiveness, specificity, and cost of the response and accompanying evaluation efforts. Congress also may face tension between, on the one hand, being specific and directive to implementing agencies and evaluators, while, on the other, allowing subject matter experts in the agency and evaluating institutions the flexibility to design appropriate evaluations. Nevertheless, the committee offers the following observations and recommendations so that future mandated evaluations may provide useful information to guide policy making.
RECOMMENDATIONS
To obtain information useful for policy making through an effective evaluation requires substantial coordination, support, and data sharing
___________________
1 Where text does not specifically name National Academies or the CARA grant program, and refers instead generally to “the evaluator,” “the implementing agency,” etc., it should not be interpreted as commentary on the experience of this committee in evaluating these specific programs.
among stakeholders (Congress, the implementing agency, grantees and partners, and the evaluator). The first recommendation provides advice for how Congress can coordinate with involved parties to support the evaluation, and the second for how Congress can support the implementing agency and, through it, as applicable, support grantees.
Figure 8-1 presents a visual depiction of the actors involved in evaluations of federal programs, their relationships, and which part of the process the committee’s recommendations would impact.
Recommendation 1: The committee recommends that Congress, when mandating evaluations, confer with the implementing agency and evaluation experts to align expectations with feasibility and resource considerations.
The committee addresses these issues below.
Aligning the Evaluation with Feasibility Considerations
To be useful, evaluations must align with the realities of the programs2 they are designed to assess. This can be challenging, however, as evaluations come in many sizes and shapes and can serve a multitude of functions. Misalignment can occur between the characteristics of the program to be assessed and the type of evaluation selected.
___________________
2 Throughout this chapter and report, “program” is used to refer to an overarching federal grant program (e.g., one of the four CARA programs), rather than to a project conducted by one of the grantees of those programs.
The committee offers general guidance in this chapter on three broad evaluation categories: formative (which refers to the formative or early stage of a new type of program), process, and outcomes (also referred to as summative evaluation). These three types are described along a continuum as shown in Figure 8-2. Appendix C contains a more detailed discussion of these types of evaluations and offers a more in-depth look into cost-effectiveness evaluations, as these were specifically mentioned in the Consolidated Appropriations Act that supported this report.
Formative evaluations help assess the feasibility and acceptability of a program and provide preliminary information on the program’s potential effectiveness. Formative evaluations are most often conducted in the last few years of pilot and demonstration projects. Programs in the early stages of their development may not yet be “delivering all intended services, conducting all activities, or reaching all targeted populations” as planned (Ryan et al., 2019). Formative evaluations can also provide information that can be used to increase the likelihood of successful outcomes by making changes in program implementation.
Process evaluations are used to identify the strengths and weaknesses of an ongoing program with the primary objective being to better understand how the program is operating and the mechanisms of implementation. They also can help determine how the program could be improved. Process evaluations are typically used to assess programs that have been operating for several years and are in the early to mid-stages of development.
Outcomes evaluations are used to systematically assess a program’s effectiveness or cost-effectiveness. Outcomes evaluations are best applied to more mature3 and fully developed programs. To answer such specific questions with high levels of confidence, outcomes evaluations typically use quasi-experimental research designs (including pre-post data or comparable control groups) permitting causal inferences and rely on detailed quantitative measurements that have been systematically collected over time.
Evaluation Misalignment in the CARA Evaluation
In the case of the CARA evaluation, the National Academies (the evaluator in this case) struggled with a misalignment between the type of evaluation it was requested to perform and the characteristics and goals of the CARA programs. Congress was particularly interested in assessing the
___________________
3 In response to the passage of the Government Performance and Results Act Modernization Act of 2010, the Government Accountability Office (GAO) released “Designing Evaluations,” which it referred to as a “guide to successfully completing evaluation design tasks” for federal programs and policies. Many of the committee’s recommendations in this chapter are consonant with that report, and it particularly emphasizes the importance of program maturity to choosing appropriate methodologies (GAO, 2012).
cost-effectiveness of four different programs—a specific kind of outcomes evaluation. Unfortunately, however, the goals and characteristics of the CARA programs did not align with such an evaluation.
The CARA programs were designed to fund a series of 3–5-year projects. Though the State Pilot Grant Program for Treatment for Pregnant and Postpartum Women (PPW-PLT) was the only one of the four with the official designation of being a pilot program, the other three were new programs to SAMHSA (the implementing agency, in this case) as well. The programs lacked the infrastructure and time needed to collect and organize a complex and data-intensive assessment, especially given that SAMHSA set caps for the percentage of funds that grantees of the four CARA programs could spend on evaluation. Additionally, SAMHSA contracted with the National Academies to perform the review of the programs more than a year after Funding Opportunity Announcements were issued for the cohorts under study (fiscal years 2017 and 2018), mandatory reporting tools were selected and in use, and grantees had already begun work. This limited options for the National Academies to design an appropriate evaluation.
Given the early development state of the projects, conducting a formative evaluation of how each of the four CARA programs were implemented is likely to have yielded more helpful information for Congress; such an evaluation would have been better aligned with the program goals and characteristics, as well as with the information available to the committee. Such an evaluation could have (1) examined how well SAMHSA was implementing the administration of key components of each program including the solicitation of proposals, selection of grantees, and monitoring of grantee progress, and (2) provided a preliminary assessment of the potential impact the demonstration projects might be having on their targeted populations and the larger communities in which they were embedded. A formative evaluation of this kind would have provided insights into how
SAMHSA could improve its administration and oversight of the congressional funds and provided valuable lessons learned from the grantees on how to implement these projects on the ground.
To some extent, the second report and the present, final report from the committee attempt to accomplish these goals, since outcomes evaluations (on effectiveness and cost-effectiveness) were not possible. However, fundamental mismatches like those encountered in the CARA evaluation may not be recoverable.
Confer with the Implementing Agency and Evaluation Experts
To achieve the alignment discussed above, Congress should confer with the implementing agency and evaluation experts. To conduct a comprehensive and useful evaluation—one that aligns with the program goals and characteristics and provides the information Congress sought—they need resources, including adequate planning time, to design an appropriate evaluation. To facilitate alignment, Congress should provide funding as early as possible to allow for ample time to coordinate with the implementing agency on how to best scope and plan the required assessment activities before grantees begin their implementation process.4 If brought in early, the parties can work together to help tailor data collection and evaluation methodology (e.g., research design, sampling, types of data required and most appropriate means of collecting those data, and analytic approach). It also allows time for the implementing agency and evaluator to set any necessary data sharing agreements, and to communicate those with grantees as applicable. Significant funding delays constrain the evaluator’s methodological flexibility and ability to conduct an appropriate assessment.
Time Frame Challenges in the CARA Evaluation
Following the passage of CARA, which initially proposed an evaluation of the four programs be conducted, there was a significant delay before the Consolidated Budget Appropriation granted the funds to SAMHSA to be able to contract with an evaluator and begin coordinating plans for the evaluation. As previously mentioned, this delay precluded the National
___________________
4 In its 2012 report “Defining Evaluations,” the GAO also highlighted the importance of planning and initiating evaluation processes prior to the beginning of implementation (GAO, 2012). Since its release, a number of studies on U.S. Department of Health and Human Services (HHS) program evaluation have found that this does not always happen in practice. This includes an HHS Office of Inspector General (OIG) evaluation of the State Targeted Response grant (HHS-OIG, 2020) and a GAO report on HHS programs intended to increase the availability of medications for opioid use disorder (MOUD; GAO, 2017).
Academies from working with SAMHSA to ensure that data collection and evaluation methodology were tailored to Congress’s requests.
Recommendation 2: To ensure an informative evaluation in the future, the committee recommends that Congress consider whether the implementing agency has the capacity, mission, and culture to (a) oversee the evaluation and (b) where applicable, support grantees in collecting and sharing data.
A. Overseeing the Evaluation
To ensure that the implementing agency and its grantees can adequately respond to Congress’s evaluation needs, the implementing agency needs sufficient internal capacity and resources, as well as a mission and culture that are supportive of the evaluation effort.
Capacity includes staffing, funding, and adequate time to oversee an evaluation. These internal resources enable the implementing agency to carry out its roles of coordinating with the evaluator and of supporting grantees in their work and in their data collection. In general, implementing agencies may be primarily focused on ensuring the funding allocated for programs is dispersed appropriately and efficiently; though they generally have systems in place for monitoring and evaluating the use of funds, legislation requesting an evaluation of newly allocated funds often requires the creation of additional systems and processes to meet the specific methodological needs of the evaluator and to support the grantees in doing so as well. Such an addition of responsibilities in oversight requires funding, staff with appropriate expertise, and an adequate time frame for updating systems.
Additionally, the pre-existing mission and culture of the implementing agency can impact its ability to support an evaluation.5 In particular, some federal agencies may be more practice-oriented than research-focused in their mandates. This may mean that an agency could have less staff expertise relevant to evaluations, data collection systems that are less suited to evaluations, or less interest in participation. Such agencies may need additional support to oversee an evaluation.
___________________
5 In 2003, GAO highlighted the importance of “evaluation culture” to the success of evaluation strategies in five federal agencies. It defined evaluation culture as “a commitment to self-examination, data quality, analytic expertise, and collaborative partnerships” (GAO, 2003). The importance of “building a culture that values data” was also highlighted in the Office of Management and Budget (OMB)’s memorandum to the heads of executive departments of agencies following the passage of the Evidence-Based Policymaking Act of 2018 (OMB, 2019).
SAMHSA’s Capacity in the CARA Evaluation
As the committee noted in its second report, SAMHSA has a history of conducting or supporting cross-site evaluations (e.g., Bray et al., 2017; Brown et al., 2018; McHugo et al., 2005; SAMHSA, 2018c; Scharf et al., 2014; Steadman et al., 1999) and therefore has had, in the past, the necessary capacity and expertise at the staff level to enable such an effort. The committee did not assess SAMHSA’s funding and staffing levels during the time period of the CARA programs to understand whether it was adequately resourced to support the evaluation. As already highlighted in previous sections, the delay in appropriations limited the time that SAMHSA had available to coordinate with the evaluator to support the evaluation before grantees began implementation.
In terms of mission and culture, SAMHSA is a practice-focused agency, and this guides much of what it and its grantees do. The committee notes a few resources that discuss SAMHSA’s planning around evaluation efforts, though it cannot comment on the extent to which these have been implemented. The 2016 21st Century Cures Act codified SAMHSA’s Center for Behavioral Health Statistics and Quality, to “provide statistical and analytical expertise and promote basic and applied research.”6 Additionally, following the passage of the Foundation for Evidence-Based Policymaking Act of 2018, the OMB released guidance for 24 federal agencies, including HHS, on how to implement best practices in the use and management of federal data for several purposes, including for program evaluation (OMB, 2019). The legislation defined evaluation as “an assessment using systematic data collection and analysis of one or more programs, policies, and organizations intended to assess their effectiveness and efficiency.” In response to this guidance, in 2022, SAMHSA’s Center for Behavioral Health Statistics and Quality (CBHSQ) office released a revision to its policy and procedures, titled “Evaluation of SAMHSA Programs and Policies.” This document generally outlined a plan for approaching the guidance from the Evidence-Based Policymaking Act of 2018, and described how SAMHSA would incorporate it into evaluations, whether conducted internally or externally through a contractor. In this document, it specifically comments on the importance of the “alignment between the type of evaluation activity and evaluation questions with program maturity, complexity, and research goals,” citing GAO’s 2012 report and referring to similar categories of evaluations laid out in this chapter (SAMHSA, 2022).
___________________
6 For more information, see https://www.samhsa.gov/about-us/who-we-are/offices-centers/cbhsq (accessed January 29, 2023).
B. Supporting Grantees
Though not all congressionally mandated evaluations will include grantees, the committee here comments on the primary roles that the implementing agency should play in supporting and guiding grantees, where applicable. Having adequate capacity, mission, and culture, as discussed above, enables the implementing agency to support grantees by (1) selecting appropriate data collection systems; (2) setting data sharing and evaluation agreements; and (3) providing resources and technical assistance (TA) to grantees.
A. Selecting an Appropriate Data Collection System
An implementing agency needs sufficient internal resources to enable them to select an appropriate data collection system; this would ensure that grantees are collecting useful data, and not taking on an unnecessary administrative burden. The selection of an appropriate data collection system should be matched to the goals of the program grant and the evaluation. Data collection tools currently being used by the implementing agency may not be relevant to the goals or evaluations of specific programs.
It is important to recognize that in many cases, implementing agencies are constrained by their existing information systems and data tools, as well as by other federal regulations that prescribe what and how information can be collected and shared across agencies. Developing new measures and tools takes time and resources. For example, applying for and receiving OMB approval for new instruments or larger samples may take a year or more, and is thus infeasible with quick deadlines for funding and startup of grant programs.
Data Collection Systems in the CARA Evaluation
For example, in the CARA experience, SAMHSA’s off-the-shelf reporting tools such as the Government Performance and Results Act (GPRA) and the Division of State Programs Management Reporting Tool (DSP-MRT) were not appropriate for assessing the effectiveness or cost-effectiveness of the programs.7 Supplemental questions were added to the DSP-MRT for the First Responder Training (FR-CARA) and the Improving Access to Overdose Treatment (OD Treatment Access) programs, but the majority of reporting was not tailored to them.
___________________
7 A 2020 GAO report on the evaluation of SAMHSA’s Substance Abuse Prevention and Treatment Block Grant identified a similar mismatch between targeted outcomes and measurement tools (GAO, 2020).
Unfortunately, SAMHSA’s options to create or use different tools were severely constrained given the long process it takes to make data system changes and the relatively short duration of the funding.8
B. Setting Data Sharing and Evaluation Agreements Early
Implementing agencies also require sufficient resources and time to set appropriate data sharing and evaluation agreements with their grantees. Establishing data sharing and evaluation agreements early is essential. If the evaluation requires any data collection and sharing across project sites or among stakeholders (e.g., implementing agency, grantees, partners, and the evaluator), then such agreements are best included as part of the funding description or in the Funding Opportunity Announcements to which potential grantees respond. Without these agreements, the grantees and implementing agency may be unable to share much of the data they have already collected with the evaluator.9 Authorization of such agreements at the outset could also enable evaluators to access client-level data, where relevant to a given program, because grantees could design consent forms for such purposes.
Data Sharing Agreements in the CARA Evaluation
For example, in the CARA experience, many SAMHSA grantees were initially unaware that they would be participating in a cross-site evaluation of the CARA programs or unaware of the specific data collection requirements. This was likely because, with the exception of the PPW-PLT program, SAMHSA’s Funding Opportunity Announcements for the CARA programs did not mention this possibility. CARA grantees in all four programs should have been notified of the evaluation and data collection expectations ahead of time, specifically by having these included in SAMHSA’s Funding Opportunity Announcements. This would have facilitated two processes: (1) the response rate from grantees to requests for information from the National Academies and NORC (the evaluators in this case) might have been higher; and (2) SAMHSA could have provided grantee-specific information, rather than aggregated and redacted/anonymized information.
___________________
8 The committee notes that the Center for Substance Abuse Treatment (CSAT) progress reports were not similarly constrained; SAMHSA may have had the opportunity to tailor these to the Building Communities of Recovery (BCOR) and State Pilot Grant Program for Treatment for Pregnant and Postpartum Women (PPW-PLT) programs. However, the reports were still not clearly structured around the program goals.
9 The committee notes that SAMHSA has established data sharing agreements for multisite programs in the past, such as in the Guide for Applicants for Phase I and Phase II of the Women, Co-ocurring Disorders and Violence Study (McHugo et al., 2005).
C. Providing Resources and Technical Assistance to Grantees
Grantees, subgrantees, and community partners might lack the experience, funding, and capacity to establish and operate effective data collection systems necessary for rigorous cross-site evaluations. They are often busy and under-resourced, and evaluation tasks can pose significant administrative burden. To ensure that the evaluator has access to high-quality and comprehensive data, there must be appropriate funding and TA for grantees to appropriately complete the needed evaluations and data collection tasks. This support must come from the implementing agency, which, in turn, needs adequate resources to be able to provide such support.
Often, implementing agencies set general limits on what percentage of the funds their grantees receive can go toward evaluation. Where Congress is calling for a separate evaluation of a funding initiative, these limits may need to be increased. Additionally, it may be necessary to allocate additional funding to the implementing agency for its role in the evaluation. This would include funding for oversight and redoing data collection system, as stated above, but would also include funds that could be allocated directly to grantees to pay for TA and the needed labor to build and operate appropriate local data collection mechanisms. Implementing agencies could provide that TA directly, or support grantees in utilizing other resources. Data are produced locally, at the grantee level, and organized at the implementing agency level. Without adequate investment at both levels, the evaluation results can be quite limited.
Technical Assistance to CARA Grantees
In the case of the CARA evaluation, grantees described numerous challenges to participating in data collection and utilizing the collection tools mandated by SAMHSA.10 They highlighted the need for additional support and communication with SAMHSA grant program officers and suggested that TA and peer-to-peer sharing across grantees might have benefited their work. Strategies shared in such a manner could even be shared more broadly to groups providing similar services. Some but not all grantees involved external evaluators, but cross-site evaluations were not supported by SAMHSA.
As the committee noted in its second report, SAMHSA has a history of conducting or supporting cross-site evaluations (e.g., Bray et al., 2017; Brown et al., 2018; McHugo et al., 2005; SAMHSA, 2018c; Scharf et al., 2014; Steadman et al., 1999) and therefore has had, in the past, the
___________________
10 For further discussion of this point, see supporting text of Conclusion 1 in Chapter 7 of this report.
necessary capacity and expertise at the staff level to enable such an effort. The committee did not assess SAMHSA’s funding and staffing levels during the time period of the CARA programs to understand whether it was adequately supported to assist grantees in the manner that some grantees would have liked. The 2016 21st Century Cures Act codified SAMHSA’s Center for Behavioral Health Statistics and Quality to “provide statistical and analytical expertise and promote basic and applied research.”11 However, SAMHSA is a practice-focused agency, and this guides much of what it and its grantees do. The committee is aware that grantees and the community-based programs they run are not research sites, as would be expected for university-based research grants, thus the capabilities, expectations, and resources available to grantees inevitably limit the types of analyses in which grantees can engage. For this reason, support from the agency could be especially beneficial.
IMPLICATIONS FOR THE FUTURE OF THE FOUR CARA PROGRAMS
In this chapter, in addition to providing general recommendations for future evaluations of federal programs, the committee has noted some of the obstacles encountered in its evaluation process of the four CARA programs. The committee was not asked to comment on whether or not the four CARA programs should be reauthorized. The committee wishes to note that, should Congress decide to reauthorize these programs and open SAMHSA funding to additional cohorts of grantees, setting them up with these recommendations in mind could enable a more effective evaluation of those cohorts and the programs overall to inform policy. For example, a future evaluation that also includes formative or process components could better capture what in the committee’s opinion includes some exciting contributions by grantees; however, that would at a minimum require that (1) the evaluation timeline were begun prior to implementation by the cohorts to be studied, (2) the evaluator were involved and funded prior to implementation, (3) the GPRA and DSP-MRT data collection tools were edited or that new ones fitted specifically to the program goals were created and utilized, (4) data sharing agreements were laid out clearly in SAMHSA’s future Funding Opportunity Announcements, and (5) additional resources and TA were provided to grantees.
___________________
11 For more information, see https://www.samhsa.gov/about-us/who-we-are/offices-centers/cbhsq (accessed January 29, 2023).
