Review of Four CARA Programs and Preparing for Future Evaluations (2023) / Chapter Skim
Currently Skimming:
2 Methods and Approach
2 Methods and Approach
Pages 35-48
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 35... ...
to evaluate grantee progress and the limitations of those data. ASSESSING EFFECTIVENESS: APPROACH Developing the Approach For this final report, the committee received additional types of data for the Center for Substance Abuse Prevention (CSAP)
|
From page 36... ...
For this report, the committee developed a logic model around which to organize the grantee information by the various actors involved in the activities of the CARA programs, displayed in Figure 2-1. The model was
|
From page 37... ...
These findings are laid out in Chapters 3–6. The Logic Model On the top level, each box represents a different type of actor involved in carrying out the CARA programs.
|
From page 38... ...
All information supplied by SAMHSA is available on request through the National Academies Public Access File,2 and information used directly in the report is publicly available in an online appendix on the National Academies Press website as "additional resources" to the published report.3 The committee focused on materials provided by SAMHSA that highlighted the perspectives and experiences of the implementers of the four CARA programs at the grantee level. Given the limited time frame and funding for this evaluation effort, and potential issues for patient privacy and data sharing limitations, the committee was unable to interview grantee 2 Public Access File available by request via the National Academies.
|
From page 39... ...
1. SAMHSA-Mandated Reporting Tools As for the previous report, the committee requested grantees' reporting materials from SAMHSA.
|
From page 40... ...
had mandated reporting through the Division of State Programs Management Reporting Tool (DSP-MRT) ; the OD Treatment Access program had an additional reporting tool, the OD Treatment Access Reporting Form, and the FR-CARA program had program-specific questions within one of the modules of the DSP-MRT.
|
From page 41... ...
number of actual client intakes at each grantee site versus the grantee's goal for client intakes, and (2) number of 6-month follow-up interviews conducted at each grantee site versus the grantee's goal for 6-month follow-up interviews.6 As some grantees reported outcomes for patients in treatment while others reported outcomes of trained personnel, the aggregated information may have mixed outcomes for two different types of populations.
|
From page 42... ...
Finally, the committee lacked additional information that might have contextualized the GPRA outcomes data. This includes control groups; information about expected outcomes/status in a given community; preprogram data; and information about other treatment or recovery programs operating in the service regions of these grantees, given that individual clients could be interacting with multiple service providers, making it difficult to determine which outcomes are attributable to the PPW-PLT and BCOR funding programs.7 Because the information the committee did receive was aggregated and not attributable to specific grantees, the committee was not able to seek out such secondary information on its own and compare that to the GPRA outcomes.
|
From page 43... ...
For example, the "Use and Reach of Prevention Efforts" section of the Behavioral Health Disparities module, asked four questions (and allowed for free text responses to each) : "How do you monitor the efforts related to addressing behavioral health disparities at the community level?
|
From page 44... ...
and the disproportionate number of responses described above, the committee used a purposeful quota sample to capture the full range of potential reports for reviewing the free text responses grantees submitted through the DSP-MRT. Given the short time frame available for analysis, the committee reviewed a total of 250 OD Treatment Access free text responses, selected proportionally across cohorts and modules and reporting periods and a total of 500 responses from the FR-CARA 8 Citation refers to the source of the material from that submitted by SAMHSA and included in the online appendix.
|
From page 45... ...
As another example, for each question querying accomplishments and barriers, the committee received the "count" of how many times a specific item was selected; however, one grantee could have selected multiple accomplishment categories and others none; thus, it was sometimes difficult to interpret much of this information. CSAP: OD Treatment Access Reporting Tool In addition to the more general CSAP DSP-MRT, SAMHSA created a supplemental reporting tool specifically for the OD Treatment Access grant program, called the OD Treatment Access reporting form.
|
From page 46... ...
Twenty-eight grantees sent material ranging from one-page summaries to multiple reports spanning several years. Progress reports sent by FR-CARA and OD Treatment Access grantees appeared to contain primarily material identical to the free text submissions to the DSP-MRT sent to the committee by SAMHSA and were not reviewed separately.
|
From page 47... ...
Importantly, as stated in the second report, grantee progress reports were inconsistent; they varied greatly in completeness, clarity, and quality. Some progress reports provided thorough descriptive and quantitative data on recruitment, retention, and services provided to clients, whereas others did not provide such data, or the data were contradictory or inconsistent.
|
From page 48... ...
In order to protect grantee confidentiality, among participating grantees, findings were often broken out into CSAP programs versus CSAT programs; as such, comments often cannot be attributed to grantees of specific programs. Overall, the report describes useful general findings that often echoed or provided greater detail to the committee's findings from other sources.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.