9

Workshop 3, Session 1: Measuring the Impact of Dissemination Projects

WORKSHOP INTRODUCTION

Lauren Hughes, associate professor of family medicine and state policy director of the Farley Health Policy Center at the University of Colorado, offered brief introductory remarks. She explained that this workshop would focus on measuring the impact of dissemination and implementation projects. Karin Rhodes, the Agency for Healthcare Research and Quality (AHRQ) chief implementation officer, reviewed AHRQ’s role related to the Patient-Centered

Outcomes Research Trust Fund (PCORTF) as well as their recent work related to developing and obtaining feedback about their strategic framework for priorities to guide their work. She then explained that AHRQ interpreted dissemination to include evidence generating, synthesis, translation, communication, assisting with implementing the evidence into clinical practice, and training. The presentations for this workshop were divided into two sessions with a speaker discussion held at the end of each discussion.

EVALUATING DISSEMINATION AND IMPLEMENTATION PROJECTS

Karen Glanz is the George A. Weiss University professor and professor in the Perelman School of Medicine and School of Nursing, associate director for community-engaged research, and program coleader for the cancer control program at the University of Pennsylvania Abramson Center. She discussed the many ways to measure the effectiveness of dissemination and implementation projects and determine which outcomes are most pertinent to stakeholders. Glanz began by discussing the Reach, Effectiveness, Adoption, Implementation, and Maintenance (RE-AIM) framework. The RE-AIM framework is an overarching structure for framing and evaluating the different dimensions or elements of implementation and dissemination research projects that investigators have used extensively for the past 20 years (Gaglio et al., 2013; Glasgow et al., 2019).

Glanz explained that a project she and her colleagues have been conducting for 24 years, called Pool Cool Field Research and Implementation, has generated practice-based evidence about implementation, maintenance, and sustainability. The project, aimed at skin cancer prevention, uses educational and environmental components, including signage, sunscreen, shade structures, and guidance for implementing sun-safe policies. After promising findings from a pilot program and subsequent clinical trial were published, the Centers for Disease Control and Prevention (CDC) encouraged Glanz and her collaborators to disseminate the program beyond the initial research sites. Between 2000 and 2002, her team conducted a pilot dissemination trial involving over 280 pools across the United States (Glanz et al., 2009). This was followed by a hybrid implementation and effectiveness trial at over 400 pools nationwide that ran from 2003 to 2008. As part of that trial, her team conducted a supplemental evaluation of the process used, as well as validation testing of the measures used (Glanz et al., 2010, 2015). This generated approximately 70,000 survey results (Escoffery et al., 2008; Hall et al., 2009).

Glanz continued to operate the Pool Cool website after the end of the study so any pool could reproduce and use the dissemination materials.¹

___________________

¹https://www.med.upenn.edu/poolcool (accessed September 14, 2022).

Glanz highlighted several lessons learned from that project that were applicable to community health programs, programs in health care settings, and others:

• Use all available types of measures.
• Be creative.
• Conduct more labor-intensive measures on a subsample of participants and sites.
• Long-term sustainability can be observed after the formal project has concluded and can serve as practice-based evidence.

Glanz said it is important for research to continue to observe patient behaviors and health outcomes after the implementation ends. This helps researchers identify and measure the outcomes most meaningful to stakeholders. She offered an example of a hypothetical study to evaluate outcomes of an implementation program related to primary care physicians and oncologists referring patients to evidence-based, effective treatment programs for tobacco dependence. She listed three possible outcomes that could be measured to evaluate implementation effectiveness: the number of patients given referrals, how many patients engage with the program immediately following the referral, and what level of cessation of tobacco use the patient ultimately achieves. The third outcome can only be measured if researchers continue observation after the end date of the implementation project.

USING DIGITAL TOOLS FOR DISSEMINATION AND IMPLELENTATION IN THE COMMUNITY

Silas Buchanan, founder and chief executive officer of the Institute for eHealth Equity, explained that his organization is a social impact consulting firm that works with people from underserved communities and communities of color to improve their digital literacy and to educate them about benefits of adopting and using technology to improve health outcomes. The organization also works with innovators in health technology to ensure that the solutions they develop are culturally responsive.

Buchanan said his organization crafts geographically expandable, web-based technology solutions that connect leaders of faith-based and community-based organizations to the people they serve. These leaders and their organizations are usually embedded in historically underserved communities. They are also known and trusted by members of those communities. Buchanan’s organization seeks to create a more inclusive environment that engages faith-based and community-based organizations beyond those that are most visible or those that researchers engage with regularly to connect them with the patients, consumers, and congregants they serve

as well as with health care providers, payers, policy makers, and academic stakeholders.

This work began with a 6-month pilot project called Text4Wellness, a healthy eating and active living campaign driven by text messages. This educational program used two-way, real-time text messaging to leverage relationships with faith- and community-based organizations. The project’s goal was to demonstrate the potential of mobile health technology to reduce health disparities by consistently reaching and engaging people from underserved communities with critical health information.

The project also sought to develop an evidence base on the efficacy of mobile health interventions with targeted populations. That evidence base could serve as a catalyst for the development of new models for public–private partnerships for leveraging mobile health to reach and engage targeted populations. These goals aligned with the U.S. Department of Health and Human Services (DHHS) National Prevention Strategy recommendations to focus on people living in communities that have been made vulnerable, provide community members with the tools and information they need to make healthy choices, and promote positive social actions that support healthy decision making. Buchanan explained that a critical component of that project was developing relationships with community leaders that community members knew and trusted. He emphasized the importance of trust to the success of projects that seek to improve community health outcomes. His organization collected data related to engagement throughout the 6-month pilot program. The 6-month pilot program had 2,500 participants who received over 25,000 text messages and over 500 shared links to various resources. All of those participants completed the program and had a 43 percent response rate to the texted questions.

This project led to a partnership with Morehouse School of Medicine to create other web-based ecosystems to connect leaders of faith-based organizations and support their health ministries. Based on the success of this program, among others, that the Institute for eHealth Equity has piloted, Buchanan’s team is preparing to launch ourhealthycommunity.com, designed to engage with community-based organizations and create connections with stakeholders from health care and health care research.

COMMUNITY ENGAGEMENT IN EVALUATING THE EFFECTIVENESS OF DISSEMINATION AND IMPLEMENTATION PROJECTS

Bethany Kwan is an associate professor and associate vice chair for research for the University of Colorado (CU) School of Medicine’s Department of Emergency Medicine and the director of dissemination and implementation

research at the Colorado Clinical and Translational Sciences Institute. She discussed the mAb Colorado project,² an implementation and real-world effectiveness study of monoclonal antibodies (mAbs) for high-risk outpatients with COVID-19. The three goals of this hybrid implementation and effectiveness study are to assess barriers and facilitators to using mAb infusion treatments statewide based on diffusion of innovations theory; develop, implement, and evaluate innovative strategies statewide to optimize equitable mAb access; and determine the real-world effectiveness and safety of mAb treatment in high-risk COVID-19 outpatients as new virus variants arise (Kwan et al., 2022b).

Kwan and her collaborators established a stakeholder advisory panel to develop a dissemination strategy for their study. They also held community engagement studios to codesign messages and materials for multiple audiences, including those for community-level messaging. The stakeholder panel, which included public health officials and representatives from different faith-based and community-based organizations, met twice a month for a year to ensure that all of the project’s dissemination and implementation activities were designed to meet the needs of the community. Kwan’s team also designed an implementation blueprint to provide a referral checklist for clinician-level effect.³

Kwan’s team, when first engaging with the advisory panel, asked the panel members to identify their goals for the project. The most frequently cited goal was to achieve equitable access to care. The panel expressed concern about reaching people in Colorado’s more rural areas, particularly given the distance rural residents would have to drive to reach an mAb infusion site. The panel believed this would reduce those patients’ interest in receiving mAb therapy. The panel was also concerned about equitable access for people who are uninsured, underinsured, undocumented, or experiencing homelessness, as well as for people from minoritized racial and ethnic groups. Approximately one-quarter of Coloradans identify as Hispanic or Latino, said Kwan, and many only speak Spanish. There are also many people from Native American communities that live in Colorado. The project collaborated with people from those communities to develop culturally responsive materials specifically for those communities. The Colorado Health Institute, an informatics and data-based organization, created maps that Kwan’s team used to assess effects at multiple levels over time throughout the project (Figure 9-1).

___________________

² Additional information is available at https://medschool.cuanschutz.edu/mab-colorado (accessed September 14, 2022).

³ Available at https://medschool.cuanschutz.edu/docs/librariesprovider320/provider-page-library/mab-implementation-blueprint.pdf?sfvrsn=9080bcba_2 (accessed September 14, 2022).

DISCUSSION

Considerations for Community Engagement

Lauren Hughes’s first question to the panelists asked them to identify key study infrastructure elements for engaging the community in evaluating the effectiveness of dissemination and implementation projects. Buchanan replied that bidirectional communication is critically important for ensuring that leaders of faith- and community-based organizations are effectively and authentically engaged. He noted that too frequently, researchers will begin outreach efforts to engage with a community after the project structure and budget have been created. As a result, community members do not have an opportunity to have input into a project that is about them. He added that it would be beneficial for researchers to share their results with the community in a format that the embedded faith- and community-based organizations can use to apply for their own grant funding. Those organizations may have been involved in providing or acquiring data for researchers funded by large grants. However, because those same organizations do not have quantitative or qualitative data for their community, their own grant applications are not as strong and they receive less funding. He also reiterated the important role that trust plays in encouraging leaders of faith- and community-based organizations to become involved in a project.

Glanz emphatically agreed with Buchanan’s comment that having stakeholders involved from the start of a project is extremely important. She explained that an effective approach that her team uses at the initiation of a dissemination project is to seek opportunities for participants from community organizations to develop a sense of ownership for the project. As an example, she has found creating staff development opportunities for community organization staff, such as the lifeguards that participated in her project, was an effective strategy for developing a sense of ownership among members of the community. Her team also found community engagement studios to be valuable for developing a sense of community ownership of a project. She noted that this strategy has remained effective in a virtual format during the COVID-19 pandemic.

Kwan commented that the regional health connectors and practice-based research networks (PBRNs) serve as critical distribution channels into the community because of the strong relationships they have with their communities. She noted that developing productive relationships with these groups takes time, and one reason the mAb Colorado project was able to move so quickly was because she had long-term, well-established relationships with these organizations. Kwan also reiterated Buchanan’s point that researchers should return data to the community in a form it can use for its own purposes.

Kwan, in responding to an audience request to discuss resources that describe how to establish and moderate community engagement studios, replied that the website dicemethods.org, run by the University of Colorado School of Medicine, has a list of community engagement methods. The list includes fact sheets and strategy guides for implementation of community engagement studios⁴ and other engagement activities, both in person and virtually. She added that usually community engagement studios consist of two 90- to 120-minute sessions, held either in person or virtually. A community navigator and academic expert serve as moderators. The studio begins with a brief expert presentation describing the evidence that the project team will disseminate and the issues researchers are seeking to understand. That is followed by a discussion based on a series of two to three key codesign questions that will inform that development process.

Hughes asked Glanz to discuss possible approaches for engaging organizations in an accessible and sustainable manner to help them move toward evidence-based practice. Glanz replied that involving organizations that participate in the research phase of a project in the dissemination and implementation process is one approach that she has found effective. She added that a member of the community that embraces the process and acts as a local champion is also helpful. Glanz echoed Buchanan’s comments, noting that building strong, trusting relationships is also necessary for developing a program that meets the community’s needs and expectations. She added that this will also increase the likelihood that the organization will implement evidence-based practices that in turn will generate practice-based evidence.

Considerations for Research Funders

An audience member asked the panelists to comment on what role they see funders such as AHRQ playing in ensuring that the evaluation of dissemination and implementation projects involves community and patient stakeholders. Glanz replied that when her team was writing the proposal for a Patient-Centered Outcomes Research Institute (PCORI) Engagement Award, the application included rules that defined the amount of money that was required to be allocated to the community. She said that this may not be the most effective approach because it may limit the resources that the project can provide. She added that it is also important to consider fairness in the way a project team allocates resources among community partners and to avoid assuming that every community or organization has the same resource needs.

___________________

⁴ Additional information is available at https://dicemethods.org/CENR (accessed September 14, 2022).

Kwan suggested that research funding agencies allow projects to use grant funds to purchase food for participants during community engagement activities. Some agencies have specific language in grant contracts barring the use of funds to purchase food for participants. She explained that while this seems like a minor consideration, it could improve the participant experience. She also suggested that applications should require inclusion of specific plans for active dissemination of research products back to the participants and the broader community in a form they can understand and use. Buchanan noted that monetary compensation is important. He suggested that a person that shares their time and data with a research project should receive more than food as reimbursement. He added that it is important to listen to community members in a manner that conveys that the project team values their contributions to the project and prioritizes making that data useful for the community. Kwan agreed that paying those who engage in planning and conducting research is critically important.

This page intentionally left blank.