10

Workshop 3, Session 2: Measuring the Impact of Dissemination Projects Part 2

GENERALIZABILITY AND TEMPORALITY IN ASSESSING EFFECT

Alisa Stephens-Shields, associate professor of biostatistics at the University of Pennsylvania Perelman School of Medicine, discussed generalizability and

temporality in assessing the effect of dissemination and implementation projects. Generalizability reflects the extent to which study findings apply to target populations. This requires minimizing discrepancies between study samples and target populations. Enrolling non-representative trial sample populations that do not represent the full range of individuals in the target population can diminish external validity. This can lead to overestimating or underestimating what the effects of the dissemination and implementation project will be in the target population.

She then discussed the Randomized Evaluation of Trial Acceptance by Incentive (RETAIN) trial that she conducted with colleagues at the University of Pennsylvania. This trial, explained Stephens-Shields, evaluated financial incentives as a means of recruiting sufficient participants to increase representativeness and generate generalizable trial results. It also explored the ethics of using incentives to increase enrollment. The study investigated whether financial incentives lead to “undue inducement” that disproportionately increased enrollment among people from systematically divested communities. She added that undue inducement is an incentive that blunts the perceived risk of participating in the trial or an incentive that is unjust.

Graphic representation of data indicating undue inducement is characterized by a difference in the slope of the relationship of perceived riskiness and the probability of enrollment according to the size of the incentive (Figure 10-1). She explained that when there is no incentive, people will be less likely to enroll in a trial if they believe it is risky. However, if paying the participants increases the probability of enrollment for the same amount of perceived risk, this reflects undue inducement. Similarly, a large difference in the slope of enrollment versus incentive size according to economic status is reflective of unjust inducement (Figure 10-2).

Stephens-Shields then discussed the Behavioral Economics to Transform Trial Enrollment Representativeness (BETTER) project, an initiative funded by the American Heart Association to test behavioral economics interventions. The study seeks to determine whether behavioral economics interventions are an effective mechanism to surmount the barriers to randomized controlled trial participation faced by people from disenfranchised racial and ethnic groups, women, people with low incomes, or people with a medical risk of cardiovascular disease. A key target outcome for BETTER is achieving an increase in the enrollment fraction. This represents the number of individuals enrolled out of the number of individuals from different subgroups who are contacted to participate in a trial. The primary outcome for BETTER’s second component will be the enrollment fraction of Black and Latino participants. She noted that the study has identified them as important subgroups that are disproportionately affected by adverse cardiovascular health outcomes but are less likely to enroll in cardiovascular health clinical trials. A secondary out-

come will examine the enrollment fraction overall stratified by socioeconomic status. The project will also measure the population to prevalence ratio—the enrollment fraction of specific subgroups divided by the overall enrollment fraction—to determine whether the intervention produces more representative samples. This measure will also enable researchers to determine the true

population effect in the target populations and examine heterogeneity among different socioeconomic and racial groups.

Stephens-Shields and her collaborators are assessing the sequential effect of an intervention using the mediation framework to examine temporality (Baron and Kenny, 1986). This framework involves measuring the direct effect of an intervention on an outcome versus the outcome when the intervention goes through a mediator. The mediation framework proved important for determining why an intervention did not work in a trial that aimed to increase condom use by African American men who had sex with men (Jemmott et al., 2015). That study found two theoretical constructs that were strongly associated with condom usage that the intervention did not affect: a negotiation skill and peer descriptive norms. As a result, the study investigators designed future interventions to target those two constructs.

In closing, Stephens-Shields provided some questions that these studies highlight related to assessing effect. Those include the following:

• How well do the data used to evaluate the effect reflect the population in which we want to apply conclusions, and how can we minimize relevant gaps?
• Are thresholds and benchmarks used in evaluating the effect meaningful?
• Have we generated sufficient longitudinal data to assess sequences of effect and provide insight into the mechanism of effect or lack thereof?

CLINICIAN ENGAGEMENT WITH A BREAST RECONSTRUCTION DECISION SUPPORT TOOL

Mary Politi, professor of surgery at Washington University in St. Louis School of Medicine, discussed clinician engagement with decision support tools using a breast reconstruction digital support tool project she collaborated on as an example. The project involved a diverse group of stakeholders and researchers from Washington University in St. Louis and the Ohio State University. These included decision scientists; plastic and reconstructive surgeons; an infectious disease expert who studied complications from breast reconstruction; and individuals with expertise in clinical informatics, data analysis, data management, and research administration. The project’s stakeholder advisory group included patient partners from varying regions of the country; surgeons from varying regions of the country; and informatics specialists who helped with the logistics of incorporating the tool into the electronic health record (EHR). Both groups worked to guide the tool development and implementation process.

Politi and her colleagues piloted the first version of a breast reconstruction decision support tool in an earlier randomized trial. They compared the deci-

sion tool to enhanced usual care and found that patients in the intervention group were more knowledgeable about the risks and benefits of the different options related to breast reconstruction (Politi et al., 2020). They also found that patients also had a slightly higher, though not statistically significant, decision process quality; patients reported increased certainty about their choice; and patients were more actively engaged in their care. An important finding from this study was that the intervention (the decision support tool) had no effect on consultation length. That indicated that the decision support tool did not affect clinical workflows, an important factor to the participating clinicians. However, after the study, clinicians reported that at times they were unsure whether a patient had used the tool, which affected their engagement with the patient. Politi explained that the clinicians wanted to use the tool to engage in shared decision making, which became the goal of a subsequent implementation project.

Politi and her collaborators relied on the Capability, Opportunity, and Motivation Model of Behavior (COM-B model) as a framework for understanding behavior change as they began considering their approach to facilitating clinician engagement with the decision support tool. They identified four required elements for clinicians to change their behavior (engage with the tool):

1. the capability to use the tool, which requires skills and training;
2. motivation to use the tool, which requires believing it will help the clinical encounter and may also involve incentivizing use;
3. opportunities to use the tool, which requires resources and possibly integrating the tool into the EHR; and
4. social norms, such as the perception that their peers are using the tool and finding it beneficial.

Politi also described another implementation framework that she has found to be effective: Normalization Process Theory (Hooker et al., 2015; Hooker and Taft, 2016; Toye, 2016). This process identifies four important components of implementation:

1. coherence, or defining the work, its benefits, and how it will affect the clinician’s practice;
2. cognitive participation, or who does the work and what are the teams involved in implementation;
3. collective action, or how the work gets done and what is needed for the intervention to become part of routine care; and
4. reflexive monitoring, or how the clinician understands the work and what the effect will be in terms of measured outcomes.

Politi then described the BREASTChoice breast reconstruction decision support tool and the process of developing it through the lens of those models. She began with their approach to addressing coherence. Patients initially receive access to the web-based breast reconstruction decision tool, which describes the different options for breast reconstruction, via the patient portal for their physician’s EHR or email. The tool includes educational modules that address whether to have breast reconstruction; types of reconstruction procedures; timing and risk for each type of procedure; and reconstruction outcome photos.

Politi explained that in terms of coherence and cognitive participation, there was minimal work required on the part of clinicians to use the tool at her institution. Politi noted that all of the pertinent clinicians at both institutions completed the training and were supportive of the tool. The decision support tool was integrated into their EHR. Clinicians could access the tool’s summary by clicking an icon below a patient’s name. At Ohio State the process was less streamlined. Clinicians were required to first click a button in the EHR to agree to add the tool and summary, followed by clicking another to allow the data to be imported into the patient chart in the EHR.

Next, Politi addressed the cognitive participation and collective action component. The researchers were required to send the enrollment link for the decision support tool to patients who consented via the MyChart patient portal or email. Schedulers from the clinician’s office could also send flyers to eligible patients that explained they would be contacted by a member of the clinical team prior to using the tool. Surgical oncologists received flyers that they could provide to their patients prior to a referral to a plastic and reconstructive surgeon.

Politi next elaborated on their application of the COM-B model. The clinicians were provided training to develop skills for using the decision support tool. The research team found that some clinicians lacked motivation to use the tool. However, the tool was automated, so clinicians were required to do little work to use it. She noted that the COVID-19 pandemic prevented the research staff from being in the clinic to observe workflow during usability testing. One surprising finding was that many clinicians were comfortable with the paper printout, which the research team did not expect given that it was a digital tool.

Politi described some of the challenges the researchers encountered related to physician engagement with the decision-making tool. While many clinicians were accustomed to viewing information in the EHR, they were less acclimated to doing so in the context of shared decision making with a patient. Some clinicians were comfortable sending information to patients for them to review. However, the researchers had to put forth substantial effort to move the clinicians to use that information to support collaborative decision mak-

ing with the patient. Another challenge to clinician engagement was finding solutions to address EHR alert fatigue. Politi noted that further research is needed to ascertain optimal design for the tool, including how best to provide alerts. Other subjects for future research include determining how to integrate clinician components into patient-facing tools; investigating the role of EHR restrictions, institution policies, and culture in the effectiveness of the decision support tool; how to build on existing workflows and engage clinical champions; and investigating whether integrating engagement with decision support tools into resident training would increase clinician engagement.

INCORPORATING PCOR INTO CLINICAL PRACTICE: A DIGITAL TECHNOLOGY CASE STUDY

Krisda Chaiyachati, physician lead for value-based care and innovation at Verily Health Platforms and adjunct senior fellow at the University of Pennsylvania’s Leonard Davis Institute of Health Economics, began by remarking that he is pleased to observe the growing emphasis on accelerating how social science, behavioral science, and health services research are making it into clinical practice and touching patients’ lives. He also acknowledged that this will be a challenging process. Chaiyachati said that there are innumerable ways in which digital technology interacts with the health care system (Figure 10-3), illustrated by the many digital tools already in use or in development (Abernethy et al., 2022). This has resulted in a large variety of stakeholders in the digital transformation of health and health care that are affected by decisions related to dissemination and implementation. This includes small technology startups, large technology firms, academia, health care delivery organizations, regulators, patients, and communities. He echoed Buchanan, noting that the growth of digital technologies in health care has implications for equity and inclusivity.

Chaiyachati next recounted a story from his perspective about the beginning of the COVID-19 pandemic. In March 9, 2020, he and a colleague were in Philadelphia watching news coverage of COVID-19 as it began to spread globally. At the time, there were reports of alarming conditions in northern Italy, cruise ships carrying people that had been infected with COVID-19 were not allowed to dock, and intensive care units (ICUs) were beginning to overflow in Seattle and New York City. While hospitals in Philadelphia had not yet been overwhelmed by patients with COVID-19, staff had become concerned owing to the increasing number of scared people calling or coming to the Penn Health system’s emergency departments. This prompted the formation of a large team at Penn Health to cultivate strategies to address the challenge of identifying people infected with SARS-CoV-2 and which patients required hospitalization.

The team at Penn was seeking to decrease the likelihood of their emergency departments being overwhelmed by developing a mechanism to ensure patients with serious illness were brought to the emergency department and those that did not require that level of care did not present to the emergency department. Chaiyachati explained that was the first design consideration for developing a useful digital tool. The second consideration was that COVID-19 was presenting with a variety of symptoms, but the associated respiratory failure and COVID pneumonia appeared to be resulting in fatalities. Chaiyachati and his colleagues designed a digital tool that could identify patients who were developing shortness of breath because that had been identified as a symptom associated with poor disease outcome. The result was a text message–based program called COVID Watch (Morgan et al., 2020). COVID Watch sent text message check-ins twice daily, in either English or Spanish, that asked individuals about their shortness of breath and if they found breathing more difficult than usual. The system would automatically connect the individual, depending on their responses, to a team of telemedicine providers who were available 24 hours a day.

Chaiyachati noted that the team engaged in substantial observation and experimentation that occurred prior to determining the final design of the program (Morgan et al., 2020). Chaiyachati ran a randomized controlled trial testing whether supplementing patient responses with objective pulse oximetry improved outcomes. However, he found that this approach did not affect outcomes (Lee et al., 2022). The design team also interviewed patients regarding their experience using the program. The team continued to modify COVID Watch as they received feedback. Chaiyachati noted an informal study conducted a comparison of COVID Watch with a similar program at Northwestern University. The Northwestern University program relied on manual enrollment and was staffed by registered nurses between 8 a.m. and 8 p.m. The study found that COVID Watch saved Penn Medicine about $2.3 million every 100 days. That benefit accrued because COVID Watch was automated and thus only required 64 staff hours per day compared with 500 hours required for the Northwestern program. Patient satisfaction for COVID Watch had a 90 net promotor score, which Chaiyachati said is comparable to the score that Apple products receive. The program also achieved a relative reduction of mortality of 64 percent (Delgado et al., 2021). In addition, COVID Watch was able to manage patient volumes efficiently, differentiate patients who required evaluation in the emergency department, and effectively monitor those individuals that did not require evaluation in the emergency department.

Chaiyachati noted that more than 30 individuals were involved in creating COVID Watch. This highlighted the complexity of designing a robust program that integrates with existing technology, engages the clinical team, and enables the research evaluation needed to generate supporting evidence. He explained that there is a need to find balance between the quality and the rigor needed to generate evidence and the ability to scale and spread innovations such as this or create parallel models in other disease conditions. Chaiyachati said that in the future, digital health tools should be designed with the capability to account for a patient’s entire life span, the variety of physical and emotional conditions that can affect them, and the effect of social determinants of health. He closed by summarizing insights he gained during the process of developing COVID Watch. He discussed lessons the team learned during the COVID Watch project. He said that digital health tools can enhance human-powered care. The efficiency of care provided with the support of digital health tools affects the degree to which patients and providers are receptive to using the tool. It is important to ensure equity and define the measures of equity early in the intervention design process. He encouraged funders to support the design stage and not focus solely on outcomes because the design of digital health interventions is critical for successful implementation and uptake.

DISCUSSION

Behavioral Economics

Session moderator Sarah Scholle, vice president for research and analysis at the National Committee for Quality Assurance (NCQA), posed an audience question to Stephens-Shields that asked her to discuss how the principles of behavioral economics can help improve implementation, dissemination, and sustained use of evidence-based interventions. Stephens-Shields explained that behavioral economics is an important component of effective dissemination and implementation because frequently evidence of effectiveness is not an adequate stimulus for people to change or adopt a new behavior. Behavioral economics provides a useful tool to inform and adjust implementation and dissemination efforts to frame outcomes as rewards versus relying on loss aversion as a source of motivation. She noted that she is just starting a new program that will use nudging techniques, a standard approach in behavioral economics, in a telemedicine setting to encourage patients to follow up on care, such as completing recommended screening tests following identification of an irregularity during a clinical marker.

Chaiyachati agreed that behavioral economics can offer critical insights into how to use people’s innate, often irrational behaviors to move them toward their desired health goals. He suggested that there are opportunities to blend behavioral health economics with social science, implementation science, statistics, and other fields to accelerate the process of translating research discoveries into clinical practices. Behavioral economics principles can be tested with small changes during the preparation phase; such tests can be done quickly in a small trial, which can reduce delays during the later stages of determining a final design. Chaiyachati said that it would be economically advantageous to fund this newer approach of rapid testing and iteration and integrate it into randomized controlled trials. He suggested that this approach could support innovation by allowing academic researchers to feel less constrained when designing a new tool. He added that industry already uses this approach in its implementation and dissemination research. Stephens-Shields commented that being adaptive during the research process implies changing as more information accrues while still following established principles.

When asked how this approach would allow interventions to address the needs of different patients and populations, Stephens-Shields said there are enrichment designs for clinical trials that allow some interim preliminary evaluation of whether an intervention is more promising in certain subgroups than others. This allows the trial to pivot enrollment strategies to target those promising groups. Chaiyachati added that different communities have different preferences for the manner in which they interact with the health

care system and those preferences change over time. He noted that while research might examine individual behaviors and preferences among specific subgroups, it can create operational challenges if an iterative process results in creating multiple different profiles and multiple different operational programs for different types of people. He added that additional research is needed to determine how to do this effectively.

Considering Metrics

Scholle asked the panelists to discuss potential metrics that would provide Agency for Healthcare Research and Quality (AHRQ) insights into optimal strategies for engaging different populations in research as well as the scalability of various interventions. Stephens-Shields replied that some of the metrics her team is using in the BETTER project relate to the engagement of different demographic groups in research. These include enrollment fraction and population prevalence ratios that allow for gauging the representativeness of a sample. She added that universal application of these types of metrics would be useful for gaining an understanding of the research landscape versus the landscape for a specific project.

Chaiyachati said metrics that are applied across patient populations and across delivery organizations would provide information about equity and the inequities that might occur with an intervention. This is not something that AHRQ has explicitly asked research communities to consider when an intervention targets a population that has traditionally had challenges accessing care. He suggested that another type of metric for AHRQ to consider would examine the spread of an implemented intervention beyond the primary institution or from clinic to clinic within the same health care system. This would assess the ability of an intervention to spread and scale to neighboring health care systems. This would be particularly germane given the challenges that arise from a lack of interoperable data systems, both within a health care system and between different health care systems.

Considering Stakeholders

Scholle asked the panelists to discuss their ideas for how AHRQ might consider the perspectives of different stakeholders and different resources, whether it is data or workflow, as a means of improving scalability and broader spread of interventions. Chaiyachati began by noting that the amount of data that could be used in health care is growing rapidly and substantially. He expressed concern that health system data infrastructure may not be able to process and manage all of that data. Additionally, some health care systems may not be able to bear the cost of processing and managing this large volume

of data. He noted this could be particularly challenging for those practices or health care systems that are not part of large academic medical centers. He suggested that funders should consider whether they could support health care systems to address these challenges.

Scholle asked the group to discuss possible strategies to address alert fatigue that may interfere with the effectiveness of nudges or other tools designed to draw a clinician’s attention. Stephens-Shields said researchers can build measures for responsiveness into an intervention trial to identify where and when diminishing returns occur related to frequency and extent of nudging. She added that researchers should be conscious of the potential for alert fatigue and incorporate constructs to evaluate and address it into intervention trials. Chaiyachati then suggested incorporating de-implementation science¹ to determine when it is appropriate to turn off certain alerts and which alerts are most critical to retain. He added this is particularly critical to the ICU setting. He then offered an example from his experience as a general internist. At times when he initially opens a patient chart in the EHR, he encounters 15–20 alerts or reminders for tasks to complete during that clinical encounter. He suggested further research into approaches to optimize context and timing for those alerts. He also suggested that an EHR alert for a clinician may not be the optimal solution. Another approach could be to develop a system that delivers this information to patients, encouraging them to take a more active role in their health care.

Resource Considerations

Stephens-Shields also responded to Scholle’s question about generalizability and what would be necessary to implement an intervention in settings with different resources. Stephens-Shields highlighted the importance of inclusive research and recruitment strategies that include health care systems in different settings with different levels of infrastructure and other resources. This strategy will assist developers to gain an understanding of the limitations on potential effects in certain settings and identify additional supports certain systems might need to implement an intervention and achieve the desired outcomes. She also suggested that AHRQ should consider investing in the infrastructure that would allow health care systems to manage and process large volumes of data.

Chaiyachati added that artificial intelligence and machine learning (AI/-

___________________

¹ De-implementation science identifies problem areas of low-value and wasteful practice, carries out rigorous scientific examination of the factors that initiate and maintain such behaviors, and then employs evidence-based interventions to cease these practices (Davidson et al., 2017).

ML) technology could support efforts to manage and analyze large amounts of data. He also emphasized the need to ensure that AI/ML do not perpetuate biases. Stephens-Shields echoed Chaiyachati regarding the need to address the issue of automated algorithms and equity.

Speakers’ Closing Thoughts

To conclude the discussion, Scholle asked the panelists for any final thoughts regarding the most important things that would be beneficial for AHRQ to consider related to evaluating the effectiveness of its implementation and dissemination projects. Chaiyachati replied that from his perspective, there is a need to move beyond individual tests and the practice level and consider scalability and spread across multiple health systems. One approach would be to dedicate funding to specifically support scalability research and development of infrastructure that would better enable scalability and spread. Stephens-Shields said that while there are many different dimensions by which to measure and evaluate dissemination effects, inclusiveness is a key dimension. She also suggested evaluating implementation and dissemination projects across multiple stages.

CLOSING SUMMARY OF WORKSHOP 3

Lauren Hughes concluded the workshop by summarizing her takeaways from the presentations and discussions.

She described several suggestions made by speakers for incorporating community engagement into effective dissemination and implementation projects:²

• Engaging with a variety of community stakeholders to gain novel insight when conducting dissemination and implementation research.
• Involving community members early in the dissemination and implementation project planning process, and continuing engagement throughout the course of the project.
• Acknowledging the contributions of communities who have shared valuable data and perspectives by providing them with results in accessible forms so that they can use the findings to apply for funding themselves.

___________________

² These points were made by the individual workshop speakers/participants identified above. They are not intended to reflect a consensus among workshop participants.

Hughes also described speaker suggestions for developing effective dissemination and implementation projects. These included³

• Developing strategies to address the need for infrastructure that will enable researchers and clinicians to access to the rapidly growing amount of health care;
• Supporting dissemination of research results beyond presentations at academic conferences;
• Leveraging behavioral economic insights to support dissemination and implementation efforts;
• Considering the use of digital health tools and technologies to enhance health equity, personalize health care, and increase the efficiency of care;
• Examining how rigid structures and workflows may impede progress; and
• Learning from how other industries innovate.

___________________

³ These points were made by the individual workshop speakers/participants identified above. They are not intended to reflect a consensus among workshop participants.