8

Developing Networks and Sharing Information

The workshop’s penultimate session featured two presentations on building networks and sharing information. The two speakers were Gregory Simon, a senior investigator at Kaiser Permanente Washington Health Research Institute and a principal investigator with the Mental Health Research Network; and Linnea Ashley, the training and advocacy director for Youth ALIVE! and the managing director of the National Network of Hospital-Based Violence Intervention Programs (NNHVIP). An open discussion moderated by George Isham followed the two presentations.

BUILDING A LEARNING HEALTH CARE SYSTEM FOR SUICIDE PREVENTION

The Mental Health Research Network, Simon explained, comprises 13 large, integrated health care systems and their affiliated or embedded research centers which together serve some 14 million Americans. Aside from providing comprehensive care, these integrated health systems also provide insurance coverage, and all have longitudinal electronic health record (EHR) and claims data harmonized into compatible data sources. Simon said that the presence of strong partnerships between the delivery systems and health plan leaders in each of these health systems is probably the most important asset to the network’s members.

Work on suicide prevention, Simon said, began less than a decade ago in direct response to a question from the network’s health system partners. At the time, the network’s health systems were engaged in an effort to improve mental health care using systematic outcome measures, one of

which was a score on the Patient Health Questionnaire-9 (PHQ-9) depression questionnaire. What providers wanted to know was if their patients were at risk of suicide attempt or death if they answered in the affirmative that they had frequent thoughts of death or self-harm. The answer then was that nobody knew the answer, so the research network decided to address that question, and the answer was that those patients who reported having thoughts of death or self-harm nearly every day were 8- to 10-fold more likely to either attempt suicide or die by suicide over the year following the visit when they had completed that questionnaire. The data collected showed that some 6 percent of patients reported they had thoughts of death or self-harm nearly every day or more than half of the days and that this 6 percent of the patients accounted for 39 percent of the suicide deaths.

This led to three sets of responses, Simon said. First, the health system partners rapidly implemented a standard-of-care process because they did not want to wait for further evidence before acting. The process consisted of providers administering the Columbia Suicide Severity Rating Scale instrument and, if the score indicated a significant risk, the care team creating a safety plan addressing lethal means, including a routine assessment of access to firearms, and recording that in the EHR. This process, implemented in 2014, was supported by provider training and tools embedded in the EHR. By 2015, health systems were monitoring provider performance and using it as part of physician performance-based compensation. Simon said that in the past 3 months of 2015, he scored at only the 83 percent level, which would have led him having his pay docked by $233 in that particular quarter. “So although I am not proud of my personal report card, and I will tell you it has improved since that time,” he said, “I am very proud of our health system in terms of a 22-month gap between publication of findings and implementation to the stage where the first author of the publication is going to get his pay docked if he does not improve his performance.”

Today Kaiser Permanente, along with the Henry Ford Health System in Detroit and HealthPartners in Minnesota, is leading a national implementation effort following the National Action Alliance for Suicide Prevention’s zero suicide model (Brodsky et al., 2018; Labouliere et al., 2018). The steps of this model, Simon explained, include identifying people at risk, engaging people in ongoing care, providing evidence-based or empirically supported interventions, and attending to transitions between the sectors of care, such as hospital to outpatient or from emergency department to outpatient follow-up. He said that the network is “fairly far along” in developing specific metrics to assess the desired changes in care process as well as both intermediate and ultimate outcomes. These metrics will be monitored in six health systems serving some 9 million members over the next several years and will be used to provide feedback that allows individual providers,

clinic managers, and researchers to assess the specific effectiveness of different improvement programs. The metrics will also make for relatively transparent comparisons among the performance of each of the health systems over time.

The second response involves the network looking at the potential benefit of more intensive programs or more intensive outreach programs to people who become disconnected from care. These programs would not be trivial to implement in terms of cost and potential intrusiveness, Simon said, and thus they would require more evidence to support wide-spread implementation. To get that evidence, he said, the network is undertaking a large trial in four of its health systems that assigns patients who completed a PHQ-9 and reported frequent thoughts of death or self-harm to a usual care control arm or one of two population-based interventions. Both interventions involve outreach to supplement, not replace, usual care, and the programs are delivered primarily by online messaging and telephone, which means they could be scalable and affordable should the data support their effectiveness. One of the interventions emphasizes training and dialectical behavioral health therapy skills, while the other is a more traditional care management program aimed at keeping people engaged in mental health care. Enrollment and randomization of 18,900 people across the 4 health systems is complete, Simon said, and the intervention will continue until the middle of 2019, with findings expected in late 2019 or early 2020.

The third response of the network’s suicide prevention research efforts has focused on developing a machine learning method to predict suicidal behavior that is better than the PHQ-9, which produces a 7-fold risk concentration and misses the 35 percent of the people who commit suicide but do not report having thoughts of death or self-harm. This does not include those individuals who are missed because they never complete the PHQ-9. This project, Simon said, involved mining EHR data from seven of the network’s members.

Simon explained that the research team developed separate models for people who visited specialty mental health providers and for people who visited general, medical, or primary care providers but had a known mental health condition. The team linked those models to data from records about nonfatal suicide attempts within 90 days and state mortality records about subsequent suicide deaths. From past mental health and medical diagnoses and service use patterns this analysis defined approximately 150 potential clinical predictors and 200 possible interactions among them.

The model to predict suicide death following a mental health specialty visit uses 62 predictors and has an overall accuracy of approximately 86 percent, while the model to predict suicide death after a primary care visit uses 43 predictors and has an overall accuracy of approximately 83 percent.

Similar predictors were selected for nonfatal suicide attempts, Simon said. He added that this approach accomplished a 10-fold risk concentration—that is, it identified a set of individuals who made up 5 percent of the group of patients in the study but who accounted for 48 percent of the suicide deaths—and missed fewer events in the bottom 75 percent of people, who accounted for less than 20 percent of the suicide deaths.

These finding have initiated another cycle of implementation and effectiveness research and looking for new methods, Simon said. “These risk prediction models will be used to augment the previous standard work processes,” he explained, with Kaiser Permanente in Washington and Northern California adding these risk scores to the EHR. The models will also be used to do outreach between visits to people who either cancel a visit or fail to show up for a visit using the ability to recalculate risk scores daily on every member in its service population. Simon noted that the technical tools to develop and validate the models are in a public GitHub repository.¹

Going forward, Simon said, the network is planning a large trial to determine if one of the new glutamate receptor modulator ketamine-like drugs, which have been shown to rapidly decrease suicide ideation, actually reduce the risk of suicide attempts and suicide deaths as well. The network is also harvesting another round of data to improve the prediction models, is adding more predictors, and is using more detailed encoding about timing. It will also extend its risk prediction work to develop models for people seen in emergency departments or people discharged from hospital, and it will look at linking EHR data to location-based data and financial data because those are known to be important predictors of suicidal behavior.

In closing, Simon offered some lessons learned in terms of building and operating a research network in the world of integrated health care systems that are responsible for defined populations and that routinely link membership data to data on injury and poisoning mortality, which, he observed, is the only way for a health system to hold itself accountable. Simon said that data harmonization is critical for rapid turnaround for data analysis and for systematically assessing data quality. Highly skilled data analysts are probably the most important resource, he said, while building reusable data tools eliminates building new models from scratch. “Finally,” he said, “the most important key ingredient is culture. We always are engaging with health system leaders who are committed to systematic improvement and see systematic measurement and accountability as core to their mission,” and in that respect, the importance of transparency and trust cannot be

___________________

¹ See github.com/MHResearchNetwork for more information (accessed December 20, 2018).

overstated. “It was not easy to get to the point of health systems saying we are going to come together and transparently share information about how many people die by suicide and whether we did the right thing by them before they died,” he continued, “but we are at that point now, and that is incredibly valuable in terms of quick learning.”

As a final comment, Simon noted how deaths from lightning strikes have plummeted since the 1940s, but not because humans figured out a way to eliminate thunderstorms. “What we did was we got better at predicting risk and developed effective public health messages,” he said. “Taking that as an example, how do we identify time, places, and situations of high risk [for suicide], and how do we develop effective messages for people at risk to avoid dangerous things and go to safe places? Those are fairly simple messages.”

VIOLENCE INTERVENTION AND COLLABORATION

Building networks and collaboration is important for both research and the dissemination of best practices, Ashley told the workshop. Network composition influences the type of research that is being done, the information being included, and the collaborators at the table. In addition, networking and collaboration open doors to new approaches to addressing entrenched problems such as gun violence and injury. “As long as we are working in silos,” she said, “we are going to continue to see problems the same way that we have always seen them and see solutions the same way, and if we want to get innovative, if we want to do new things, we are going to have to make friends.”

Hospital-based violence intervention programs (HVIPs), which by design require collaboration among different members of a health care team, are at their core efforts to promote positive alternatives to violence. HVIPs are not alone, as there are other organizations, such as law enforcement, that have similar goals, although they take a different approach to confronting violence. Given that each organizations’ methods may differ, it is important when forming networks with these other organizations to keep in mind the common goal when inevitable differences of opinion or even personality conflicts among collaborators arise, Ashley said.

Because communities affected by firearm violence are at the forefront of the problem, it is imperative those communities be involved in any network or collaboration, she said. In fact, Ashley urged that members of those communities be involved in the next workshop convened on this subject. “When we are talking about solutions, when we are talking about the problems, it is imperative that they be in the room, and that they are able to talk about their own experiences, and when we come up with potential solutions, that they are on board or that they are able to bring their own solutions,”

Ashley said. Nonprofits and community organizations are also important collaborators, she added, “because those are the groups that have been doing the work before funding was available and will continue to do the work if everybody else’s attention pivots to another problem.”

When looking for collaborators, universities are good places to start, she said, because they have students who are eager to be involved in research designed to address gun violence and injury and who will then join the workforce with connections already formed to people and organizations outside of the university. Other places from which to identify potential collaborators include national organizations such as the Brady Campaign and a variety of local community and state organizations. Ashley encouraged people to look for some level of overlap in a potential collaborator who on the surface may not appear to share the goal of addressing firearm violence.

Partnerships can be flexible, Ashley said, but they do have certain ground rules. It is important for all partners to know their roles in the collaboration, stay in their own lanes, learn to trust one another, and learn from one another. While the overarching goal of all collaborators will be to reduce gun violence, she said, other goals can be involved, such as wanting to reduce community tension and improve community safety and dialogue or a desire to expand research or expand the reach of a small program.

Partnerships, Ashley said, change perceptions because they help break down silos. “When you partner with somebody else and look at the work that they are doing, it can expand your idea of what is both possible and also where your work might fit in with somebody else’s work,” she said. She added that partnerships succeed when the collaborators trust one another, so it is important to include trust-building activities into a collaboration. For example, the annual Health Justice Alliance Conference now includes a pre-conference fun day that allows potential collaborators to interact with one another in a way that encourages and nurtures relationships built on trust and connection.

In building a partnership it is important to consider the barriers to working relationships between systems. For example, Ashley said, Youth ALIVE!’s Caught in the Crossfire program, an HVIP, works with the police but is mindful of the way in which it works with law enforcement because some clients may not react well to frontline staff who appear to be too closely tied to law enforcement. “We have to set clear boundaries so that everybody knows what information is being shared,” she said. At the same time, working with the police provides a number of advantages, such as having easy access to police reports having to do with clients. “You just have to think about the benefits and boundaries and be respectful on both sides,” she said.

In the HVIP world, networks can streamline research by creating large pools of subjects from the many small HVIP programs that exist. Networks

also create natural opportunities for sharing the results of experiments, spreading best practices, and benefitting from experience that other programs have gained in their communities, Ashley said. Conferences and working groups associated with networks can be valuable places for getting exposed to new ideas, making new contacts, sharing best practices, learning about challenges that other programs face, and expanding the reach of successful programs. For example, she said, the best practices that have been discussed at the national conference of the Healing Justice Alliance include treating violence as a public health issue, the value of HVIPs, the need to involve prevention professionals, the importance of listening to communities, and the role that trauma-informed care and mental health services can play in preventing gun violence. Other best practices that Ashley listed include not dismissing practice-based evidence generated by groups working in the community that may not hit the gold standard of research, particularly regarding culturally specific practices; implementing both screening and interventions, because just screening for the problem is not helpful; and making research applicable to the community.

In closing, Ashley briefly discussed a tool that Youth ALIVE! developed called the Screening and Tool for Awareness and Relief of Trauma, a six-question screen that pulled from other available screens. Based on how people answer the questions, she said, there are four brief interventions they can receive in addition to a psychoeducation piece on the common symptoms of trauma. The interventions include a breathing exercise for relaxation, hand massage for relaxation and grounding, a sleep hygiene awareness exercise, and a safety plan. The goal for having these brief interventions, Ashley said, was to be able to deliver something immediately and not merely refer an individual to a later session that the person may or may not attend.

DISCUSSION

When asked to describe their respective networks, Simon answered by saying that the Mental Health Research Network is not tied together by a defining passion for any one particular issue. In fact, he said, the network did nothing on suicide prevention until health system leaders asked the network to address that problem some 6 or 7 years ago. In contrast, Ashley described the National Network of HVIPs as including a broad range of people with overlapping interest in a single overriding issue—violence prevention and intervention—although that does not prevent the network from working on related issues.

Isham asked Simon if the Mental Health Research Network would be able to disseminate research findings and solutions, and Simon replied, yes, it could, with some work. Given the research emphasis of this group, he

said, it would have to be sure the quality of the data was good enough to disseminate a finding widely. He did say that the network is starting to build relationships with leaders in health systems who are responsible for service delivery, in addition to its relationships with leaders of mental health and primary care specialties.

On the topic of “translating research into practice,” Simon said he finds that phrase an abomination that should never be spoken again. Research must live in practice and take orders from practice, he said. “Research must know that its bosses are practitioners and the people those practitioners serve. If that happens, I do not think that much translation is necessary.” Ashley said she agrees with that idea in theory, but that there is a big gap between what happens in academic research and what happens on the ground. She noted that the NNHVIP is set up so that researchers and frontline workers are in close proximity to each other in real time in order to avoid looking for theoretical answers to theoretical questions that do not have any immediate bearing on the challenge of reducing violence in the community.

Isham then asked the two speakers if networks make it easier to use machine learning to extract useful information from large datasets. Simon replied that predicting suicide death, which is a relatively rare event, would require large datasets to make good predictions. Predicting suicide attempts would require a smaller dataset, and predicting suicidal ideation an even smaller dataset. However, scale does not always translate into generalizability because predictions generated in one place may not generalize to another. For the work that he and his colleagues are doing, he said, the more important factor about being part of a network is that the data come from multiple health systems spread across the United States, so the sample set is diverse in terms of socioeconomic status, race, and ethnicity. Ashley agreed that generalizability is an issue because every community is unique. What is generalizable, she said, is the threshold level of violence in a particular community that makes having an HVIP beneficial and, for those concerned about money, also cost-efficient.

Therese Richmond noted that Simon’s group has an advantage in that Kaiser Permanente, as both an insurer and provider, has a captive member population, so even if a Kaiser member is treated outside of the system, it still gets information back to add to a member’s EHR. Simon agreed and said that every health care system should have a link to mortality records, as that is the worst possible outcome. He also commented that he was unsure if the prediction methods he spoke about could be extended to prediction of being a victim of either accidental firearm injury or firearm injury by assault. “Those things may be more unpredictable, but that is an empirical question that we could certainly answer with the data, and we plan to do that,” Simon said. On the other hand, he continued, being a perpetrator

of violence may be more predictable than being a victim, although Kaiser does not have access to data showing who among its members have been perpetrators of violence. While there is no technical barrier to linking its membership data to criminal justice behavior, Simon said, the sensitivity of doing so is high, and today Kaiser is in a deliberative process of discussions with its members about when it is all right to use that kind of data. Ashley added that it is important to be mindful of the fact that perpetrators and victims often change places, in that people who experience violence can go on to inflict violence on others. This dynamic can affect how people are treated within the hospital system and whether they will continue accessing services, she said.

Isham then asked Simon and Ashley if they knew what percentage of Americans their networks covered. Ashley could not say, but she noted that the American Hospital Association joined the network in the past year, so the network now covers a much larger portion of the population than before. Now, more hospital systems are aware that there are approaches for addressing violence in the community, and as a result, Ashley said, she receives a large number of calls from hospitals and community groups wanting to establish a full-fledged violence prevention program. Simon said that his network covers approximately 4 percent of America, which he said is “big enough to do a lot of fancy math, but 4 percent of America is not big enough to change health care.” This latter realization has the Mental Health Research Network considering how to collaborate more with people and organizations that are not just like Kaiser Permanente, he said.

An unidentified participant asked if either of the two networks work with clergy or faith-based organizations, given that these are typically among the most trusted members of the community. Ashley said that some of the network’s individual programs have established relationships with clergy. One non-member effort in San Diego, for example, was using clergy as its initial contact point with individuals, but there is not a formal arrangement between the national network and any particular faith-based organization.

In response to a question about the use of “prevention professionals,” Kyle Fischer explained that the idea behind creating that title was to provide a means for health systems to recognize an intervention specialist who comes from the community and who does not fit into any existing category in the current medical model. “The long and short of it is, public health does not pay, so if we are going to try and pay for these services, it is going to have to be through the traditional medical model,” Fischer said. The national network, Ashley added, is currently refining a 35-hour certification curriculum for prevention specialists.

Fischer said that while there may be similarities between prevention professionals and community health workers—and in some states, com-

munity health workers and prevention professionals could be the same person—he and his collaborators decided to develop a new classification out of frustration with the fact that every state has a different definition for community health workers and different rules on how to pay them. His position is that if a person lives in a state that pays for community health workers, then that person should be a community health worker and get trained in the core competencies for prevention. Otherwise, the prevention professional pathway can serve as a new pathway for payment. Ashley reiterated the importance of getting the specific training needed to be able to serve as a prevention professional even without that formal title.

When asked how their networks are funded, Simon said that his network has been funded for about 7 years by the National Institute of Mental Health (NIMH). He said that the infrastructure budget for the network across 13 sites is just under $1 million per year, including what are known as indirect costs. The coordinating center’s annual budget is a few hundred thousand dollars, and each participating site receives a sustaining budget of approximately $60,000 per year. Research projects receive separate funding through individual research grants that network members submit. The case that Simon can now make to NIMH is that the network’s established infrastructure is able to conduct less expensive large clinical trials. For example, a pragmatic trial of interventions to prevent suicide attempts enrolled 19,000 people at a per person cost of under $400, which is two orders of magnitude less than that of a typical NIMH-funded trial of a similar size and nature. “We can do big randomized trials at a cost of about 1 percent of what you are used to paying, and we are proud of being cheap,” Simon said.

Ashley said that her network gets some funding from Kaiser for her position, but the majority of the network functions on volunteer hours and with money from membership dues. She said that the network has received some federal funding in the past and expects to receive some federal funding in the year ahead, but for the most part the network has been held together by volunteers and organizations who believe in its mission.