Workshop Overview¹

The National Strategy for Suicide Prevention, first released in 2001 and updated in collaboration with the National Action Alliance for Suicide Prevention in 2012, outlined goals and objectives that are meant to work together in a synergistic way to prevent suicide in the United States. Some of these goals call for the use of innovative data science approaches to identify individuals, populations, and communities at high risk for suicide, which is the topic for this workshop. Specifically, goal 4 calls for the promotion of responsible media reporting of suicide, accurate portrayals of suicide and mental illnesses in the entertainment industry, and the safety of online content related to suicide. Also there is a goal to increase the timeliness and usefulness of national surveillance systems relevant to suicide prevention and to improve the ability to collect, analyze, and use this information for action (goal 11) (U.S. Department of Health and Human Services Office of the Surgeon General and National Action Alliance for Suicide Prevention, 2012). Box 1 contains the 13 goals of the National Strategy.

___________________

¹ This workshop was organized by an independent planning committee whose role was limited to identification of topics and speakers. This Proceedings of a Workshop was prepared by the rapporteurs as a factual summary of the presentations and discussions that took place at the workshop. Statements, recommendations, and opinions expressed are those of individual presenters and participants and are not endorsed or verified by the National Academies of Sciences, Engineering, and Medicine, and they should not be construed as reflecting any group consensus.

Emerging real-time data sources, together with innovative data science techniques and methods—including artificial intelligence (AI)/machine learning (ML)—can help inform upstream suicide prevention efforts (CDC, 2021). Nonclinical, natural-language processing data from social media platforms, and data from wearable devices, are currently being leveraged by technology companies as part of their suicide prevention efforts to identify individual users who are at high risk for suicide. These “social suicide” prediction algorithms have particular relevance to individuals who are reluctant to engage the health care system, since timely access to relevant education and health care resources may encourage help seeking during a crisis. De-identified social media discussions on suicide and suicide-related behavior are also being analyzed to determine potential hotspots that can help target public health resources to communities at high risk. For example, suicide risk is higher in Indigenous populations.²

Although innovative, real-time data sources, including social media data, and algorithms that predict suicide and nonfatal suicidal behavior can potentially enhance state and local capacity to track, monitor, and intervene “upstream,” these innovations may also be associated with unintended consequences and risks, such as potential bias in AI algorithms that could contribute to health disparities. Also, utilizing these technologies could paradoxically increase the risk of harm in some cases, especially if the technologies are not validated. Thus, the ability to leverage real-time data sources and novel methodologies to identify and mitigate suicide risk needs to be balanced with minimizing unintended consequences.

This challenge was the impetus for the recent Workshop on Innovative Data Science Approaches to Assess Suicide Risk in Individuals, Populations, and Communities: Current Practices, Opportunities. Hosted by the Forum on Mental Health and Substance Use Disorders at the National Academies of Sciences, Engineering, and Medicine, the virtual workshop consisted of three webinars held on April 28, May 12, and June 30, 2022. The main goal of the workshop was to explore the current scope of activities, benefits, and risks of leveraging innovative data science techniques to help inform upstream suicide prevention at the individual and population levels. Addressing that goal will require careful thought about how best to frame the issue and then exploring what data are available that can be used to assess at-risk individuals and determine the best intervention. To that end, the workshop organizers invited speakers with diverse backgrounds and expertise, including in areas such as

___________________

² This topic is covered in another National Academies of Sciences, Engineering, and Medicine report published in 2022: Suicide prevention in indigenous communities. See https://www.nationalacademies.org/our-work/suicide-prevention-in-indigenous-communities-a-workshop (accessed August 29, 2022).

mental health and suicide prevention, AI/ML, big data, social media algorithms, natural language processing, app development, public policy, ethics, and public communication.

This Proceedings of a Workshop summarizes the presentations and discussions that took place during the three webinars that composed the workshop. The presentations and discussions are organized thematically, rather than strictly chronologically, to allow for similar points made by different participants to be synthesized and streamlined. The proceedings highlights individual participants’ suggestions to help advance innovative data science techniques including AI/ML learning to help inform upstream suicide prevention efforts at the individual, community, and population levels. The suggestions are discussed throughout the proceedings and summarized in Box 2. Appendix A includes the Statement of Task for the workshop. The webinar agendas are provided in Appendix B. Speaker presentations and the workshop webcast have been archived online.³

Benjamin Miller, president of Well Being Trust, said in his opening remarks that social media “has become a major way that we interact with the world and those around us.” Miller said that people at risk of suicide often turn to social media platforms for help. In response, a number of these platforms have proactively deployed sophisticated AI/ML algorithms to identify individual users who are at high risk for suicide, and in some cases these platforms may, if they deem it necessary, activate local law enforcement to prevent imminent death by suicide. He added, this is one approach to taking advantage of social media information to detect at-risk individuals and to get them the help they need, but there are many more approaches in place, in planning, or under consideration—and, quite certainly, many more approaches that have not yet been imagined.

U.S. Surgeon General, Vice Admiral Vivek Murthy, spoke about the toll of suicide and the importance of finding improved ways to identify at-risk individuals and to intervene to prevent suicide. In January 2021 the U.S. Department of Health and Human Services (HHS) and the Office of the Surgeon General, in collaboration with the National Action Alliance for Suicide Prevention, released The Surgeon General’s Call to Action to Implement the National Strategy for Suicide Prevention.⁴ The report outlines actions that communities and individuals can take to reduce the rates of suicide and to help improve resilience. Also in 2021, the Surgeon General issued an advi-

___________________

³ See https://www.nationalacademies.org/our-work/using-innovative-data-science-approaches-to-identify-individuals-populations-and-communities-at-high-risk-for-suicide-a-workshop (accessed August 29, 2022).

⁴ See https://www.hhs.gov/sites/default/files/sprc-call-to-action.pdf (accessed August 19, 2022).

sory on Protecting Youth Mental Health.⁵ The advisory lays out the policy, institutional, and individual changes needed to address the mental health crisis among young people, which has been exacerbated by the COVID-19

___________________

⁵ See https://www.hhs.gov/sites/default/files/surgeon-general-youth-mental-health-advisory.pdf (accessed August 19, 2022).

pandemic. Murthy said that the COVID-19 pandemic has increased feelings of depression, anxiety, and loneliness for many, and that “using data science to assess suicide risk is both important and increasingly urgent.”

Suicide Detection and Prediction

Gregory Simon, senior investigator at the Kaiser Permanente Washington Health Research Institute, described four types of actions that might be performed by data science tools applied to some of the new types of data that are becoming available. The first of such actions would be to use inference to create generalizable knowledge. For example, the tools might be applied to discover that students who are subject to online bullying are at high risk for suicidal behavior. A second action would be the detection of hotspots for a community-level intervention; one might, for instance, find that students in a particular high school are at high risk for suicidal behavior. Third, data science tools could be used for detection at the individual level—say, of a student who is at the present moment experiencing suicidal ideation or a mental health crisis. Finally, the tools might be used for individual-level prediction—for example, finding that a specific student is more likely to attempt suicide in the coming month. In laying out the four possibilities, Simon emphasized that it is important to keep in mind the difference between detection and prediction: detection refers to uncovering something that is happening right now, such as someone at immediate risk of suicide, whereas prediction refers to foreseeing something that is likely to happen in the future. “It’s going to be important for us when we talk about these tools to say which of these jobs are we speaking of,” he concluded.

What can social media data bring to the table in terms of helping at-risk individuals and lowering the country’s growing suicide rates? Several speakers addressed that question with context-setting presentations that provided overviews of the issue and suggested ways to proceed.

Artificial Intelligence–Based Suicide Prediction

Mason Marks, the Florida Bar Health Law Section Professor of Law at Florida State University and senior fellow at the Petrie-Flom Center for Health Law Policy, Biotechnology, and Bioethics at Harvard Law School, described two types of AI-based suicide prediction approaches—medical suicide prediction and social suicide prediction. He said that suicide rates in the United States have been steadily increasing over the past 20 years (Figure 1⁶), “so it’s

___________________

⁶ See https://www.cdc.gov/nchs/data/databriefs/db398-H.pdf (accessed September 19, 2022) for more up to date data.

clear that we are not doing as good a job as we could be or should be to reduce suicide rates.” Marks said that part of the problem is that historically, physicians have not been very good at predicting who will attempt suicide. Also, the methods available to physicians and others to predict suicide are often little or no more accurate than chance. That is one of the reasons why the use of AI is exciting to people in this area, he said.

Marks explained that medical suicide prediction is largely used for research at this stage. Moreover, it is typically undertaken by hospitals or health care systems, and it involves physicians and physician researchers or other medical scientists who have access to patient health records. This access allows them to know which patients attempted or died by suicide. Researchers can then train machine learning algorithms on those records to identify patterns and pick out words or phrases, including diagnoses or medications, that are more commonly associated with suicide than other words or phrases. They can then deploy the algorithms to find those data points in new patient cohorts and predict suicide risk in those populations. Because medical suicide prediction occurs within the health care system, it is typically subject to the Privacy Rule⁷ promulgated under the Health Insurance Portability and Accountability Act (HIPAA) and other regulations associated with clinical care and medical research. This kind of research is typically reviewed by an institutional review

___________________

⁷ See https://www.hhs.gov/hipaa/for-professionals/privacy/index.html (accessed August 22, 2022).

board to provide ethical oversight, and the results are often published in peer-reviewed medical journals, which helps contribute to the work’s scientific rigor and transparency.

By contrast, Marks said that social suicide prediction is usually undertaken by corporations, and instead of being used primarily for research, social suicide prediction will often trigger real interventions, such as sending the police to a person’s home for a wellness check. An example is Crisis Text Line,⁸ a company focused on young adults and adolescents that relies primarily on texting. Some companies, such as Gaggle and GoGuardian, focus on students and market their services to schools and school districts. Because these companies do not have access to medical records, they typically cannot train their algorithms using actual suicide data, so they end up using proxies for suicide data, particularly information pulled from social media and data produced when consumers use internet-enabled services. Companies analyze this information to make inferences about consumers’ health and other characteristics. Because these analyses are performed outside the health care system, they are not subject to the HIPAA Privacy Rule.

In addition, Marks said the use of social suicide prediction has generally not been reviewed by institutional review boards, and although companies might do some sort of internal review, there is generally a lack of transparency regarding their use of predictive algorithms. Furthermore, the algorithms are usually proprietary and companies may treat them as trade secrets. Perhaps most concerning, interventions triggered by the algorithms, such as wellness checks, carry a certain risk of harm to the individuals reported. Individuals may be hospitalized or otherwise detained against their will, or they may be medicated without their consent. Sometimes a violent confrontation with the police results, and in several cases, police carrying out wellness checks have shot and killed the people they were summoned to help. Confrontations with first responders can actually increase the risk of suicide, which means that these systems can sometimes do more harm than good, Marks said.

Medical suicide prediction carries certain risks as well—such as the possibility of stigmatizing someone who has contemplated suicide—but the risks are more concerning for social suicide prediction, Marks said, due to the lack of transparency and the fact that less is known about social suicide prediction methods (Barnett and Torous, 2019; Marks, 2019).

Among the risks that are associated with both types of suicide prediction is the possible damage caused by inaccurate predictions. Both types of systems produce a large number of false negatives and false positives, Marks said. A false negative can leave a suicidal person without support, but false positives can also be quite dangerous. For example, if the fact that a person

___________________

⁸ See https://www.crisistextline.org/ (accessed August 22, 2022).

has been flagged by a social suicide prediction algorithm is leaked or shared, that person could be denied access to insurance, employment, or housing and may never know the reason why. People could also have their civil liberties violated—such as being held against their will—because of a false-positive suicide prediction.

Marks suggested several ways to improve AI-based suicide prediction methods. First, AI-based suicide prediction should be viewed as part of a larger system. For instance, if there are no safe and effective interventions that can be triggered once suicide risk is identified, the prediction algorithm should not be used until the treatment shortcomings are addressed. In the case of social suicide prediction, he said consumer consent should be sought and individuals should be allowed to opt in or opt out of the prediction systems employed by social media companies. Furthermore, he added that the prediction systems should be linked primarily to “soft touch” interventions rather than “firm hand” interventions—e.g., referring someone to mental health services instead of calling the police for a wellness check. Also, Marks said that social suicide prediction systems should be limited to research settings until there is greater transparency and more evidence to support their effectiveness. He concluded that first responders should be better trained in how to respond to mental health emergencies.

The Promise of Social Media Data

Nick Allen, a clinical psychologist and professor at the University of Oregon and the co-founder of Ksana Health, spoke about the differences between industry and academic settings in terms of innovation and knowledge generation and the implications of those differences for suicide prediction and prevention. Allen said that when thinking about digital technology and data science methods, it is essential to know that traditional research methods are greatly outpaced by changes in consumer behavior. He mentioned that a typical R01 grant funded by the National Institutes of Health generally takes about 7 years, maybe longer, from the conception of the project to its completion, but in the real world things tends to move much faster. There are many examples of such rapid technological changes over the past few decades, he said, pointing in particular to the adoption of mobile cellular and broadband technologies. He added that the use of technology services can change even more rapidly than the use of devices. “If you were doing research on social media use using traditional methods and traditional approaches,” he said, “you really would have a challenge adapting to this change.”

With these changes in consumer technology use come changes in human behavior. Allen shared data to show the changes in the ways in which adolescents prefer to communicate (Figure 2) (Rideout and Robb, 2018). From

2012 to 2018 the percentage of 13- to 17-year-olds who said their favorite way to communicate with friends was in person dropped from 49 percent to 32 percent; the percentage of those identifying texting as the preferred communication method stayed about the same, while the percentages of those who named social media and video chatting grew sharply (7 to 16 percent and 2 to 10 percent, respectively). Allen emphasized that “we’ve got a change that’s occurring at a pace that’s really hard for traditional research methods to model.”

Meanwhile, over the past couple of decades there have also been major changes in the mental health landscape—particularly increases in mental health challenges among young people—including an increase in the suicide rate. Allen said that “this presents us with a situation where we clearly need to come up with models that adapt to these rapidly changing landscapes, both in terms of the epidemiology of mental health difficulties as well as the affordances of consumer behavior and consumer devices.”

Against this backdrop, Allen said, it is important to understand the strengths and weaknesses of traditional health research approaches versus business innovation approaches. The traditional health research approach emphasizes internal validity and, excluding alternative explanations, has a slow funding and research cycle, assumes a static environment in terms of the risk factors, and uses rigorous measures and methods. By contrast, the busi-

ness innovation model has an emphasis on external validity (“If it sells, it’s working”), involves rapid iteration based on user-centered design processes (one of its goals is to “Fail fast”), and assumes a dynamic environment created by business competition and technological innovation. One weakness of the business innovation model is that there can be confusion between a product’s market fit and the actual health benefit or effectiveness of the product. People may assume that because a product sells, it must do what it is supposed to do, and that is not always the case. An advantage of this approach is that it focuses on feasibility, engagement, and usability, so a product that fails on one of those criteria will generally be abandoned quickly. Allen stressed that the two different approaches have some complementary strengths and weaknesses and it is essential to find ways to leverage those complementary strengths to find solutions more quickly.

Data of various types are central to a number of technology business models, such as subscription models (paying for access to data) and contextual and behavioral advertising (both of which use data to provide targeted advertisements). Allen said that in pursuing these core business models, many technology businesses collect large amounts of the types of data that could be used for suicide predictions. He added that there is both a business case and potentially a health use case for these same data. He said that in mobile computing, smartphones are used to collect data on a whole range of different kinds of behavior—including language, patterns of usage, circadian rhythms, autonomic physiology, geographic movement, acoustic voice quality, facial expressions, social interactions, physical activity, and sleep patterns—that research has shown are linked to mental health.

Turning to the issue of preventing suicide, Allen said that one of the most effective methods for preventing suicide is to intervene at moments of highest risk. He suggested two ways in which this could be done. The first is by reducing access to means—for example, by storing firearms and ammunition in a way that makes them more difficult to retrieve or by putting pills into blister packs rather than having them in bottles. The second is by providing support and diversion at high-risk moments with what are called just-in-time adaptive interventions (JITAIs). A JITAI is an intervention design that aims to provide support at a critical time point by adapting to the dynamics of an individual’s internal state and context, which is measured continuously (Nahum-Shani et al., 2018).

Allen offered a vision for the future of digitally enhanced mental health care, noting that it is critical to build systems that are flexible and allow users to get the right level of care and support for the different levels of needs they have in a seamless way. Mobile sensing, smart nudges, and communication tools could all play a major role in turning that vision into a reality, he concluded.

Social Media and Population-Level Digital Mental Health

Munmun De Choudhury, associate professor at the School of Interactive Computing at The Georgia Institute of Technology, spoke about efforts to proactively assess a person’s or a population’s risk of developing mental health challenges. De Choudhury noted that mental health disorders affect millions of people worldwide and are one of the leading causes of disability and death. She said that current approaches to predicting and preventing suicide often rely on clinical interviews, patient self-reports, family observations, and assessment scales, but these approaches are limited by the fact that they do not allow for frequent monitoring of risk for suicidality. The data are not collected frequently enough to monitor for important changes in people. Also, many people suffering from mental illness do not have easy access to health care or social services (Coombs et al., 2021). Furthermore, many mental health challenges are characterized by such things as negative perspectives, self-focused attention, a loss of self-worth and self-esteem, and social disengagement (Bravo et al., 2018; Wilburn and Smith, 2005). It is in filling these voids that social media can play a particularly important role, De Choudhury said.

One of the advantages of applying social media to predicting suicide is that large and increasing percentages of youth and young adult populations are online (Figure 3) and using various social media applications. For younger people in particular, these applications are very much a part of their lives.

De Choudhury shared the findings of a study that explored the potential to use social media to detect and diagnose major depressive disorder in individuals (De Choudhury et al., 2013). The team recruited 476 people (243 male,

233 female) through Amazon’s Mechanical Turk, a crowdsourcing interface. The researchers asked participants various questions, including about diagnoses of clinical major depression, and rated them on two standardized scales of depression, the Center for Epidemiological Studies Depression Scale⁹ and the Beck Depression Inventory.¹⁰ After getting consent to access the participants’ Twitter feeds, the researchers measured various aspects of those feeds related to their engagement, their emotions, and their linguistic styles. They created a predictive classifier that could determine which individuals might be at the highest risk, up to a year in advance of a reported onset of major depression.

With that classifier the team then developed a social media index that compared the frequency of standard Twitter posts with Twitter posts that were indicative of depression. The researchers compared the results of that index with actual reported measures of depression across the United States on a state-by-state basis and found a relatively good agreement (a correlation of 0.52 according to a “least squares” regression fit). Thus, this index makes it possible to assess the rate of depression not only across various geographies such as cities and states but also across demographics such as sex and also across time. De Choudhury said that an advantage of this approach is that it makes it possible to make such assessments more efficiently and more frequently. Also, the approach could practically help determine which areas in the country have a shortage of mental health providers relative to the estimated need.

De Choudhury also described a study aimed at predicting when an individual will start talking about suicide in an online forum (De Choudhury et al., 2016). Working with conversations on Reddit, the researchers focused on individuals speaking about and seeking support for mental illnesses and suicidal behavior. Looking backward at the conversations over time, the researchers looked for various attributes in past conversations that might indicate a transition to talking about suicide. Their analyses found that various linguistic cues were associated with either an increased or decreased risk of suicidal ideation in the future. For instance, the transition to suicidal ideation was found to be associated with heightened attentional focus, poor linguistic coherence and coordination with the community, reduced social engagement, signs of hopelessness, and anxiety, among other factors. These cues were shown to have predictive power in identifying which individuals would go on to engage in suicide ideation discussions in the future.

De Choudhury cautioned that not everybody is on social media, and thus social media needs to be thought of as part of a broader data ecology and to be applied in the context of other conventional data sources. She emphasized that

___________________

⁹ See https://cesd-r.com/ (accessed August 22, 2022).

¹⁰ See https://www.apa.org/pi/about/publications/caregivers/practice-settings/assessment/tools/beck-depression (accessed August 23, 2022).

social media can be used to augment existing and conventional data sources in the context of suicide prevention. She described a recent study where researchers developed ML models to estimate weekly suicide fatalities in the United States (Choi et al., 2020). The researchers combined suicide-relevant data streams from social media or the web with health services data streams from clinical sources, such as data on emergency department visits, and also micro-economic data. “Our idea was to harness the strengths of each of these data sources using ML, and then take those individualized predictions of weekly suicide fatalities to combine them intelligently with a single composite score,” De Choudhury said. With this model the researchers made forecasts of weekly suicide death counts, and these predictions were significantly better than those made using models that rely on only one or two data sources. The value of such an ability is that it can provide real-time numbers for suicide rates as opposed to the currently available suicide statistics, which are typically 1 to 2 years behind.

De Choudhury posed three questions for more research, including: (1) What evidence is needed to show that these sorts of prediction approaches are ready for real-world use? (2) How can models be supported in those inevitable cases of failure, given that ML algorithms are never going to be perfect? (3) What can be done to address the questions of social justice that will arise with the use of social media–based suicide prediction and to make sure that the new approaches do not exacerbate existing inequities?

Working with Social Media Data to Predict Suicide Risk at the Individual and Population Levels

Kenton White, co-founder and chief data scientist at Advanced Symbolics, an AI company in Ottawa, Canada, said that suicide, unlike many other health conditions, is almost entirely preventable yet many lives are lost every year by death from suicide. He listed three reasons for this. First, there is a social stigma associated with suicide, so many at-risk people do not want to talk about it. Second, it is difficult to screen for at-risk people who are not already getting help from a mental health professional. Third, the resources for addressing suicide are very limited. He said, for example, “anyone that has tried to get mental health help during COVID-19 realizes that there’s a shortage of mental health professionals right now, and there’s a waiting list for people who want to get help.”

White said that a major problem is finding where to put limited resources so that help can be provided where it is needed. Social media might offer a solution, he added, saying that “we have got websites where people are talking about what they are feeling, what they are doing, in real time, and we could be using such online platforms to help with screening and get the resources where they are needed.” (Online screening is discussed in Box 3.)

White said that many social media companies are willing to help. However, they are concerned about the legal ramifications of providing information on individuals. For this reason, Advanced Symbolics is looking to apply a scientific approach to help at a population level. White said that one thing that sets the work of Advanced Symbolics apart from other work with social media data is that it starts with a statistical sample, which makes its research more like medical research, where samples are generally representative of the population. Furthermore, it is looking beyond individual posts to get infor-

mation on an individual’s entire past history to provide a historical context to what is happening in the present.

White said that the Advanced Symbolics team is collaborating with clinicians to complement the insights from social media. When the team’s algorithm identifies someone who might be depressed or experiencing a precursor for self-harm, Advanced Symbolics shares the information with their clinician partners. The clinician partners review the entire history and inform the Advanced Symbolics team if they are seeing any clinical patterns that could indicate risk of self-harm. White said that from this work, the team has learned that they should not only be looking for keywords in people’s social media posts but also pay attention to posting patterns. For example, insomnia is one sign that a person may be dealing with mental health issues, and if an individual begins making more posts late at night, it could be a sign of increased risk. Other things they look for are people disconnecting from their social networks and communities without any explanation or friends asking whether they are okay. White added that other factors that are predictive of self-harm include a history of substance misuse or physical abuse or trauma. He noted that all this information could be obtained from the entire history of a person, not just a single post.

White said that the idea behind the work is to construct a sample that is nationwide and includes not just social media posts but also as much additional information and history as possible to create a predictive index (Vogel, 2018). He said that when his team did this in 2016–2017, it had success with showing correlations between the index and increased suicide rates, and it was slated to begin a program with the Public Health Agency of Canada.¹¹ The goal was to move beyond the current situation of waiting a year or two for data on suicide rates and to be able to identify suicide hotspots before they happen. White noted that the nationwide project unfortunately did not launch because of high concerns about the potential misuse of data, such as the Facebook–Cambridge Analytica data scandal in which a data analytics company harvested data from millions of Facebook users to influence America voters.¹²

White said that his team has learned valuable lessons from the experience, however. The first is that any such research really has to be done “from a position of trust.” With that in mind, the team is now working with Indigenous communities in northern Canada, through partners who have established trust with the communities by working with them over many decades on mental

___________________

¹¹ See https://www.mobihealthnews.com/content/canada-will-use-ai-monitor-suicidal-social-media-posts (accessed August 29, 2022).

¹² See https://www.theguardian.com/news/2018/mar/17/cambridge-analytica-facebook-influence-us-election (accessed August 19, 2022).

health issues. He said that at this point, however, the study is not far enough along to have generated results.

Ethical and Pragmatic Suggestions to Improve the Ability to Identify Individuals and Populations at Risk for Suicide

Glen Coppersmith, chief data officer at SonderMind, offered what he referred to as “ethical and pragmatic” suggestions for improving the ability to identify individuals, groups, communities, and populations at risk for suicide. Coppersmith said that technology companies have access to meaningful data around suicide risk that could be useful for entities trying to prevent suicide. He urged technology companies—working individually or in consortia—to make aggregated data feeds accessible to trusted parties. He underscored the importance of technology companies working with these trusted parties to offer training resources to the caregivers of those at risk of suicide, noting that these companies have insights into the people who are at risk and what might help their caregivers have greater success in caring for them.

Coppersmith shared a figure to illustrate the sort of data that social media companies have available to them (Figure 4) (Coppersmith, 2022). The hashes on the top represent one person’s interaction with the health care system over the course of many years, with each hash indicating an encounter with a clinician or an emergency room. These interactions are traditionally where health care providers get information about the individual’s mental health and well-being. By contrast, the hashes along the bottom represent all of the times the individual posted on a single social media site. “There’s a lot of information that is being generated in the time in between the health care visits,” Coppersmith emphasized. Coppersmith said his team has built ML algorithms that can take social media data and make predictions about suicide risk or other mental health issues in between those interactions with the health care system—“filling in the gaps,” as he termed it.

Coppersmith suggested three guiding assumptions when working with social media data to predict suicide risk. The first is that technology companies have—or could have—the ability to identify at least some of the people at risk for suicide based on their behavior. Second, technology companies also have the ability to group users into meaningful cohorts, such as veterans, health care workers, or geographic area. Third, however, the organizations that have the information about suicide risk—that is, the technology companies—are not the organizations with the capabilities to prevent suicide. “They’re not health care organizations,” he said. He added that many of these companies can only act within the constraints of their brand and legal departments.

Coppersmith said that it is helpful to think of suicide-related information in terms of three spectrums. The first spectrum is the timeline before a mental

health crisis develops. At one end of that spectrum is information from the moment of crisis itself, while on the other end is stable situation, such as the social determinants of health. Where the information falls on the spectrum will influence how it can be used and what sort of interventions it can inform. The second spectrum is the level of aggregation, from a single individual on one end to all humans on the other, with groups of individuals in the middle. Where information falls on that spectrum will inform the particular risk. The third spectrum is the fidelity of the information. On one end of that spectrum is information that is fully calibrated—with meaningful data points that are important and correct in some way that will allow an action to be taken. On the other end of that spectrum is directional information that has some value but cannot be acted upon by itself.

Most of the interventions that people think about in terms of how social media data might be used to prevent suicide involve information at the far end of these spectrums: individualized data from the moment of crisis that is precise and highly calibrated. This is the sort of ideal data a team of mental health professionals could use to intervene in a moment of crisis. However, Coppersmith said, attaining such information is not very pragmatic for a host of reasons, in large part because technology companies are unwilling to release such sensitive information. This raises the question of what technology companies can do with this information, such as aggregating it and adjusting it in various ways, so that they can hand off the modified information to mental health professionals for use in suicide prediction and prevention. Coppersmith suggested that one approach could be by grouping data from a number of individuals to make it possible to address many of the privacy concerns while still providing useful information. This is the approach taken by the U.S. Census, for example.

In a similar vein, during the COVID-19 pandemic Coppersmith and colleagues tracked social media postings from a cohort of users that represents

the general U.S. population and another cohort that represents health care professionals, and used a natural-language processing algorithm to calculate the estimated suicide risk for the two populations (Fine et al., 2020). “This gave us a chance to draw some conclusions about how the COVID-19 pandemic, as it was unfolding, was affecting mental health professionals, and gave us some leading indicators to some problems that we have all now seen play forward,” he said.

A combination of willing partners in the technology industry and willing public entities and health care entities could work together to make such things happen, he said. There are many nonprofit groups and public sector groups that would work to prevent suicides among certain groups at certain times if only they had the information about when and where the risks actually are occurring. Furthermore, there are plenty of researchers who, if they had access to aggregated data of the correct sort, would find ways to inform policy decisions and resource allocations. In conclusion, Coppersmith urged technology companies that are interested in preventing suicide to make aggregated data available to trusted parties that take care of people at risk of suicide.

Learning from Search Engine Data

Elad Yom-Tov, senior principal researcher at Microsoft Research, discussed ways that search engine data can be used to study issues in health and medicine and also to nudge people to better health practices. Yom-Tov said that search engines such as Bing, which Yom-Tov’s team works with, or Google can provide information and insights that almost no other data source can provide. He offered several examples of how such data can be useful at the population level.

Yom-Tov said, first, search engine data can provide temporal resolution of individual behaviors. People ask questions on Bing many times a day—sometimes 20, 30, or even 50—and although each individual is anonymous, it is possible to follow an individual’s search engine queries over time and know with high probability that all the queries came from one person. A second advantage of search engines is the truthfulness of the supplied data. Because a search engine is an anonymous platform, people do not have to identify themselves as they do for other social media platforms. Yom-Tov said for example, that Google Trends¹³ can be used to analyze the popularity of the top search queries in Google Search across various regions and languages. The third advantage of search engines is that they can provide rich data—because they are able to collect a lot of data that is typically not obvious. A fourth advantage

___________________

¹³ Google Trends is a tool for showing the volume of searches on Google across time and geography. See https://trends.google.com/trends/?geo=US (accessed August 19, 2022).

of search engine data is access—because the vast majority of internet users use search engines, it is possible to get data covering most of the population, including people from all different segments of society.

Yom-Tov offered a suicide-related example of what could be done with the data. The Werther Effect refers to the observation that when there is a publication about a high-profile suicide, it increases the risk that other people will decide to kill themselves as well (Yom-Tov and Fischer, 2017). This has been documented at country levels, but Yom-Tov observed that search engine data would allow for a much finer resolution. He shared data on a study to assess the effect of news stories on the intentions of internet users. The researchers analyzed search engine data to determine which phrases and words were more likely to cause an increase in suicides and which ones would decrease it. The result was a model that can predict whether a user report about a suicide will increase or reduce the number of suicides or not affect it at all.

Another study looked at suicides in India, which are often stigmatized and so likely to go unreported (Adler et al., 2019). The researchers compared official government statistics on suicide in the different states of India with search data on people asking about suicide methods. The researchers found a correlation between the two sets of numbers, but in some states the official suicide numbers were significantly less than would be expected from the search engine data. The implication is that in these states there is underreporting of suicides because of the stigma.

Moving from the population level to the individual level, Yom-Tov said that when an individual searches for information on suicide, Bing will put a banner at the top of the search results with a link to a support help line. The particular banner depends upon keywords in the search and on the user’s location. Beyond suicides in some cases, it is possible to identify particular medical issues that the person may have—such as cancer or a mental health crisis—and provide helpful links relating to those as well. Using mouse-tracking data, the researchers were able to find that when people who had asked questions about suicide were presented with a page of results including a banner with a help link at the top, they spent much of their time looking at the help line number. This is encouraging, Yom-Tov said, but it is still not clear from these data whether presenting these individuals with such a banner actually changes their behavior.

Yom-Tov said that it is difficult to conduct experiments that test the effectiveness of such banners in changing behavior because of both regulatory and ethical reasons. However, it is possible to analyze “natural experiments” that shed light on the issue. Moreover, there are several possibilities, he said, including training advertisement systems to offer advertisements for helpful sites to people who are contemplating suicide. Such targeting systems can be trained to be very specific so it is possible they could offer help to people

well before someone is at the point of no return and looking for ways to kill themselves. Again, Yom-Tom cautioned that a variety of ethical issues arise in this sort of approach (Yom-Tov and Cherlow, 2020).

CURRENT OFFLINE APPROACHES TO PREDICTING AND PREVENTING SUICIDE

Several speakers spoke about current offline approaches to suicide prevention based on data that can be collected from tools including electronic health records, hospital records, and demographic databases.

Best Practices from the Department of Veterans Affairs

Lisa Brenner, professor of physical medicine and rehabilitation, psychiatry, and neurology at the University of Colorado, Anschutz Medical Campus, and the director of the Department of Veterans Affairs Rocky Mountain Mental Illness Research, Education, and Clinical Center, presented an overview of the Department of Veterans Affairs (VA)/Department of Defense (DoD) Clinical Practice Guidelines¹⁴ for the assessment and management of patients at risk for suicide. Brenner said that the viewpoints expressed in her presentation do not necessarily reflect the views of the VA or DoD. Again, she cautioned not to consider the VA/DoD Clinical Practice Guidelines as the final word because much more needs to be done in the context of clinical care in terms of identifying those at risk of suicide and specifying how best to intervene.

Brenner said that the most recent version of the VA/DoD Clinical Practice Guidelines, released in 2019, offers 22 evidence-based recommendations. Five of those recommendations are related to screening and evaluation for suicide risk. She said that when evaluating patients, it is recommended that risk factors be assessed as part of a comprehensive evaluation of suicide risk, including such things as current suicidal ideation, previous suicide attempts and psychiatric hospitalizations, current psychiatric conditions and symptoms, and the availability of firearms.

Brenner said that VA clinicians apply a two-stage screening process. First, they use the Columbia Suicide Severity Rating Scale¹⁵ to detect who may be at risk for suicide and is in need of further evaluation. Second, they use the

___________________

¹⁴ See https://www.healthquality.va.gov/guidelines/mh/srb/ (accessed August 23, 2022).

¹⁵ See https://www.hrsa.gov/behavioral-health/columbia-suicide-severity-rating-scale-c-ssrs (accessed September 16, 2022).

VA Comprehensive Suicide Risk Evaluation¹⁶ to inform clinical impressions about acute and chronic risk and the patient’s disposition. Brenner noted that the process is implemented in a clinical setting in which false positives or negatives can be addressed. “We would not only depend on the screener. We would have lots more information based on the way the patient is presenting, based upon their history, perhaps based upon what their family members are reporting, and we would be able to use that in addition to the screener and then be able to complete the evaluation,” she added.

There are a number of potential weaknesses to this approach, she said. For instance, some individuals may be uncomfortable communicating emotional distress to health care providers or even to friends or family. Some of them may be more comfortable communicating about these issues online. Also, health care providers may not see a patient often enough to ask about suicide at the most important time. Furthermore, health care providers may not feel comfortable asking about suicide.

Brenner highlighted several observations on the use of social media algorithms to predict suicide risk based on her VA experiences. She said that there are differences between how different cohorts use social media, noting that it is essential to incorporate those differences into social media algorithms. Also, research has found that individuals at different geographic locations post differently about suicidal behavior; therefore, social media algorithms should take geographic location into account (Morese et al., 2022). It would also be important to discover how various factors—age, sex, race, ethnicity, cultural background, primary language, and so on—affect what people post and influence the algorithms used to assess risk. She also reiterated the legal and ethical issues, and wider implications concerning suicide risk detection and interventions in social media platforms (Celedonia et al., 2021). Intervening based on limited data, for example, raises a variety of both ethical and legal concerns. Consent is another important issue, she added. How much do people on social media platforms understand about how their data may be used? Should those data be shared with family members or care providers? How should an individual’s privacy and autonomy be taken into account when considering an intervention, and how can patient preferences best be considered?

Brenner mentioned a number of knowledge gaps that affect the VA/DoD Clinical Practice Guidelines and that likely also apply to the use of social media algorithms for suicide prediction and prevention. One is the issue of identifying acute versus chronic risk. Another is determining whether risk stratification is actually reliable and valid in assessing risk. Concerning evalu-

___________________

¹⁶ See https://www.mirecc.va.gov/visn19/cpg/recs/3/resources/VA-CSRE-Printable-Worksheet.docx (accessed September 16, 2022).

ation, there are questions about the extent to which screening and evaluation should be connected and about the most appropriate settings in which to do evaluations. These are all challenging questions, she concluded.

Suicide Prediction and Prevention in the Military Health System

Dan Evatt, who leads a research team at the Psychological Health Center of Excellence, provided an overview of identification and management of suicide risk in the Military Health System (MHS) and discussed analytic and practical consideration related to implementing suicide analytic screening in the MHS. Evatt said that in 2020 the suicide mortality rate was 28.7 deaths per 100,000 active service members. Evatt said that in the MHS, suicide risk can be identified either by a health care provider or through an algorithm, and in either case the suicide risk is classified as low, intermediate, or high. Based on the assessment, there are various management and care pathways that can be chosen. Evatt added that the goal is to use the best science and tools to determine the risk and the pathways.

Evatt said that prediction models for suicide attempts and deaths are improving. The ML models are more accurate in predicting suicide ideation, attempts, and deaths compared to theoretically driven models (Schafer et al., 2021), and they perform better than traditional methodology (e.g., multiple logistic regression) and clinician-based prediction (Bernert et al., 2020; Burke et al., 2019). Furthermore, health care systems have started to use some of these new models. For example, the VA recently rolled out the Recovery Engagement and Coordination for Health—Veterans Enhanced Treatment or REACH VET, which increased outpatient encounters and resulted in a 5 percent decrease in documented suicide attempts (McCarthy et al., 2021).

Evatt spoke about a simulation of suicide predictive models that his team ran in the MHS. The analysts wanted to know how many at-risk individuals the models would identify, how many of those people would die by suicide, and how many would be false positives. Using estimates of performance based on published research, the analysts were also interested in how good an intervention would have to be in order for a predictive model to lead to a meaningful reduction in suicides in the MHS. The analysts identified high risk by using varying thresholds of the 99.9th percentile risk of suicide, the 99th percentile risk, and the 90th percentile risk. The model assumed that individuals over a particular risk threshold would be provided with some type of suicide prevention services.

The analysts found that if they chose the highest risk threshold, the interventions resulted in a very small reduction in the number of deaths by suicide because even though the people above the threshold were at a higher risk of suicide, the number of people in that top 0.1 percentile who would

die by suicide was actually very small. Most of the people who would die by suicide were outside the top 0.1 percentile because even though those people were lower risk, there were so many more of them.

So why not lower the risk threshold? The problem, Evatt said, is that then the number of false positives jumps up because more people are being considered. The combination of considering more people and getting more false positives means that the number of false positives skyrockets as the threshold drops. And if tens of thousands of people are getting prevention interventions that they do not actually need, that is an “untenable situation,” he said. There is a “tough trade-off” between providing interventions to so many people that there are huge numbers of false positives getting interventions that they do not need, or having a negligible effect on the total number of suicides.

Evatt also described some VA and DoD programs aimed at service members and veterans who are at high risk of suicide. One of the biggest barriers to getting these people the help they need is the stigma associated with mental health issues. Therefore, many of the programs aim at reducing that stigma or finding ways for people to get information and help without worrying that their help-seeking will become known to people outside the health care system. “We want to make it easier for people to get into care. We want to give them education and information,” he added. For example, the Real Warriors Campaign¹⁷ promotes a culture of support for psychological health by encouraging members of the military community to reach out for help whether they are coping with the daily stresses of military life or other psychological concerns. Evatt emphasized that “we want to improve access to care. We want to break down those barriers and really what we are doing is directing service members to the resources they need.”

Alternative Crisis Response

John Franklin Sierra, health systems engineer in the Los Angeles County Alternatives to Incarceration Office, described that county’s alternative crisis response efforts, which are intended to be used in crisis cases when law enforcement does not need to be involved.

One goal of the efforts is to have as many people as possible use the 988 Suicide & Crisis Lifeline¹⁸ (formerly known as the National Suicide Prevention Lifeline), instead of dialing 911. As a result of the National Suicide Hotline Designation Act passed in 2020, Congress designated 988 as a new

___________________

¹⁷ See https://www.health.mil/Military-Health-Topics/Centers-of-Excellence/Psychological-Health-Center-of-Excellence/Real-Warriors-Campaign (accessed August 23, 2022).

¹⁸ See https://988lifeline.org/ (accessed August 21, 2022).

nationally available suicide and mental health crisis hotline. In July 2022 the 988 Suicide & Crisis Lifeline officially went live across the United States.¹⁹ Sierra said that there will still be some 988 calls that require a 911-level response, either from law enforcement or emergency medical services. But the aim has been to get 988 accepted as the default number and to only opt into a higher-level response from 911 when there is a clear public safety threat or a clear medical emergency. Otherwise, crisis calls are triaged through 988 and civilian services.

Sierra explained that in triaging the 988 calls, a request for help is classified into one of four levels. Level 1, the least critical, includes calls where there is no crisis or the crisis is resolved with something like a referral to a treatment provider. Level 2 calls require immediate remote support through, for example, a transfer to a suicide prevention hotline. Level 3 calls require in-person support such as what could be provided by emergency medical services (EMS) or a psychiatric mobile response team, but the public is not in immediate danger, and law enforcement is not involved. Level 4 calls involve crimes or an immediate threat to public safety, and law enforcement is dispatched. “The hope with this kind of system,” Sierra said, “is that we can reliably push calls to 988 and to civilian mobile crisis responses as much as possible and only rely on 911 law enforcement and EMS when there is a clear public safety threat or medical emergency reason.”

The performance of the system will be monitored to see where improvements need to be made. For example, the number of 911 calls that can be diverted to 988 will be tracked, and the performance will be gauged against a certain standard. Counties will also be watching to see whether people who need an in-person mobile crisis response from a civilian team actually get that response within a reasonable amount of time. And the data will include information on such personal characteristics as age, sex, race and ethnicity, and geography. “Providing equitable access to these services is of primary importance in the county,” Sierra commented. “We know we need to keep track of data accordingly and ensure we are actually meeting those equity goals,” he added.

Sierra also spoke about the role of social media in the alternative crisis response system. He said that because the people answering 988 crisis hotlines are trained to de-escalate crises and will only engage 911 when there is a clear public safety threat or medical emergency, it makes sense that if social media algorithms are going to provide links to crisis services, those algorithms must link to 988 by default. The algorithms should only link to 911 as a last resort if 988 cannot be reached. Sierra said that social media algorithms used to determine whether a person is in crisis will always have limits and potential biases.

___________________

¹⁹ See https://www.samhsa.gov/find-help/988 (accessed August 23, 2022).

He suggested that, ultimately, a human being trained in culturally competent triage and de-escalation needs to intervene as soon as possible. “We know that there are many different ways in which people’s crises manifest, and if they are not triaged competently, they could lead to an unnecessary 911 escalation,” Sierra emphasized.

Jonathan Goldfinger, a pediatrician, intergenerational trauma researcher, and former chief executive officer of Didi Hirsch Mental Health Services,²⁰ said that Didi Hirsch is among the largest providers on the 988 Suicide & Crisis Lifeline. The organization helped develop strategic plans for California in preparation for the projected 988 infrastructure needs, volume growth, and other needs for equitable access to the Lifeline. They also helped with code-signing and testing the 911 to 988 diversion algorithms with the Los Angeles County’s law enforcement and EMS as described by Sierra. Goldfinger and Didi Hirsch collaborated with state and local leadership, suicide prevention experts, people with lived experience, law enforcement, and others to create a 988 implementation plan, and support California’s 13 988 centers in funding and preparing for the Lifeline’s new operational, clinical, and performance standards meant to allow broader access to care. He said that the 988 number, as envisioned, could help provide more equitable access to vital mental health and substance use services and help decrease suicide and mental health stigma.

Linking Datasets to Predict Suicide Risk

Holly Wilcox, professor in the Department of Psychiatry and Behavioral Sciences and the Department of Mental Health at the Johns Hopkins Bloomberg School of Public Health, who serves as co-chair of the Maryland Suicide Prevention Commission, discussed various aspects of suicide prediction and prevention at the state and local levels, with a particular focus on the value of linking multiple datasets to increase predictive power.

First, she described GoGuardian Beacon, a software tool designed to help elementary, middle, and high schools identify signs that a student may be at risk for suicide, self-harm, or harm to others. It is part of a suite of educational software offered by GoGuardian that also includes a filtering and monitoring application and a classroom management application. The Beacon application monitors students’ online activities when they are logged on with their school user ID and password, and if it detects any indication that a student is at risk of any of the targeted behaviors, it flags the content and notifies school administrators. It is then up to the school to decide how to respond, and different schools have different response protocols.

___________________

²⁰ See https://didihirsch.org/about-us/ (accessed August 23, 2022).

Wilcox said that there have been different public responses to GoGuardian Beacon. For example, when a 12-year-old boy in the Clark County School District in Nevada was detected and rescued, the media portrayal was positive,²¹ but in Baltimore, where there has been a great deal of tension between residents and police, GoGuardian has been portrayed negatively because of concerns about the police becoming involved with potentially suicidal students.²² Wilcox suggested that more research is needed on the impact of such tools as well as which response protocols are most effective and beneficial to students and under what circumstances.

Also, Wilcox discussed the Maryland Statewide Suicide Data Warehouse,²³ which was established in part with funding from the National Institute of Mental Health. She said that one of the reasons for its establishment was a recognition of the promise of ML-based algorithms in examining multiple risk factors—“this whole idea of being able to link data that usually sit in silos and never come together.” At the time, she said, very few researchers were combining multiple risk factors to predict and prevent suicide.

Wilcox said that one of the sources of data for the suicide data warehouse is the Chesapeake Regional Information System for our Patients, which is a regional health information exchange covering Maryland and the District of Columbia. It receives data from multiple settings and links them together, and the data it provides to the suicide data warehouse for predictive analysis work are stripped of identifiers. Among the other sources of data provided to the suicide data warehouse are electronic health records, insurance claims data, hospital discharge data, information on deaths and their causes for the Office of the Chief Medical Examiner, Census-derived geographical data, Medicare and Medicaid data, and data from the VA. “We really want to know which of these sources are useful in suicide risk prediction,” she said, and also to provide a framework for other states to follow if they wish to use similar predictive analytics and modeling, using commonly available data sources. Some of the most useful data are in narrative form, such as narratives and toxicology reports from medical examiners, discharge summaries, and social workers’ notes. In this case, natural-language processing is used to extract key points.

___________________

²¹ See https://news3lv.com/news/local/ccsds-goguardian-software-saves-12-year-old-boys-life (accessed August 23, 2022).

²² See https://www.forbes.com/sites/lisakim/2021/10/12/school-issued-laptops-in-baltimore-are-monitoring-students-for-risk-of-self-harm-as-concern-mounts-nationwide-over-surveillance/?sh=511f15be5a48 (accessed August 23, 2022).

²³ See https://publichealth.jhu.edu/departments/mental-health/research-and-practice/violence/our-work-in-action (accessed August 23, 2022).

The Maryland Commission on Suicide Prevention²⁴ is looking at various ways to add to and improve its suicide prediction and prevention efforts, Wilcox said. It is trying to learn from both the VA health system and Kaiser Permanente in terms of both predictive modeling and on how to use the predictions from these models to better help patients while at the same time respecting their privacy. It is looking at other potential sources of data, such as the suicide prevention toolkit of Epic Systems Corporation.

Furthermore, Wilcox described a pilot study in which she and her colleagues sent caring text messages to those who had been treated in the health care system for suicide risk and had then returned home. More than three-quarters of those in the study reported that the messages had a positive impact on their mental health, while 67 percent said the texts reduced their suicidal ideation, and 74 percent said the texts helped prevent them from engaging in suicidal behavior (Ryan et al., 2022). After the success of that study, the state suicide prevention office set up a program that would send caring text messages to anyone who signed up for them. It is relatively new, she said, but already 3,000 people have signed up to receive the messages.

Identifying Suicidal Individuals through High-Performance Computing

John Pestian, professor at the University of Cincinnati Department of Pediatrics, director of the Computational Medicine Center in the Cincinnati Children’s Hospital, and faculty member at the Oak Ridge National Lab, spoke about using data analytics on high-performance computers to identify people at increased risk of suicide. Pestian said that suicide is a complex, difficult-to-predict phenomenon. Suicidal behavior differs between sexes, age groups, geographic regions, and sociopolitical settings, and variably associates with different risk factors, suggesting etiological heterogeneity. Improving recognition and understanding of clinical, psychological, sociological, and biological factors can potentially help the detection of high-risk individuals and assist in treatment selection (Turecki and Brent, 2016). In many cases, for example, an individual may make a number of suicide attempts with increasing risks of lethality before the final attempt that is successful. He said this is because the psychological pain they feel—the “psychache”—gets worse with each attempt.

Such patterns indicate that there is an opportunity for early identification of individuals who might attempt suicide. Pestian explained that one way to

___________________

²⁴ See https://health.maryland.gov/bha/suicideprevention/Pages/governor’s-commissionon-suicide-prevention.aspx (accessed September 16, 2022).

identify patterns for predicting suicide attempts is to collect large amounts of data, including data from electronic medical records and external data such as information on social determinants, and use those data to develop models that can help identify those individuals most at risk of suicide attempts. Because there is so much data, training the models on using the data requires high-performance computers.

Pestian described two analyses related to the identification of suicide risk using high-performance computers. The first analysis trained a computer language model to analyze writings and identify the emotions expressed in them (Pestian et al., 2012). Researchers collected notes written by 1,319 people just before they died by suicide. The notes were digitized and then each reviewed by three separate annotators, who identified the emotions in the notes. The emotions identified in the suicide notes included anger, blame, fear, guilt, hopelessness, sorrow, forgiveness, happiness, peacefulness, hopefulness, love, pride, thankfulness, instructions, and information. This model could be used for a population-based assessment to understand the language and gain insights into people contemplating suicide.

In a second analysis, researchers designed a prospective clinical trial to test the hypothesis that ML methods can differentiate between the conversations of suicidal and nonsuicidal individuals. The researchers recorded and analyzed the conversations of 30 adolescents who presented to an emergency department for suicide risk and 30 adolescents who were nonsuicidal (as controls). Using ML algorithms, the researchers were able to differentiate the individuals at risk of suicide from the controls with greater than 90 percent accuracy based on spoken language differences.

In the future, Pestian said, researchers are looking to collect data from medical records, social media postings, environmental data, census data, information on the social determinants of health, and data from published scientific research. “The data would be stored in a ‘foundation database’ and would be analyzed with high-performance computers. The ultimate goal is to be able to predict the mental health trajectories of individuals and make it possible to offer interventions before an individual reaches a crisis point,” he concluded.

Leveraging Digital Health Applications to Assist in Mental Health Treatment

Mary Czerwinski, partner research manager of the Human Understanding and Empathy (HUE) Research Group at Microsoft Research, described the development and testing of an app called Pocket Skills (Schroeder et al., 2018). Czerwinski said that mobile mental health interventions have the potential to reduce financial and time burdens associated with in-person therapies and to increase engagement and comfort in providing disclosure. Pocket

Skills was designed to help people undergoing dialectical behavioral therapy (DBT).²⁵ The work, which was done in collaboration with Marsha Linehan, the inventor of DBT, was intended to make the therapy more broadly available as well as to quantify the effectiveness of the therapy. Mindful of the fact that many mobile mental health applications do not follow evidence-based principles, Czerwinski’s team used Linehan’s DBT skills training manuals and worksheets²⁶ as the basis for Pocket Skills.

As the name implies, Pocket Skills helps people develop various skills, such as mindfulness, emotional regulation, and distress tolerance. The learning has been gamified so that users gain points by developing various capabilities. All of the patients who were tested with Pocket Skills were already in DBT, she said, but many were having difficulty doing the homework given to them by their therapists. The application, which walks patients through the steps of developing a skill and asks questions about it to test their understanding, turned out to be very useful in helping these patients complete their homework. One advantage of putting the mental health behavior training on a smartphone was that it made it possible to track and analyze users’ behaviors, including what skills they chose to use, how often they used the application, and which aspects were most effective.

Czerwinski described a study with 73 suicidal patients that was carried out to test the effectiveness of Pocket Skills (Schroeder et al., 2018). The patients’ application usage was tracked over 4 weeks, and various outcomes were examined, such as when the participants used the skills, whether particular skills were more or less effective, and whether skill-level effectiveness influenced overall depression, anxiety, or skill-use improvement throughout the study. The study found that skills were used in the participants’ times of need. Results showed that the context in which skills were used was very important, and that skills were more or less effective for different subgroups of people. In addition, characteristics such as age, disorder type, how close family members were, and type of medication influenced the effectiveness of different skills.

Czerwinski said because individual characteristics influence how effective the various skills are, one approach to improving the overall effectiveness of DBT would be to use ML to personalize the interventions. “It’s really important that you allow participants or users of your applications to tell you what works for them in that . . . particular context, that particular moment,” she emphasized. She added that it is also possible to use ML to predict which

___________________

²⁵ DBT is a type of cognitive behavioral therapy that can be used to treat people with suicidal ideation and various other complex disorders, helping them to develop concrete coping skills.

²⁶ These are comprehensive resources that provide vital tools for implementing DBT skills training.

skills will work for which people in which contexts. It requires a minimum of about 3 weeks of user data to be able to make such predictions, but by combining these data with expert feedback, one can recommend skills based on emotional, personal, and environmental contexts.

Czerwinski concluded that in developing mental health applications, it is important that the design takes the unique emotional, environmental, and personal contexts of individuals at risk for suicide into account and allows for evidence-based support with personalized skill recommendations.

CURRENT ONLINE APPROACHES TO PREDICTING AND PREVENTING SUICIDE

Several speakers described the current state of the art of online approaches to predicting and preventing suicide and what possibilities the future might hold.

Google’s Efforts to Help Users in Crisis

Megan Jones Bell, consumer and mental health director at Google, spoke about expert- and partner-informed strategies to improve helpfulness to users in crisis. She agreed with other speakers that a person going through a mental health crisis can feel isolated, overwhelmed, and distressed. To get through those moments, access to the right information at the right time can make all the difference. Bell said that with searches for mental health therapists or mental health help on the rise, Google is working to quickly support users experiencing mental health challenges by providing them access to reliable information. She added that “our goal is to surface authoritative information that you can trust, create access to helpful resources you need in the moment, and show empathy for everyone who is experiencing mental health issues.”

Bell said that in order for Google to uncover the right crisis information, it is first necessary to recognize that a user is in crisis in that very moment, which in turn requires understanding the nuances of human language. Because it is not always obvious from the wording of a query that a user may be in crisis, AI and ML can facilitate identification of users who need help. Google recently updated its “Search” to use its AI Multitask Unified Model (MUM) to automatically and more accurately detect personal crisis queries in 75 languages in order to show users the most relevant information when they need it. The MUM is used to detect when users are looking for information related to suicide, domestic violence, sexual assault, and substance use. It is capable of a nuanced language understanding, and can also read images, videos, and other forms of media.

Once Google Search has identified a crisis query, it does several things to provide access to high-quality information and immediate help. At the top of

the search results, it provides contact information for relevant hotlines that are available to provide immediate help 24/7. “For example,” Bell said, “if you type in, ‘how to kill myself’ or ‘ways to commit suicide,’ we will provide 988 phone number in the US and other local hotline numbers in other countries.”

Something similar is done on YouTube, which is part of Google. When a user does a search on YouTube that indicates a crisis situation, the platform shows crisis resource panels to help users connect with local organizations that can help them in a moment of critical need. The panels appear on the “watch page” below videos offering a combination of educational and emotionally resonant content alongside prompts to take action if needed. The watch page is the place where users spend the most time so contextualizing mental health resources there increases the reach of critical health information.

Bell said that in the past, YouTube’s crisis resources had only appeared in response to searches related to suicide and self-harm. But because mental health and well-being go far beyond suicide and self-harm, the company is now expanding the range of topics that display crisis resources on YouTube search results to include issues such as depression, sexual assault, substance use disorders, and eating disorders. Bell said that the company has also introduced a new video block for suicide-related queries. She mentioned that if someone searches for something like “I want to die,” the search results include a section that shares videos telling stories of help and recovery as well as video content from national hotline providers.

Bell added that “mitigating risk for suicide also involves intervening upstream of an acute crisis. And we have invested in evidence-based approaches that help raise awareness of mental health conditions through screening, credible health information, and actionable resources.” She added that both Google Search and YouTube make it easy for users to access clinically validated mental health self-assessments for conditions such as depression, anxiety, postpartum depression, and posttraumatic stress disorder. These self-assessments—which are the same as those used by many medical and mental health professionals—are meant to help people understand how their self-reported symptoms might map to known mental health conditions. Also, both websites provide high-quality, expert-informed health information about common mental health conditions through knowledge panels located on the search results page, to make it easier for users to find information on topics including the symptoms and common treatments for depression and anxiety.

Bell referred to a set of principles published by the National Academy of Medicine on how to define authoritative health content sources and the ways in which those sources attain and maintain their authority. She said that YouTube uses these principles to identify and designate authoritative health sources (Kington et al., 2021). Bell added that “we know that authoritative health information needs to be complemented with personal stories of hope

and inspiration. Sharing stories about mental health can normalize the issue and reduce stigma that deter people from getting help.” YouTube has worked with the National Alliance on Mental Illness²⁷ to develop a guide for creators with tips on how to speak from personal experience, work with experts, and use inclusive language in the mental health content that they put on their platform.²⁸ Bell closed by noting that Google and YouTube have various policies that remove any content from search features that might be harmful to users’ mental health or that promote dangerous behaviors.

Koko: Offering Mental Health Help Online

Rob Morris, co-founder and chief executive officer of the tech-based nonprofit organization Koko, described methods that Koko has developed to identify and help users of various internet platforms who are struggling with mental health issues such as depression or suicidal thoughts. Morris said that most individuals with mental illness suffer for approximately a decade before receiving help and yet millions are reaching out for help on social media platforms every day. Thus, there is a huge opportunity to help these people. Morris said that this is what Koko is doing successfully.

Koko integrates suicide prevention resources directly onto social networks and large online communities to identify risk and triage the appropriate response, from immediate crisis to well-being courses to peer supports. Morris said the toolkit that Koko uses has two components. First is a keyword library that detects high-risk terms and phrases that people post or search for online. Then, once these terms and phrases are detected, Koko reaches the users on the platforms they are using by providing free resources and interventions that are embedded directly in those platforms. Most platforms have some processes in place to detect some of the more common search terms or phrases related to suicide or depression, and they will respond to the presence of the keywords in various ways, such as offering a public service announcement or directing the user to a helpline. However, Morris said the keyword lists of these platforms always have huge omissions. Slang terms, misspellings, and evasive language also create problems, so the typical platform detection methods miss many cases of users with mental health issues.

Koko addresses this challenge by having tens of thousands of keyword terms that detect not only common words and phrases but also slang, evasive language, misspellings, and other variants that point to problems but would be missed by most detection methods. Koko gets keywords from its network of partners, scrapes the web to get various terms from social platforms, and

___________________

²⁷ See https://www.nami.org/About-NAMI (accessed August 23, 2022).

²⁸ See https://www.googblogs.com/author/dr-megan-bell/ (accessed August 19, 2022).

also uses text transformers to generate new keywords and phrases. Morris said that the goal of having and integrating all these keywords is not to predict suicide but rather to reach users who are already crying out for help publicly on internet platforms. Once reached, their mental health trajectory is redirected right at the moment they are most willing to seek help.

Morris shared an example of a user searching for “TWSH,” which is shorthand for trigger warning, self-harm. After detecting this search, Koko’s program suppressed the search results and instead showed a public service announcement with links. Koko also sent a direct message to that user with KokoBot, which offered various services, such as a connection with peer counselors. Morris called this approach “high touch.” Sometimes a “low-touch” approach is used instead, in which case there is no direct message, and Koko is simply pinned in the search results with a note like, “Need someone to talk to?”

Morris said that because of privacy considerations, Koko does not collect any personally identifiable information. People can make these connections anonymously. Koko also works with other groups beyond the large social networks—anything where there is user-generated text is an opportunity to apply this paradigm, he said. In addition, Koko is exploring ways to work with virtual-care providers to help “fill in the cracks” by responding more quickly to users because a user message might not be seen by a mental health care provider for 12 hours, 24 hours, or even 48 hours after it has been sent.

Morris described the interventions that Koko can point a user to, saying that the ideal is to find resources within the platform the user is on to avoid the next step and inconvenience of going to another platform. For this reason, Koko provides a suite of services including referrals to crisis lines, online peer support, self-injury support, and eating disorder support, among others. KokoBot sits at the nexus of these referrals, routing people to different services depending on how the user is presenting.

Among the options are single-session interventions, which are short, targeted interventions that can be easily integrated into any platform and are accessed with a link. They were developed with researchers at Stanford, Dartmouth, and Stony Brook universities and cover topics such as eating disorders, anxiety, and self-harm. There are five now, but Koko is planning to develop many more, Morris said. A study done on the use of these interventions on Tumblr found a high completion rate and significant improvements in hopelessness, self-hatred, motivation to stop self-harming, and body image (Dobias et al., 2022). At the heart of Koko’s service is what it refers to as a “precision peer support”—an AI–powered peer-support platform on which users who have received help then become support providers and help other people. By helping others, they help themselves, Morris said. He added that the majority of Koko users who seek help go on to help other people.

Morris described Koko’s crisis triage. He said that simply giving people in crisis a toll-free number for a support center is not enough; only a small percentage will actually use it, particularly among young people. Thus, the Koko team developed an intervention designed to make it more likely for people identified online as at-risk to take advantage of the offered support. A randomized controlled trial found that those users who were provided with this intervention were 23 percent more likely to use the crisis service referrals (Jaroszewski et al., 2019). He concluded that the intervention works by helping users overcome the common barriers and misconceptions that will typically keep users from following up on referrals to these services.

Peer-to-Peer Mental Health Support

Tim Althoff, the director of the Behavioral Data Science Group in the University of Washington’s Allen School of Computer Science & Engineering, described a research collaboration with a social media platform called TalkLife,²⁹ which is focused on providing peer support to people with mental health challenges. He discussed how AI/ML and natural-language processing techniques can help improve the support on these platforms.

Althoff said that access to mental health care is poor across the globe. He added that “we may never have enough mental health professionals to meet the increasing need.” He mentioned that, currently, millions of people use online social media platforms such as TalkLife to seek—and provide—support through conversations with their peers who have faced similar challenges, but those peers typically have little training on how best to help others. Thus, TalkLife was created to help volunteers support others better online. Althoff emphasized that the intervention is aimed at the people on TalkLife who provide support to others rather than the people in crisis themselves.

Althoff said that the people who log on to TalkLife for support are already in distress and this is a very challenging situation in which to offer support. But a key component for providing effective mental health support is simple empathy; that is, the ability to understand or feel the emotions or experiences of others (Elliott et al., 2011). For example, Althoff said that someone on TalkLife might say that they think their whole family hates them. Then an empathic response by a peer supporter might be something like, “I can imagine that makes you feel really isolated.”

So Althoff set out to create an automated way to help peer supporters express more empathy. The first step in that process was to develop an ML algorithm that could measure empathy expressed in text by adapting scales

___________________

²⁹ See https://www.talklife.com/ (accessed August 23, 2022).

used in clinical psychology for use in text-based conversations on social media. The next step was to operationalize each of the types of empathy communication by scoring responses on a scale of 0 to 2 (no expression of empathy, weak expression, or strong expression) for each way of communicating empathy. Althoff said that, for example, “I understand how you feel” would be a weak interpretation response, whereas “If that happened to me, I would feel really isolated” would be a strong interpretation response.

Althoff’s team developed a model that identified this sort of sharing in peer-to-peer conversations on TalkLife, scored each expression of empathy, and provided the supporting evidence or rationale for the score. They found that responses that rated as “strongly empathic” on emotional reaction or interpretation got nearly 50 percent more likes than those rated as “not empathic.” And while the responses rated “strongly empathic” on exploration actually received fewer likes, they got nearly 50 percent more replies, which are instrumental in getting a conversation going between two people on the platform. Thus, he said, whenever the model detects empathy in conversations there are also higher degrees of engagement.

The team also examined the effects of total empathy—which was determined by adding the three individual scores to get a number between 0 and 6. They found that when strong empathy was communicated, there was a much higher chance of an individual seeker explicitly following and creating a longer-term relationship with the person who supported them. Using large-scale language models and reinforcement-learning technology, they built a model that could look at a conversation between a help seeker and a peer supporter, rate the level of empathy in the responses, and offer the peer supporter suggestions on a sentence-by-sentence level on ways to be more empathic.

Training such a model requires large datasets because the machine learns by offering suggestions for rewriting sentences and being rewarded for sentences that express greater empathy. However, because increasing empathy is very challenging, it is quite difficult to collect data where empathy has been increased, so Althoff’s team chose to look instead at the reverse process of decreasing empathy. That is, it would find posts that are already expressing empathy and remove some of the parts that play a role in that expression of empathy. In this way they were able to create a dataset containing posts of both high empathy and low empathy. Such an approach has the potential to scale to millions of response pairs that can be used to train ML systems.

Once a machine has been trained to know the difference between more and less empathic responses, it can be put to work offering suggestions to peer supporters. Althoff shared data from a randomized controlled trial of one human–AI collaboration approach undertaken with peer supporters on TalkLife, which showed that those who had been given machine-generated suggestions expressed nearly 20 percent more empathy than those who had not

received the real-time suggestions (Sharma et al., 2022). Also, the researchers found that 63 percent of participants found the feedback they received helpful, and 78 percent of participants wanted this type of feedback system to be deployed on TalkLife or other similar peer-to-peer support platforms. The randomized trial suggests that human–AI collaboration on empathy can be effective. This is an illustration, Althoff said, of the power offered by collaborations between intelligent systems and humans—in this case, peer supporters.

CONSIDERATIONS FOR MOVING FORWARD

A number of speakers discussed specific issues to take into account as developers work to create ways to use online data in predicting and preventing suicide.

Moving from Individual Predictions to Population Priorities in Suicide Prevention

Philip Resnik, professor in the Department of Linguistics and the Institute for Advanced Computer Studies at the University of Maryland, began his presentation by asking that, given that the mental health care system is currently overburdened and primary care clinicians are at the front lines to help, “What do we do about this overburdening?”

A major part of the answer, he said, is to get the technologists—those developing ML algorithms, for instance—and the psychologists and other domain experts to talk with each other. It is important that domain experts help the technology community to understand what is needed, and it is equally important that technologists help the community of experts in these issues understand what is possible. In particular, Resnik offered suggestions in three areas: (1) moving from classification to prioritization, (2) helping clinicians understand people’s risk in context, and (3) gaining insights from lived experiences.

Resnik said that the dominant paradigm in ML for suicide prediction is classification—putting people in different risk categories (e.g., no, low, moderate, severe risk) according to the machine’s calculations. There are several problems with this approach. First, it is usually very difficult to understand why an individual gets classified in a particular way. Second, the classifications do not provide enough granularity; they are just a rough grouping. And given resource limitations—it is impossible to give help to everyone who might be in the medium- or high-risk category—it is important to be able to prioritize.

Resnik said that unlike most ML approaches, “from a care provider point of view, you are not classifying an individual. You are dealing with a population of individuals who need to be paid attention to, and you need to figure

out how to prioritize among those individuals, so it is not a classification, it is a ranking.” Resnik said that a clinician-centric view of the problem leads one not only to move away from simply classifying individuals as belonging to a certain risk category, but also to identify which social media content (e.g., which sentences and words) were important in assigning that particular risk level or score (Shing et al., 2020). Those bits of evidence can also help clinicians in assessing sets of individuals in terms of who is most at risk or who can be most helped by an intervention. When one takes this approach there is also a different measure of success, he said. Instead of measuring success by how accurately individuals have been put into risk categories, one measures success by how many high-risk individuals can be identified within a limited time budget. The important concept here, he emphasized, is moving beyond simple classification with ML and giving clinicians the information they need to efficiently identify the people who most need help.

But even that is not enough, he continued. Along with this information, health care providers need details to help them understand people’s risks in context. One way to do this is a top-down approach, using predetermined categories such as a sense of entrapment or frantic hopelessness (known to be signals of suicidal crisis; see Rogers et al., 2022) to understand what is going on with an individual. But ML also offers the possibility of a bottom-up approach, examining the writings of at-risk individuals to identify categories or constructs that might not have been previously recognized. Resnik said that researchers in his lab are using a computational method called topic modeling to identify topics or categories from textual content produced by a population of individuals. Many of these categories overlap with traditionally recognized constructs such as social isolation, but what is important in this type of bottom-up analysis is that the categories themselves and how they are characterized are specific to the population being analyzed. The categories for a population of military veterans, for instance, would not be characterized in the same way as those for a collection of students.

A similar type of approach might be used to provide further insights from other aspects of lived experiences, Resnik said. Clinicians are most successful if they take individuals’ lived experiences into account, but these are difficult to analyze with ML because each person’s lived experiences are unique. He said, “You have to be able to find cross-cutting insights and generalizations if you are going to be able to make effective use of the information about people’s lived experiences.” He added that “we need a solution that takes the lived experiences into account and also allows us to extract generalizations and useful insights from a population.”

Resnik’s team applied this approach in analyzing more than 16,000 responses on Reddit to a question about what had helped keep formerly suicidal individuals alive when they were contemplating suicide (Resnik et al.,

2022). Looking for insights that might help inform interventions, the team used a form of computer-assisted content analysis in which subject-matter experts assigned labels to the various ideas expressed in the posts as discovered using a topic model. They found various factors that had helped individuals avoid suicide attempts that were already recognized in the field, such as social connection, religion, and treatment, but they also found factors that were not so widely established, such as the presence of pets, interest in video games, and even a desire to find out what would happen next in television shows and YouTube channels. The next step is to determine how to use such insights in working with individual patients, but he said the key takeaway message is that the bottom-up approach can provide many details and insights that are different from what one gets from labeling with ML or the usual top-down approach of looking for factors that are already known.

Resnik emphasized the importance of access to relevant data and to the advanced technologies that can effectively analyze those data. He said that the language-processing technologies available to those in the mental health field are a decade behind the technologies used to filter email spam or analyze legal documents or predict the next word when typing on a smart-phone. He added that the use of such technologies with datasets relevant to mental health will come with various ethical issues, noting that it will be important to find a balance between the concerns about privacy and the value of the data in helping people.

Youth-Centric Technology Approaches for Mental Health

Jana Haritatos, chief science officer at Hopelab,³⁰ which focuses on technology and youth mental health, spoke about how members of Gen Z³¹ think about mental health and what could be done to make mental health technologies more appealing to this group of young people. Haritatos said that individuals classified as Gen Z comprise nearly up 20 percent of the U.S. population. She added that Gen Z is the most racially and ethnically diverse generation in history (Fry and Parker, 2018), and they are on track to also become the most digitally connected generation in history (Lebow, 2022). Unfortunately, they are also a generation in mental health crisis, with a nearly 60 percent increase in suicide rate among young people over the past decade (Curtin, 2020). She emphasized that these burdens are disproportionately felt by youth of color; lesbian, gay, bisexual, transgender, and queer (LGBTQ)

___________________

³⁰ See https://hopelab.org (accessed September 18, 2022).

³¹ See https://www.pewresearch.org/fact-tank/2019/01/17/where-millennials-end-and-generation-z-begins/ (accessed August 18, 2022).

youth; and youth from other underrepresented communities (Bridge et al., 2018; Gorse, 2020).

Haritatos said that those in Gen Z tend to talk about mental health issues in a more direct and expansive way than previous generations but they do not necessarily turn to or even trust the established mental health systems for support with these issues. Instead, they are more likely to find their own paths, particularly online with help from peers. Furthermore, many of these adolescents and young adults see social and structural issues as mental health issues.

Given that perspective, Haritatos asked, how should online mental health efforts be shaped to be more appealing to and effective for members of Gen Z? She said that because these young people are already expressing a range of needs online, it will be important to meet them where they are with responsive, real-time support.

Haritatos pointed to Koko (described in another presentation by Robert Morris) as another example of an approach to providing online resources to at-risk users that is likely to be effective with members of Gen Z. One thing that makes Koko particularly appealing to today’s adolescents and young adults, she said, it that it integrates suicide prevention resources directly into social networks and large online communities, identifying risk and triaging the appropriate response from providing immediate crisis help to offering well-being courses and peer supports. Approaches like Koko “really open up opportunities to problem-solve with social platforms around risk assessment and triage,” she said, but, importantly, they also open doors to the provision of more upstream preventative interventions.

Goldfinger said that one of the ways to reach out to these young people is to work with celebrities and influencers who could reach one of the high-risk populations. Didi Hirsch has tested this approach through a partnership with Selena Gomez³² and her Rare Beauty makeup brand. Goldfinger said that “when you have Selena Gomez sharing a post about suicide, you have an enormous amount of people sharing current risk, future risk, and protective factors.” Goldfinger added that “she was able to reach many more young people in a more meaningful way” than a provider of suicide prevention services could do on its own through a single post. He continued that responses from these young people could also be data that could be harnessed for population-level intervention and innovation.

Haritatos also described imi,³³ a free digital, evidence-based mental health tool intended to provide LGBTQ+ teens with inclusive, identity-affirming,

___________________

³² She is an American singer, songwriter, actress, and mental health advocate, with one of the world’s largest social media followings across platforms.

³³ See https://hopelab.org/product/imi/ (accessed August 23, 2022).

well-being support to improve their mental health and lower their risk of suicide. The tool was developed by Hopelab in partnership with CenterLink and the It Gets Better Project³⁴ with input from hundreds of LGBTQ+ teens. Haritatos said that there is a huge opportunity currently to harness the power of ML to identify the vulnerable youth most likely to benefit from tools like these and to provide immediate responsive support and information in the digital spaces where they already are.

In taking advantage of that opportunity, Haritatos said, it is important to keep in mind that the web is constantly changing and evolving, so solutions are needed not only for the web today but also for the future. She said that major technological companies are already redefining social spaces and contexts, and surveys show that members of Gen Z are already inhabiting the metaverse.³⁵ Mainstream companies are already racing to set up all types of services and goods in the metaverse, Haritatos said, so it is essential to have health care—and specifically mental health care—available in the metaverse as well. And, indeed, that is already happening. In late June 2022, Emplomind, a UK-based provider of mental health coaching, announced that it was set to launch the first ever metaverse mental health hub on Decentraland, a three-dimensional, browser-based, virtual world platform (Cureton, 2022). Haritatos urged that it is essential for “AI, ML, and suicide prevention research communities to play a defining role while all of this is still taking shape.”

Haritatos said that because the members of Gen Z are incredibly savvy about digital spaces, they are well positioned to play major roles in shaping the digital and online technologies that will be increasingly important in their lives. “They’re uniquely positioned to be leaders in the realm of technology and internet culture because they’ll be both inheriting the problems of today and enacting the solutions of tomorrow,” she said, “and they’ve demonstrated a tendency to demand more equity and more responsibility.” As an example of how this is already happening, she pointed to Youth AI,³⁶ a “community of young people learning and coding together to drive the next generation of AI work, with ethics and real-world problem solving at the center.” She also mentioned the work that Hopelab is doing with young people who are contributing their social media data to help in the discovery of new indicators of loneliness and other parts of mental health in digital phenotypes. She concluded that “we believe that mental health assessments and solutions of the future will be co-designed by the next generation and will tackle some of the biggest social issues of our time.”

___________________

³⁴ See https://itgetsbetter.org/ (accessed August 23, 2022).

³⁵ The term metaverse refers to a digital or online environment with user experiences that can include augmented or virtual reality.

³⁶ See https://youthai.org/ (accessed August 23, 2022).

Ethical Issues Consideration

Camille Nebeker, associate professor in the Herbert Wertheim School of Public Health and Human Longevity Science at the University of California, San Diego, discussed how to think about ethical issues, particularly those that arise in the context of digital health and the use of AI.

Nebeker provided background on the framework for the ethical, legal, and social implications (ELSI) of technology, which originally came into prominence as part of the research into the human genome by the National Human Genome Research Institute of the National Institutes of Health.³⁷ ELSI are now being applied in various other health contexts, such as digital health and the use of AI in health care (Celedonia et al., 2021; Cordeiro, 2021; Gomes de Andrade et al., 2018).

Nebeker said that among the most important ethical principles when dealing with human health issues and medical research in which people take part are autonomy, agency, respect for persons, beneficence, and justice. It is particularly important to keep those principles in mind when thinking about ways to communicate information to people so that they can effectively choose; that is, to offer informed consent—whether to take part in a medical procedure or study, she said. Nebeker emphasized that the informed consent process should be accessible and meaningful, but unfortunately in many cases it is not. She added that one of the major barriers to informed consent is the ability of individual patients or research participants to understand the procedures in question well enough to make an informed choice; that is, they need to have a certain level of literacy concerning the different aspects of participation. Nebeker continued, research participants need literacy not only in medical areas to understand research questions but also in data issues so that they can develop a reasonably clear understanding of what data will be collected, how it will be handled, how it will be analyzed, what information can be extracted from the data, how and with whom it will be shared, and what sorts of risks exist if the data are not kept properly confidential. The issues of consent and literacy go hand in hand, she said.

Ethical principles also require that bystanders be taken into account. These may be people whose data may be collected in passing, such as those who have made posts on Facebook to the account of a person being monitored, or they may be the moderators in the sort of peer-to-peer networks that Althoff discussed. These people need to be protected as well, Nebeker said. The principle of beneficence calls for the benefits of an action to outweigh the harms or risks, which requires careful consideration of both benefits and risks.

___________________

³⁷ See https://www.genome.gov/ (accessed August 17, 2022).

In doing so, Nebeker said, it is important to involve many different voices in the conversation.

Regarding legal issues, Nebeker singled out two main areas of consideration for product developers: (1) making sure that research adheres to all appropriate regulations, such as those concerning privacy issues or conflicts of interest, and (2) avoiding lawsuits and other legal repercussions. For example, a product that claims to lessen the risk of suicide but sometimes leads to an increased risk instead would face the risk of lawsuits from the family of anyone who attempted or committed suicide after using that product.

Concerning social implications, Nebeker said that researchers need to think about people’s expectations concerning a product and its potential downstream impacts. As an example of the sorts of social implications that can arise, she pointed to the Facebook emotional contagion study.³⁸ Because Facebook did not disclose that it was manipulating the feeds of users as part of the study, it faced a tremendous public backlash (Alberton and Gadarian, 2014; Kramer et al., 2014). Nebeker stressed that it is crucial to inform people of all the important details of a study, including what data will be collected and what information will be returned to participants. It is also important for researchers to understand what sorts of information the participants find valuable, which requires a two-way interaction between researchers and participants.

Traditionally, the informed consent process has been carried out in person, but currently the process is increasingly being carried out indirectly, with participants going through forms on a tablet or their phones. That makes it particularly important that the information be presented clearly, with such aids as embedded videos and words that are highlighted so that people can look up their meanings. Even with well-designed consent processes, though, the forms are inevitably long enough that some people will skip through much of the explanation and end up providing their consent without truly understanding such things as who will be accessing their data and how their data will be stored and used.

Thus, Nebeker and colleagues have been working to identify ways to help people be better informed in the consent process. Part of that work is helping researchers make better choices concerning the technologies they are using to understand human health and behavior—such as wearable sensors or social media monitoring. They have developed a decision support framework that formalizes the key issues that researchers should consider when designing a study (Nebeker et al., 2020). These areas include access and usability, privacy, data management, and risks and benefits. Concerning privacy, for instance, researchers should consider how best to convey to participants which personal data will be collected and how those data will be used and shared. Concerning

___________________

³⁸ See https://www.pnas.org/doi/10.1073/pnas.1320040111 (accessed August 17, 2022).

risks and benefits, researchers should consider the evidence for the product’s validity and reliability, how best to disclose the risks, and whether the benefits outweigh the risks. Nebeker said that the framework could also be used to help researchers determine the best questions to ask of vendors who provide AI-based products for use in research.

Designing Systems for Equity

Rayid Ghani, professor in the ML Department at Carnegie Mellon University, spoke about practical lessons and challenges in building fair and equitable systems for suicide risk assessment. He said that inequity often results from the fact that most mental health organizations, whether they are government agencies, nongovernmental organizations, or mental health services organizations, generally are operating with very limited resources. He added that because the organizations naturally want to help as many people as possible, they tend to focus on people who are the easiest or least expensive to help, leaving behind those who are more difficult or expensive to help, which can lead to inequity among groups. Similarly, many AI/ML systems are built with an implicit goal of efficiency, he said. “So, out of the box, if you just take a system and use it, even though you don’t tell it what to do, it assumes your policy goal, your societal goal is efficiency.” So unless equity is built in as an explicit goal, the results are often inequitable.

Ghani said many people assume that when AI systems produce inequitable results, it is because of the data that were used and that using unbiased data will fix the results. Unfortunately, he said, it is not that simple, adding that “none of these problems are so simple we can just fix one thing and the world changes.” So the goal should not be making the data or some other single part of the system unbiased; it should be to make the overall system function in a way that leads to fair and equitable outcomes. As an example, Ghani said that if a system is designed to be totally fair but then the outreach program is done through phone calls in English, the system will be inequitable no matter how fair the data or the system are because the intervention is biased against people who are not reachable by phone and who do not speak English. This has actually happened in many cases, he said.

Ghani said that a system does not have to be perfectly equitable in order to be used. The status quo is not perfectly fair, so it can be enough for a system to improve on the current situation, although one needs to think about just how much better a system needs to be than the status quo in order for it to be worth using. In designing an equitable system, one needs to answer four questions, he said: (1) Where does bias come from? (2) How does one determine what types of biases to care about? (3) How can the biases be detected? and (4) How can the biases be reduced?

Ghani said, there are many sources of bias. It is not just the data; much of the bias is the result of an inequitable world, he added. For example, using data from medical records systems, it is possible to tell which people have a certain disease based on a diagnosis code, but just because a person does not have that diagnosis code does not mean the person does not have the disease, he said. Many people with the disease do not get tested and thus never receive the diagnosis. So one major source of bias in data is the fact that there is not equitable access to health care services. But bias can also be introduced into data by the choices that are made concerning the data. For example, using data only from electronic health records or only from social media or only from county mental health services can ultimately lead to bias in the results derived from that data. And the choices made in developing an AI system can introduce yet other types of bias.

Regarding how to determine what types of bias to address, Ghani said the data science field has recently been working on different definitions of bias, and it turns out that the definitions are very similar to the ones that philosophers and ethicists have talked about for centuries. Some of the possibilities include seeking to allocate equal resources to each group, allocating resources proportionately to need, making sure that people of a certain group are not missed disproportionately, and making sure that there is not a disproportionate number of false positives from a certain group. He shared a framework to help people determine which sorts of bias to focus on when developing interventions. The framework is in the form of a decision tree where one answers questions and then, depending on the answer, moves on to the next question until, after going all the way through the tree, one ends up at the desired approach to addressing bias (Figure 5) (Rodolfa et al., 2020). At one point in the decision tree, for instance, the question is whether one’s intervention is punitive (could hurt individuals) or assistive (will help individuals). If an intervention is punitive, one wants to avoid biases that will lead to unnecessary interventions, whereas if it is assistive, the goal should be to avoid biases that exclude people disproportionately.

Ghani said that when designing suicide interventions, it is important to have discussions with the participants in the suicide prevention system—the medical professionals, the at-risk individuals and their families, and so on—and learn about their values. Those values should then inform prioritization of the choices about which biases to address.

On detecting biases, Ghani said that there are various tools that can be used to audit interventions to see how well they are doing at avoiding biases.

As an example, Ghani pointed to the Aequitas Open Source Bias and Fairness Audit Tool,³⁹ which was developed at Carnegie Mellon.

Once biases have been prioritized and detected, the final question is how they can be reduced. Again, Ghani said, there are no easy answers. “Sometimes people will say, I don’t want my system to be racist, so I’m not going to use race,” he said, adding “but it’s not that simple.” Race is embedded in many things, even it is not explicitly included, so simply removing race as an explicit variable in the model does not guarantee the output will not be racist. Thus it is important to examine the overall outcomes of the model for racism and other biases and work to reduce those issues in the outcomes, not just in the inputs to the model and the model itself.

Ghani reiterated that today’s AI systems are generally focused on being efficient. They have the potential to be equitable, but they have to be designed that way, “and that process has to be an integral part of every project, from scoping the project to engaging the communities that are being affected, to validation and monitoring.” He cautioned that equity in data science and AI is a new area with many unknowns and tentative ideas and solutions. “So if you’re working in this space, be careful,” he said. “Don’t take things at face value. A lot of these things that are coming up haven’t been validated. So, before you jump in and start using some of these tools that are supposedly

___________________

³⁹ See www.datasciencepublicpolicy.org/aequitas/ (accessed August 23, 2022).

dealing with equity in the use of data and machine learning, question it, validate it, test it, because it’s a new area and it’s important to develop it because it can have a huge impact,” he concluded.

Intervening throughout the Entire Crisis Care Continuum

Goldfinger said that it is increasingly important to monitor across the entire crisis continuum to predict suicide and intervene in real time for all populations in a unique and inclusive way. He offered a vision for the future for how this could be done.

First, Goldfinger said it is important to link information about childhood trauma and suicide collected from both health care and the metaverse, to better inform people who are providing the real-time interventions. That linked information could also help technology companies to improve the algorithms to identify individuals who are at risk and need intervention. He said that “we know that trauma influences who we associate with in the real and digital worlds and how we behave in both”; therefore, it would be essential to consider adverse experiences as data points for suicide prediction and intervention efforts. Second, he said there is a need to have volunteer-based interventions such as 988 crisis centers and Crisis Text Line, which often come from the communities and are developed by the communities, integrated into the metaverse. Third, he said there is a need for public–private partnerships among digital firms, academic researchers, frontline service providers, and individuals with lived experience—throughout the entire crisis care continuum—to codesign gold standards for digital suicide risk assessment, intervention, and information sharing. “This needs to be an all-of-society effort,” he emphasized.

Goldfinger outlined three principles to enable this growth mindset. The first is knowledge. He said that “we can’t solve what we don’t understand and what we don’t measure.” The second is money. He said that “we have to talk about the business incentives first and foremost to ensure sustainable change in all of our ecosystems that touch upon the lives of those at risk of suicide.” He said that community-based organizations that provide mental health and suicide prevention services, including Didi Hirsch, have largely been underfunded. He added that there “should be financial incentives for data sharing, data mining, and de-identified data research.” And the third is time, he added, noting that “the time to change can actually be accelerated and we can be more efficient with our time through partnerships.”

Opportunities for Research on Risk Assessment

Richard McKeon, chief of the Suicide Prevention Branch at the Substance Abuse and Mental Health Services Administration (SAMHSA) Center for

Mental Health Services, identified several areas in which there are unresolved questions that will need to be taken into consideration in developing data-driven resources to help individuals at risk of suicide.

He observed that, while there are large amounts of data to work with, often there are not enough data at the times and places they are most needed in order to make critical decisions that can mean life or death. This is particularly true for determining the best way to respond in emergency and crisis situations. McKeon said that whether it is dealing with a caller on a crisis hotline or working with a patient in an emergency department or mobile crisis unit, clinicians perform similar suicide risk assessments. The only difference is the location, which has implications for the person’s current safety. For example, what is needed to keep a person safe on a crisis phone line is very different from what is needed if that person is in an emergency department. McKeon said that what is striking to him is how little research has been done on crisis interventions, including the question of the short- and long-term impact on mortality of specific types of interventions. “There are basic questions we do not know the answers to,” he said. For instance, do emergency interventions—which are sometimes involuntary—provided during a time of crisis have an effect on long-term survival?

One study supported by SAMHSA examined factors affecting decision making at crisis call centers (Gould et al., 2016). The researchers found that even for callers at imminent risk, about a third of the time it was possible to reduce their level of risk to the point where the call taker did not think that any kind of emergency rescue was needed. In another third of the cases a voluntary emergency intervention took place, such as a family member transporting a loved one to the emergency room. The final third of the situations were resolved with an involuntary emergency intervention. “One of the key variables in that difference was the extent to which the caller collaborated,” McKeon said.

He added that while the research literature questions the empirical validity of assigning low-, medium-, and high-risk classifications in such crisis situations, “the reality is that we need to make these determinations anyway because lives depend on it and to not intervene has the same life-and-death implications as to intervene.” And the importance of addressing such questions will only increase now that the 988 crisis number is being implemented nationwide, he said.

Regarding ongoing care for at-risk individuals, McKeon spoke about the people who can fall through the cracks of suicide risk assessments. He said that the Zero Suicide Model⁴⁰ involves identifying people who are found to be at an elevated risk of suicide and then providing them with evidence-based

___________________

⁴⁰ See https://zerosuicide.edc.org/ (accessed August 17, 2022).

care, and this can be very effective, but health care organizations that track suicide-related mortality among their patients often find that many suicides are taking place outside the identified group of high-risk individuals. This does not mean that it is a mistake to give high-risk patients particular attention, he said, but it does imply that organizations need to consider also reaching out to those who have not been identified as being high risk.

McKeon suggested that more work needs to be done on suicide clusters and hotspots, noting that there is a particular lack of such information in tribal communities where there have been numerous tragic suicide clusters. He added that “understanding more about the development of such clusters and the relationship of these clusters to preexisting, high-risk characteristics in a community is a key area for future inquiry.”

Opportunities for Policies to Improve Mental Health Care in the Digital Environment

Kacie Kelly, chief innovation officer at Meadows Mental Health Policy Institute,⁴¹ which was set up to improve the quality of mental health care in the state of Texas, presented suggestions for policy approaches that could help improve mental health care in a digital environment. Kelly observed that the U.S. mental health system is “designed for disease alleviation as opposed to proactively detecting people early before they’re in significant decline.” Thus, people with mental illness tend to be diagnosed when they are in crisis and declining, either in an emergency department or in the justice system “where it is just much harder to offset the trajectory that they’re on.”

Instead, Kelly said, the goal of the mental health care system should be preventing diseases or detecting them early when they are much easier to treat. She explained that this goal can be advanced by taking advantage of the tremendous amounts of data available both in the health care system and in social media about individuals’ daily activities and how they are living their lives. Kelly said that “we believe we really must advance policies that allow for communities to safely and responsibly utilize the data that is available out there to promote health and identify people who may be early on in their decline before they’re in crisis.”

Kelly suggested four specific areas in which data can be used to improve mental health care, including the prediction and prevention of suicide. The first is to increase care seeking. By using data more precisely, it should be possible to reach out to different segments of the population and “sell” high-quality mental health care programs in ways that people can understand. Sec-

___________________

⁴¹ See https://mmhpi.org/ (accessed August 23, 2022).

ond, data can be used to increase the supply of care by optimizing the mental health care workforce and by creating tools that can match people with more automated care in those cases where automation can effectively substitute for human interactions. Third, integrating technology solutions into processes used by human experts can help them work more efficiently; specifically, by helping clinicians in measuring outcomes and making decisions on delivering care; technology improves clinicians’ ability to get the right care to the right person at the right time in the right place. Fourth, technology and data can be used to create better measures of outcomes in order to ensure the quality of mental health care (Kelly and Schuler, 2021).

Kelly also spoke about the barriers to effectively using data and technology for mental health care. She said that one of the things people in the field regularly hear is that patients are nervous about others accessing their data. However, she said patients often tell researchers that they would actually like for their digital life data to be used and are interested in having things like passive monitoring and smart nudges built into their everyday lives to help them make healthier choices. She said that privacy is a concern among patients, especially teenagers, and it is something that needs to be addressed, but it is not an insurmountable problem. She added that patients are generally interested in their data being used for health purposes as long as they have the final say over who has access to their data and how it is used. So policies that address that particular issue will be essential in helping overcome any resistance from patients about using their data.

Kelly said that applying data and technology in mental health care can also be beneficial for clinicians. Research has shown that clinicians are eager to integrate digital tools and data science into their clinical practice, particularly those tools that optimize their time and provide more patient information than they can get from office visits (Ahmed et al., 2020; Chi et al., 2021; Marwaha et al., 2022). There is also growing interest among clinicians in decision-support systems that they can integrate into their practices to improve their performance and optimize the use of their time, ultimately allowing them to see more patients and treat them more effectively. Technology can also be used to support patients between visits and to allow clinicians to integrate telehealth into their practices. However, Kelly said, clinicians are also reporting that because many of these technological approaches are not reimbursed by health insurance companies or Medicare, “it is more difficult for them to be able to prioritize learning about these new tools and integrating them into their practices.”

In light of these issues, Kelly suggested that it is essential to establish national policies to enable interstate telehealth practice. She said that the federal government instituted policies during the COVID-19 pandemic that allowed an expansion of telehealth, but that those policies are about to expire. Given the emerging evidence that telehealth may be useful in reducing stigma--

associated barriers to accessing mental health care because people do not need an office visit for their sessions, she recommended that the Centers for Medicare & Medicaid Services (CMS) “make permanent the interstate licensure orders that were put in place, and continue to reimburse equally for phone and video appointments.” The phone-based option is particularly important for many underserved communities, she added.

Again, she suggested that it is critical to incentivize the integration of outcome-based digital tools through reimbursement, funding, and research. She called for CMS to develop regulatory and reimbursement mechanisms and pathways for innovations in outcome-based digital tools. Kelly said there has been a great deal of investment in such tools recently, but if they are to be effectively integrated into the care system, policies are needed to categorize and evaluate them for effectiveness, and to determine appropriate payment. These policies need to go beyond the typical fee-for-service model, she added, noting that fee-for-service models fail to incentivize quality and outcomes, so they do nothing to encourage the adoption of new digital tools that improve the quality of care. The existing reimbursement codes “are not a viable means of payment for all digital tools,” she said, “and we really do have to think about different ways that digital tools provide care.” She suggested that CMS establish a public–private partnership with subject-matter experts to help develop effective new policies that will encourage the adoption of digital tools that lead to better mental health outcomes.

Furthermore, Kelly called for establishing policies to harmonize data that exist both inside and outside the health care systems—including data from social media, wearables, federal databases, and health care systems—in order to take full advantage of these data. Specifically, she suggested that the new policies should keep the health and safety of the consumer in mind at all times and that consumers should be part of the policymaking process; that HIPAA should be expanded to include health data outside health care systems; and that the White House Office of Science and Technology Policy should partner with the Department of Health and Human Services to create a task force charged with developing a scalable model for integrating and harmonizing these data to create better health (Kelly and Schuler, 2021).

Opportunities to Improve Data and Enhance Effective Detection and Response

Colin Walsh, associate professor of biomedical informatics, medicine, and psychiatry at the Vanderbilt University Medical Center, spoke about opportunities for public health policy to enhance effective detection and response to suicide risk. In particular, he focused on the gaps in data and the challenges in detection and response to suicide risk.

Walsh observed three types of gaps in the current data. First is data completeness. For instance, he said, Vanderbilt Medical Center has been visited by millions of patients over the decades, but it does not have complete records on all those people. The records Vanderbilt has are mainly from visits those patients had with Vanderbilt physicians; information from health care interactions outside the Vanderbilt system is generally not available in the medical system’s records. This particular gap could be addressed by policies that encourage and regulate information exchange between health care providers. Walsh said that “with the right intention and the right design, organizations can share data in a secure, privacy-preserving way to give us better insight into a crisis like suicide.” The second data gap Walsh observed is lag. For example, Tennessee collects data on hospital discharges around the state, but the data are only reported yearly in an annual report. “That means for a long period of time, we don’t necessarily know what’s going on,” he said. The third data gap that Walsh observed is validity. Researchers rely on large datasets, but the type of structured data often found in datasets—generally letters and numbers that are easy for a computer to read—are not always completely reliable. “They don’t always mean what we hope they mean,” he said, and he offered an illustrative example. When researchers carried out an expert review of 5,543 patients whose records included ICD-9 codes⁴² for suicide and self-inflicted injury, they found that only 3,250 (58.63 percent) had expert-confirmed histories on nonfatal suicide attempts (Walsh et al., 2017). “That is a really big problem,” he said.

Regarding detection challenges, Walsh first discussed equity, noting that “if we don’t think about that right upfront or if we realize there is an issue but we don’t commit to fixing it, we’re going to have specific problems there.” Policymakers could play a role in ensuring that such equity considerations are a priority for people working in this field. A second challenge in effective detection is the rarity of suicide within a given population. Walsh said that “as bad as suicide is and as much as we want to prevent every single one, when you think about the level of an entire population, an entire city, or even an entire health system, it does end up being rare.” He said that, for example, when researchers calculated the likelihood of fatal overdose within 30 days of someone receiving a prescription for an opioid medication among patients in Tennessee, they found that it was only 0.007 percent. “It was extremely rare,” he added. A third type of detection challenge arises from gaps in communication among those involved in suicide risk detection and response—the individuals, their families, emergency medical services, mobile crisis clinicians, social media, and so forth—which is usually challenging and often does not

___________________

⁴² International Classification of Diseases (ICD) is the official system designed to promote international comparability in the collection, processing, classification, and presentation of mortality statistics.

happen. Walsh stressed that the creation of incentives to improve that communication would be an important accomplishment.

Walsh identified four challenges in responding to suicidal behavior. Equity was again the first concern. In designing responses, it is not enough to create a design that is based on what works with one community and assume it will work with other communities, he noted. It is important to understand the needs of different communities, particularly those of underrepresented minorities, when crafting responses, Walsh said. The second response challenge is access. Walsh said that, historically, it has been more difficult to get access to mental health care and to get that care paid for than it has been for physical problems such as heart failure or pneumonia. “That’s a huge gap that I think we need to think about,” he said. The third response challenge is workforce. Many states are experiencing severe workforce shortages in various professional health care disciplines, including behavioral health specialist.⁴³ Furthermore, the workforce shortage exacerbates health disparities. Rural areas and small towns are disproportionally affected compared to major cities. He said that policies are needed to address these shortages and workforce issues. The final response challenge Walsh observed is a lack of care integration. Developing truly effective care will require increased integration and improved transitions among different parts of the mental health care system. “Working as a collaborative, cultivating connection, and building bridges are key themes to think about,” he said.

Walsh mentioned a few initiatives in Tennessee that are attempting to address some of the gaps he described. He said that, for example, the Electronic Surveillance System for the Early Notification of Community-Based Epidemics (ESSENCE) monitors trends in suicidal behavior and risk factors and provides alerts for increases in suicide-related visits to emergency departments. ESSENCE seeks to address two of the data gaps Walsh listed in his presentation, completeness and lag. “The idea here is to streamline and remove friction for individuals to get alerts on what’s going on so people can be more aware of what’s happening in more real time,” he said. In addition, the Tennessee Suicide Prevention Network (TNSPN)⁴⁴ engages various types of participants in suicide prevention, including first responders, clinicians, law enforcement, educators, nonprofit organizations, academics, and suicide survivors and those with lived experience. Walsh said that one of TNSPN’s goal is to increase communication among these various groups and thus address communication gaps in response.

___________________

⁴³ See Mental Health Care Health Professional Shortage Areas, https://data.hrsa.gov/topics/health-workforce/shortage-areas (accessed August 17, 2022).

⁴⁴ Tennessee Suicide Prevention Network is a statewide public–private organization and association of clinical, public health, public safety, clergy, and social actors that provides suicide prevention resources. https://www.tn.gov/behavioral-health/need-help/suicide-prevention/tspn.html (accessed August 23, 2022).

Colleen Carr, director of the National Action Alliance for Suicide Prevention, said that it is important to ground discussions concerning the use of data science in suicide prevention in the perspectives of individuals who have lived experience with suicidal thoughts and attempts (a topic of discussion in Box 4) because the people whose lives will be affected are at the core of efforts to prevent suicide.

REFLECTIONS AND SUGGESTIONS

Sean Joe, the Benjamin E. Youngdahl Professor of Social Development at Washington University in St. Louis, reflected on the workshop presentations and discussions. Joe observed that the discussions during the workshop focused more on detection—that is, identifying an at-risk person in the moment—than on prediction of future risk. That is a reflection of the current state of abilities in the field, he noted.

One overall theme from the discussions, Joe said, is that thanks in large part to advances in AI/ML, the field is poised to make great strides in helping individuals at risk for suicide cope more effectively with their challenges. It is becoming possible to more fully and consistently understand the needs of individuals who might be at risk for suicide, to identify skills that lead to overall improvement in people’s lives, and to design evidence-based and personalized supports. However, he cautioned that in personalizing responses, it will be important to pay attention to such factors as ethnicity and cultural expression. A related capability is helping peers to be more empathetic in their conversations with at-risk individuals on peer-to-peer support sites. AI is being used to inform decision-support systems and to make interventions more effective, he said.

On the other hand, there is relatively little success in predicting which specific individuals are more likely to attempt suicide within a specific time period; that is something that will need additional work. There has been some progress in detecting imminent risk, Joe said, but he emphasized that the traditional approach of assigning individuals to different risk levels is unsatisfactory in many ways; alternative approaches to identify and prioritize individuals who are at most imminent risk of a suicide attempt are needed. It is also important to use novel approaches that combine algorithmic judgments with clinical assessments, he said.

Given the incredible amounts of data concerning individuals’ online behavior, Joe reiterated a number of ethical concerns regarding how those data may be used. For researchers in the medical and behavioral sciences, there are already strict safeguards for privacy and data security in place, but that is not true for data being collected by online sites, such as search engines or social media platforms. He said more thought needs to be given to the ethical principles that should govern these actors, but he stressed that it is vital to give individuals the ability to consent to the use of their own data and that in order for their consent to be meaningful, they need to be given enough information and explanation that they understand which parts of their data will be used, how they will be used, and the potential risks and rewards of using those data. Obtaining trust will be important with various groups, particularly young people and various ethnic groups, and any attempts to address suicide risk in these groups will need buy-in and input from the groups, he said.

Joe noted that the value of data grows not only with its volume but also with its diversity, and it is particularly useful to have repositories that integrate data from local, state, and national levels; such repositories can be useful not only in predictions but also in helping researchers solve longstanding scientific questions regarding risk detection and the importance of timing and interventions.

Going forward, Joe said, it will be vital to use the critical insights that can be gained from working with people who have lived experiences. That experience should inform every aspect of the work to prevent suicide, and those with lived experiences should be involved in decisions on research and standards of care, he added.

Younger generations—Gen Z in particular—have a different, more intimate relationship with social media spaces than older generations, and it will be important to engage them in the process of finding solutions to their mental health challenges. Furthermore, Joe noted, young people especially tend to see mental health in a more holistic way, as part of everything they do; thus any solutions to mental health issues should not attempt to compartmentalize them but rather to reflect and take into account all the different aspects of their lives in both their digital and social communities. This holistic approach to mental health is valuable for everyone, not just young people, he added.

Some of the most promising approaches to dealing with suicide and other mental health issues involve partnerships among industry, academia, and health care, Joe said. Each brings different things to the table: vast troves of data from social media and other internet companies, academic expertise in analyzing data using tools such as natural language processing and ML, and clinical knowledge and experience with mental health conditions. Such partnerships have already produced innovative ideas about crisis detection and supportive responses. Moving forward it will be important to think about how to incentivize and support these partnerships, he added.

In all of these efforts, Joe emphasized that it will be important to plan for equity in results when designing systems involving ML and AI. Because the world itself is not equitable, a process that is itself neutral may not produce equitable outcomes. In particular, the often-implicit goal of efficiency in AI-based systems may conflict with the goal of equity, he said.

Ultimately, the effectiveness of these new technology-based systems to assess and detect suicide risk and to provide interventions will depend to a large extent upon having effective policies governing such things as data ownership and usage, appropriate reimbursement for technology-based mental health care, and interstate medical practice.

Joe concluded that while many gaps remain to be filled—gaps in data, detection, equity, and understanding the progression and pattern of suicide

risk—suicide prevention is possible, and there is a significant role to be played by technology combined with data from social media and other online sources.

REFERENCES