3

Detection, Indications, and Warnings

The workshop’s first panel session featured three presentations addressing different approaches to spotting rare events in their nascent stage. The three speakers were Arvind Satyam, chief commercial officer at Pano AI; T. Charles Clancy, senior vice president, general manager of MITRE Labs, and chief futurist at MITRE; and Elisabeth Paté-Cornell, the Burt and Deedee McMurtry Professor and founding chair of the Department of Management Science and Engineering at Stanford University. Robert Schock, senior fellow at the Center for Global Security Research at Lawrence Livermore National Laboratory and planning committee member, moderated a discussion following the presentations.

HEADING OFF MEGAFIRES WITH ARTIFICIAL INTELLIGENCE

When Arvind Satyam and his colleagues at Pano AI thought about how they could apply artificial intelligence (AI) to the problem of protecting physical spaces and infrastructure from wildfires, they realized that the number of wildfires had not increased dramatically, but the number of megafires has, as a result of hotter, dryer weather, extended for longer periods of time. California, for example, has had six of the ten largest fires in the state’s history occur since August 2020. He noted that researchers studying places like California and the east coast of Australia, where megafires have become a huge problem, have concluded that the single biggest determinant of whether a fire will turn into a large incident is how quickly it is spotted and confirmed and how quickly local fire officials mount a response. It was this realization, he said, that led him and his colleagues to focus on detecting fires when they are small so they could provide actionable intelligence to end users, including fire authorities; emergency response systems; city, state, and federal response agencies; utility companies; and insurance firms.

Today, most incidents come to the attention of fire officials thanks to people calling emergency phone lines, and while those alerts are valuable, they rarely include precise locations that can shave critical minutes off the all-important initial response. Pano AI’s approach was to deploy pairs of ultrahigh-definition rotating cameras and use software to simulate the experience of a person sitting on a traditional fire watchtower. The result is an AI system that analyzes the continuously rotating image to spot texture, movement, and gradients in the images indicative of the smoke plume emitted by even the smallest fire. Training the AI enables it to distinguish between smoke and clouds, even on hazy days, and to do so for different terrains.

Pano AI has deployed this system in California, Colorado, Montana, and Oregon—Australia is on tap to deploy the system in the beginning of 2022¹—and the AI learns from each of these different environments. California alone, for example, has more than 20 microclimates from which AI training can occur. At the time of the workshop, the AI data set contained more than 300 million images, and Satyam said the system’s performance continues to improve as that data set grows. To deal with potential false positives, the company has a fully staffed intelligence center with a team of analysts who can review prior footage to aid in quickly determining if a detection is a true incident.

Satyam pointed out that the multiple camera placements in each location allow the system to triangulate and pinpoint the exact location of the fire. This not only aids the local fire authority, but it enables the local utility to understand where the incident is relative to its transmission lines and enables insurance companies to know where the incident is relative to its insured assets. He added that an optical zoom feature allows local fire officials to actually see flames connected to a smoke detection more than 10 miles from a camera, which lets them determine how to coordinate a response, such as calling in an aerial attack.

On a final note, Satyam said the system can now pull in data from the GOES-16 and GOES-17 weather satellites, and it can also access feeds from other cameras that happen to be in the deployed environments. One goal, he said, is to overlay more infrastructure assets to provide better information for end users.

EXPERIENCES WITH ANTICIPATORY ANALYTICS AND RARE EVENTS

Charles Clancy explained that his projects with IARPA aimed to leverage emerging big data capabilities and advances in AI and machine learning to develop anticipatory analytics for predicting the next war and forecasting events such as a terrorist attack, social unrest, and local election outcomes. The initial effort focused on social media and expanded to include cyber threat data and classified government data.

Clancy explained that a system, which emerged from this work, used time-referenced data from a wide variety of sources, such as the velocity of a particular hashtag on social media, to extract features that would then feed into models that generated the relevant forecasts. The idea was to build models based on the available data and let the system determine which models are generating useful output and which are generating less useful output. Each model creates several candidate forecasted events that feed into a fusion engine, which identifies duplicate forecasts and either suppresses or advances certain forecasts based on quality metrics. The system builds these quality metrics over time from the various models providing the data.

A key feature of the system is that a person hand codes actual ground truths of events that have happened in the targeted class of events and in the targeted region of the world. The coding includes the type, severity, time, and location of these real-world events. The system then learns by comparing forecasted to actual events, accounting for noise. For example, the forecasted event might be off by a day, its location might be off by a few miles, or the severity might have been different. The results of these comparisons go into a scoring system that assesses forecast accuracy and lead time. The goal is to forecast an event days or even a week or more in advance.

The first conclusion from this work, Clancy observed, is that models that rely exclusively on the time-delayed ground truth—so-called base rate models—work well. As a theoretical example, if the goal is to know how many protests are going to happen in Mexico City in a particular month, the easiest thing to predict there will be the same number as happened in the previous month. In essence, the base rate models make sophisticated guesses based on probability distributions.

Typically, Clancy was attempting to use these models to forecast the number of events that would happen during 1 month with a resolution² of 1 day. Improving the forecasts involved determining how to leverage big data to modulate up or down the number of events in a particular class that would occur over the next month. “The unsatisfying part of this is once you have the number of events you expect the next month, you just randomly

___________________

¹ Pano AI, 2020, “Pano AI’s Early Bushfire Detection Technology Gives Southern Cross Forests More Tools to Prevent Bushfires in New South Wales,” release date March 8, 2020, https://newsdirect.com/news/pano-ais-early-bushfire-detection-technology-gives-southern-crossforests-more-tools-to-prevent-bushfires-in-new-south-wales-353650975.

² The reporting resolution of 1 day means time was quantized in to 24-hour periods. Event times were reported as dates, but not dates and times.

generate your forecasts to equal that total, subject to the a priori distribution that you have from your base rate model,” Clancy noted.

As a result, he became concerned that intelligence agencies would be using this system to make actual intelligence assessments and forecast events when the created forecasts were, for the most part, randomly generated according to a probability distribution. The problem is that if the models generate enough events, the matching algorithm will say that most of them are close to an actual event as long as the total number was correct. Clancy observed, “This worked well for predicting any sort of event that had sufficient volume for the law of large numbers to kick in, but if you did not have enough volume of events, this thing works terribly.” In other words, Clancy explained, this approach does not work well for rare events in a meaningful time frame, although it might be able to generate a forecast that 37 years from now, plus or minus 5 years, China will invade Taiwan, to cite a possibility at random.

To handle rare events, Clancy created a separate set of models designed by human experts with subject-matter expertise. These models use “small data” and inject their forecasts into the AI system, but this approach performed no better than having a human executing the same task. This approach, he noted, helped automate some of the processes, but it did not have the ability to piece together disparate facts from multiple data sets in a way that could outperform humans. A lesson for the Intelligence Community is that it is important to get the analysts with subject-matter expertise involved in the development and execution of these projects, because using this type of volumetric approach will not work for rare event forecasting.

In terms of research opportunities going forward, Clancy suggested that advances in AI and machine learning have created more sophisticated approaches for generating synthetic data through generative adversarial networks. Reinforcement learning is also dramatically improving the ability to create agents that can detect events. Toward that end, he is creating synthetic, big data sets with a rare event buried in them to train algorithms to find that event. He also noted the challenge of capturing analyst expertise at scale, which the AI community is tackling by designing algorithms to work with humans rather than assuming that the AI algorithm will arrive at a useful answer by itself. The goal is to make AI part of the analyst team and help the analysts be more efficient in applying their subject-matter expertise to a problem.

Clancy mentioned the large gap in the ability of AI to do contextual and causal reasoning that would enable AI to tackle the small data problem. Some in the AI community have suggested using AI to detect anomalies, but the problem is that an analyst still needs to look at the anomaly to determine whether it is a false positive. Cybersecurity uses this approach, but it generates so many false positives that analysts do not find them useful because it creates too many leads for them to examine. He added that anomaly detection is not likely to be useful for rare events because of the large number of anomalies that occur every day.

RISKS, WARNINGS, AND SIGNALS: A SYSTEMS/RISK ANALYSIS PERSPECTIVE

Elisabeth Paté-Cornell began her presentation by discussing a general model for optimizing a warning threshold so that it does not give too many false positives or false negatives. This stochastic model includes a warning threshold, which generates alerts (true or false), and a critical threshold, above which there is real trouble. Two other components of this model are (1) the effect of memory on response to signals, which occurs when people remember the accuracy of the last alert(s) and (2) the lead time provided. That lead time is the time elapsed between the time when the signal appears because the hazard level has crossed the warning threshold, and the time when the hazard level crosses the critical threshold. If the warning threshold is set too high, the lead time will be short, although there will be fewer false alerts. If it is set too low, there will be more time to respond, but more false alerts.

Optimizing a warning system thus involves balancing the trade-offs among the up-crossing rate of the stochastic process, the memory effects, and the length and use of the lead time, explained Paté-Cornell. The process for doing so is to model first the stochastic process of the system’s evolution (e.g., how the water in a river goes up and down), then to assess the up-crossing rates associated with possible alert thresholds. The next step involves analyzing people’s responses to a warning signal, given what they remember of the accuracy of past warnings and the benefits of the lead time that the signals provide. One can then determine the optimal level of an alert by balancing the trade-off between Type 1 and Type 2 errors (false positives and false negatives). The final step is to

determine the value of the information provided by a warning system in terms of people’s responses and decisions in the face of a hazard.

Paté-Cornell’s first of two examples of how such a model can produce useful warnings involved monitoring satellites in low Earth orbit to avoid collisions with debris, based on the Ph.D. thesis of Richard Kim³ under her supervision. She compared two monitoring systems of different cost and reliability. The U.S. Space Surveillance Network uses more than 20 ground-based optical systems and radars, along with several orbiting satellites, and is considered the more sensitive. The international scientific optical network uses a larger number of telescopes around the world but is less accurate. Together, the two systems send information to satellite operators, who then make tactical decisions about moving a satellite, given a warning signal to avoid a collision.

The issue Paté-Cornell and her co-author explored was to determine which strategic option was better for improving the warning system. As an illustration, they compared the options to spend $100 million to add one sensor to the U.S. Space Surveillance Network or to add 35 sensors to the international optical network at the same cost. To answer that question, they used a Bayesian framework to update the probability of a threat, given the complex message coming from the two systems and the probability of failing to issue a timely alert. Based on the illustrative values that they used to describe the behavior of users, and given the information that they may receive from these networks, it turns out that adding 35 small sensors to the international optical network would provide information that is worth more than the information gained by adding one large sensor to the U.S. Space Surveillance Network.

“This Bayesian framework helps support rational decisions, both tactical and strategic,” said Paté-Cornell, “The value of the monitoring system is based on the costs of the failure risks, and the model provides a basis for comparing two monitoring systems with different capabilities.” She cautioned that their quantitative inputs were merely illustrative and that these results should not be used for making decisions.

The objective of the second example, based on the thesis of Isaac Faber⁴ under her supervision, was to anticipate and prevent catastrophic cyber-attacks by generating early warnings of cyber threats using a hybrid system involving AI and a human operator. The response to a warning signal involves a trade-off between the risk of stopping a legitimate entry and the benefits of preventing an attack—for instance, to protect a hospital’s system from cyber threats.

There are two broad categories of actors, Paté-Cornell explained, those with legitimate access and those with malicious intent who are attempting to pass through a series of cybersecurity gates to reach critical files. These gates are the steps in the sequence of operations that need to be successful for the attack to actually occur (the “kill chain”). The challenge is to develop a gate policy that optimizes the trade-off between blocking threats and allowing legitimate access. The questions are, When should the computer act alone? and When should the human operator intervene? Both the computer and the human expert have access to (imperfect) information from the main stack, which is continuously updated by external information and by experience with various actors. When the system first starts operating on a problem, the human expert makes many of the decisions on whether to open or close a gate, because at that point, the AI system, although it may have access to a lot of information, does not have enough knowledge to process all new signals. Over time, though, the AI learns how to do this, but it can also decide to pass the hand to the human operator when it recognizes that the uncertainties or the possible outcomes are too large.

To train the system, Paté-Cornell and her co-author put 18 honeypots⁵ around the world and got about 600,000 visits to those honeypots. Some came from unique Internet Protocol (IP) addresses, some were on blacklists, and nearly 12,000 had multiple entries. The warning system used the frequencies of those attacks to identify threats. Most of the blacklisted events came from China, with the United States, France, and Russia also accounting for a significant number of the threats. She noted that the honeypots generated a database that issued signals to the

___________________

³ R.H. Kim, 2018, “Managing the Risk of Satellite Collisions: A Probabilistic Risk Analysis of Improving Space Surveillance Systems,” Ph.D. thesis, Stanford University, Stanford, CA, http://purl.stanford.edu/rx304kb4324.

⁴ I.J. Faber, 2019. “Cyber Risk Management: AI-Generated Warnings of Threats,” Ph.D. thesis, Stanford University, Stanford, CA, https://purl.stanford.edu/mw190gm2975.

⁵ A “honeypot” is a cybersecurity mechanism that uses a manufactured attack target to lure cybercriminals away from legitimate targets. They also gather intelligence about the identity, methods, and motivations of adversaries.

defenders about the attackers and sometimes to the attackers about the defenders as well. The raw data from the honeypots were used to update the central database of IP addresses, along with the probability that each entry was from an attacker. Both the AI system and the human agent then used the updated central database to decide which gates to open next.

One takeaway from this example, said Paté-Cornell, is that one can use observed behaviors (along with external information) as signals of threats to inform the decision of the hybrid AI–human cybersecurity system to open or close a gate in a computer system.

She noted that warnings can be qualitatively interpreted, but that quantification with probabilities helps with interpreting warnings and comparing options. This is particularly true when the warning messages are complex, as was the case of satellite monitoring by two systems. She also pointed out that an AI and human hybrid system is helpful when using a database that evolves constantly and when making quick decisions is critical when dealing with attackers who move fast.

DISCUSSION

All three speakers noted the importance of continually updating the database that powers an AI system to reflect new or changing underlying conditions or a better understanding of the factors that can influence the underlying conditions. Satyam pointed out how important this was for the fire detection system because it is deployed in different environments.

When asked how she measures the precision of the AI and human in her system, Paté-Cornell observed that the problems that the system addresses deal with uncertainties, and that the best information to describe them is the probability distributions of the factors involved. These distributions of probability allow rational decisions without pretending that there is a known “precise” (certain) value to the factors involved. The objective is thus to represent these uncertainties as well as possible to support critical decisions.

Nestor Alfonzo Santamaria from the Organisation for Economic Co-operation and Development (OECD) asked Clancy and Paté-Cornell how they escape the trap of giving false confidence in results that have high uncertainty. Clancy replied that models can provide confidence intervals associated with their forecasts and provide the provenance of the underlying data used to produce a forecast. The systems he has built, for example, would provide traceability back to the underlying social media messages that led to deriving a particular conclusion. Analysts could then make their own judgment about whether to agree with the forecast. Santamaria said that approach is sufficient when the analysts are experienced, but he talks to policy makers who are not experts in the field, to which Clancy replied that he has not seen a forecasting system that should be producing policy recommendations. Paté-Cornell added that her goal is to represent uncertainties as best as she can and to inform the decision maker about those uncertainties. It is then up to the decision makers to decide how much risk they are willing to take, based on imperfect information.

Satyam noted that this issue is why it is important to spend time coupling human intelligence on top of information produced by AI and of creating a virtuous loop that ends up producing more accurate results than either AI or human would produce on their own. Christopher Barrett added that in a changing environment, the goal is to build a cohesive, integrated representation of a system that can convey meaningful information to the decision process. The goal is not to generate precise, predictive outcomes.

Both Clancy and Barrett pointed to the challenge of dealing with a system where the data come in so slowly that by the time a system can use that data to make a prediction, the prediction is no longer relevant. One project Clancy worked on involved an emergency department clinician typing into the search box of the electronic health record to find the diagnostic code used to denote that a patient had COVID-19. This real-time data provided a week or two lead time on the typical data that enabled him to improve the fidelity and timeliness of the actionable output that decision makers could use. Both Clancy and Paté-Cornell noted there is often a trade-off between the speed of a response based on less-than-perfect information and what that response would be if there was more time to gather information.

Schock asked the panelists to forecast where the forecasting field will be in 5–10 years. Paté-Cornell replied that she believes there will be progress in how AI systems process information and their ability to make automated

decisions. Satyam noted that as climate change produces more extreme weather events, there will be richer data sets with which to train Pano AI’s systems, and it will be able to integrate more data feeds that will be polling in real time. Clancy expressed his hope that national security events will not increase to the point that there will be more good training data for these models. The synthetic models will improve, though, simply because there will be more computational power available, as well as better detectors that are better grounded in reality.

Jeffrey J. Love from the U.S. Geological Survey (USGS) pointed to the need to test and validate models designed to predict rare events. For that reason, he wonders about the value of models for predicting the future likelihood of nuclear war, since those models cannot be validated. What they might be useful for, Love pointed out, could be to help provide insights into various concepts that might lead to a nuclear event. Along the same lines, David Sweeney from DTRA asked how to train models where there is either no training data or insufficient ground truths. Theodore Plasse, DTRA, interjected that the goal is not to predict when nuclear war might happen. Instead, the goal is to mine data about the equipment and processes that go into making a nuclear bomb to inform a system that could generate predictions about who is trying to make a bomb, the precursor event to the event.

Getting back to Sweeney’s question, Clancy said using synthetic data is one possibility to addressing the unavailability of training data or ground truths. Another potential approach is transfer learning, which involves training the AI on data from an adjacent field and then revising the model using the small amount of data in a new field. He noted that there is a growing number of successes using this approach. A third tactic is to leverage other facets of AI, such as inference and reasoning that are well-developed fields that capitalize on applying analyst expertise to the problem.