National Academies Press: OpenBook
« Previous: 4 Evolution of Test and Evaluation in Future AI-Based DAF Systems
Suggested Citation:"5 AI Technical Risks Under Operational Conditions." National Academies of Sciences, Engineering, and Medicine. 2023. Test and Evaluation Challenges in Artificial Intelligence-Enabled Systems for the Department of the Air Force. Washington, DC: The National Academies Press. doi: 10.17226/27092.
×

5

AI Technical Risks Under Operational Conditions

This chapter will consider the risks of incorporating artificial intelligence (AI) within Department of Defense (DoD) operational systems. These AI-enabled systems have several realistic threats, some based on adversarial AI and others based on the risk in deploying the AI-enabled system in an operational environment. The employment of AI-enabled systems can have significant benefits in augmenting the capabilities of the warfighter, but there are other risks inherent in the use of AI-enabled systems that must be considered. In particular, this chapter answers the second of the committee’s three primary tasks, “consider examples of AI corruption under operational conditions and against malicious cyberattacks.”

5.1 INTRODUCTION

An AI-enabled system includes technical risks that must be considered during the test and evaluation (T&E) of any system that incorporates AI elements. The most likely risks to AI-enabled systems are enabled through cyber access to the AI components, be they the training or operational data, the model, the software implementing the component, or the output of the AI-enabled system. Thus, the first and most important risk is exploiting vulnerabilities in the system to access, manipulate, or deny the elements of the AI component. Therefore, during T&E, the traditional cybersecurity testing should be augmented by attacking the AI component’s availability, integrity, and confidentiality and privacy. These particular attacks are of high consequence to any AI-enabled system as an adversary may be

Suggested Citation:"5 AI Technical Risks Under Operational Conditions." National Academies of Sciences, Engineering, and Medicine. 2023. Test and Evaluation Challenges in Artificial Intelligence-Enabled Systems for the Department of the Air Force. Washington, DC: The National Academies Press. doi: 10.17226/27092.
×

able to exploit cyber vulnerabilities of the system to affect the operation of the AI component. Strong cyber defense is the first line of defense against many (probably most) adversarial attacks. However, the Department of the Air Force (DAF) T&E process cannot test for all potential cyber breaches of AI models. Therefore, in-depth defense is called for, which employs a zero-trust architecture and T&E of AI-based software defects and operational performance degradation.

Beyond the traditional cyber risks that would enable the adversary to access the AI component, other risks are unique to the operation of the AI component. These involve the data supply chain (training and test data), the model, and the manipulation of the domain to control the operation of the AI component. These risks are described in more detail in this chapter. Mitigating these risks may involve new T&E approaches that mimic an adversary’s actions during all phases of the AI-component life cycle to assure that the AI component is resilient during the development, training, deployment, retraining, and adaptation of the AI implementation to different operational environments.

Even with strong cyber defenses in place, the adversary can still attack an air force AI-based system. In particular, physically-based attacks using camouflage, concealment, or deception are possible avenues to biasing, denying, and poisoning training data, training labels, and operational data. Particularly insidious are backdoor attacks, which poison the training data (without changing the training labels) and then use triggers during actual operations to misdirect or degrade the AI models during missions. This motivates the need for DAF red-teaming to uncover these potential attack vectors and their likely effects. DAF-specific adversarial training and detection algorithms that target the vulnerabilities discovered by red-teaming can be incorporated into AI-based systems and tested by DAF T&E processes.

AI-based systems are subject to environmental and adversarial effects that can degrade performance. The current DAF requirements-driven T&E processes can uncover the effect of these attacks if the AI model performance is degraded below the required, acceptable ranges. Thus DAF DT&E that tests performance with hold-out datasets can be employed as a defense against some attacks. In addition, OT&E performance tests can ferret out operational environmental degrading effects, which may motivate model retraining using new datasets.

Some attacks, referred to as backdoor attacks, only manifest once the adversary triggers them. Unfortunately, it is intractable to test against all possible triggers, and techniques that hide triggers to make them difficult to detect by humans and machines have been developed. Thus, the mitigations, in this case, are (a) to make such attacks hard to accomplish (which will be discussed further below) and (b) in the case where the attack intends to degrade overall performance, to build monitor systems that check the AI components during run-time for performance degradation and take appropriate action. The T&E role, in this case, is to cause the

Suggested Citation:"5 AI Technical Risks Under Operational Conditions." National Academies of Sciences, Engineering, and Medicine. 2023. Test and Evaluation Challenges in Artificial Intelligence-Enabled Systems for the Department of the Air Force. Washington, DC: The National Academies Press. doi: 10.17226/27092.
×

AI model to fail through a simulated backdoor attack and then to test the monitor to ensure it detects the deviation.

The DAF T&E process is responsible for uncovering attacks and environmental situations that generally degrade performance. The DAF will have to depend on cyber defenses to broadly restrict adversarial attacks by denying access to data (training and testing) and queries (to protect against inversion and privacy attacks). T&E can also test monitors that look for out-of-spec performance deviations. Finally, T&E can test against threats that red teaming has uncovered to the extent that these threats are detectable. This implies a very close relationship between red-teaming and T&E.

5.2 GENERAL RISKS OF AI-ENABLED SYSTEMS

From a T&E perspective, the areas of an AI-enabled system that must be protected are the availability, integrity, and confidentiality and privacy of the AI system:

  • Availability and integrity of the AI output
  • The integrity of the AI model, data, and software
  • Confidentiality and privacy of the model and training data

Several risks are inherent to using AI in operational environments. Awareness is the first step in mitigating such risks, so the committee discusses here some main challenges that have been identified.

AI is dependent on its training data, and its predictions are only as good as the data it has been trained on. Limited data in some scenarios can lead to inherent biases and risks. For instance, having no training data from below-freezing weather conditions means the AI model will not be accurate in such operational conditions.

Additionally, AI implementations are very sensitive to distribution shifts, which often happen gradually and slowly degrade the performance of an AI element, independent system, or joint cognitive system. This could be caused by slow changes (e.g., in the landscapes in satellite images, degradation of sensors due to dirt, and other factors). Even updates to sensor software can result in distribution shifts in the data. Detecting these distribution shifts is a major challenge in AI.

At the extreme, out-of-distribution (OOD) AI predictions occur when a model is presented with data outside of the distribution it was trained on. Unfortunately, AI models cannot robustly detect if data are OOD; in other words, they do not know what they do not know. Instead, AI models learn to make predictions based on patterns and relationships specific to the training data. As a result, they do not generalize well to novel data, and even a bit of extra signal noise in a sensor collection, which might be invisible to the human eye, can potentially confuse an AI system. For example, in object classification in image data, small perturbations of the image data can create a movement toward a centroid within the ML classifier that will mis-classify

Suggested Citation:"5 AI Technical Risks Under Operational Conditions." National Academies of Sciences, Engineering, and Medicine. 2023. Test and Evaluation Challenges in Artificial Intelligence-Enabled Systems for the Department of the Air Force. Washington, DC: The National Academies Press. doi: 10.17226/27092.
×

the object. This was famously demonstrated by the noise introduced in the classification of stop signs, causing the system to classify the object as a speed limit sign.1

5.3 AI CORRUPTION UNDER OPERATIONAL CONDITIONS

In January 2023, the Office of the Undersecretary of Defense for Policy issued DoD Directive 3000.09, which states that the Director of Operational Test and Evaluation (DOT&E) “[e]valuates whether autonomous and semi-autonomous weapon systems under DOT&E oversight have met standards for rigorous V&V and T&E in realistic operational conditions, including potential adversary action, to provide sufficient confidence that the probability and consequences of failures have been minimized.”2

This latest policy on autonomous weapon systems illustrates the nature of placing adversarial attacks against AI-enabled systems in the context of all threats in realistic operating conditions. Thus, while this report focuses specifically on the question of adversarial attacks against the AI components, it is important to continue to test and evaluate AI-enabled systems against a broad spectrum of adversaries and attacks in a realistic operational context. In addition to development and test datasets, the collection and use of operational data, especially in training AI components, is key to mitigating threats to the AI component and the system as a whole.

The consideration of adversarial AI should be in addition to all traditional cyber threats associated with more traditional integrated systems. Software vulnerabilities, supply chain vulnerabilities, insider threats, network vulnerabilities, denial of service attacks, privilege escalation, and root of trust attacks are just a few of the traditional threats that remain in AI-enabled systems and must be addressed. These threats frequently dominate the new categories of AI corruption based on the incorporation of AI technology. While these novel attacks are important to detect and mitigate today, the ability of an adversary to attack the software and networks of the integrated system may be an even larger risk to the operation.

Also of note is the rapid evolution of attacks against AI-enabled systems. This results from the attention on AI systems in academia, the private sector, and the government. This is similar to the rapid evolution of cyberattacks in the early 2000s when attention was centered on network-enabled systems. The committee would expect adversarial attacks against AI-enabled systems to follow the same pattern; rapid evolution of individual attacks followed by a more comprehensive set of mitigations on attack strategies and consistent policy and taxonomies on AI attacks. Some examples of adversarial attacks on AI are shown in Box 5-1.

___________________

1 K. Eykhold, I. Evtimonv, E. Fernandes, et al., “Robust Physical-World Attacks on Deep Learning Visual Classification,” arXiv:1707.08945, https://arxiv.org/pdf/1707.08945.pdf.

2 N. VanHoudnos, B. Draper, J. Richards, J. Schneider, and N. Carlini, 2022, “DoD Zero Trust Strategy,” Washington, DC: Department of Defense, https://dodcio.defense.gov/Portals/0/Documents/Library/DoD-ZTStrategy.pdf.

Suggested Citation:"5 AI Technical Risks Under Operational Conditions." National Academies of Sciences, Engineering, and Medicine. 2023. Test and Evaluation Challenges in Artificial Intelligence-Enabled Systems for the Department of the Air Force. Washington, DC: The National Academies Press. doi: 10.17226/27092.
×

To properly scope the discussion on AI corruption, it is important to clearly define the concept of AI corruption. Although there is significant literature on various attacks against AI systems, there is no standard definition to date for AI corruption. In this context, the committee defines AI corruption as:

AI corruption is the deliberate or unintentional manipulation of the data, hardware, or software of an AI-enabled system that causes the system to produce missing, inaccurate, or misleading results, to deny or degrade the use of the system, or to force the system to expose hidden information used in the training or configuration of the AI component.

The result of AI corruption is a decrease in the quality attributes of an AI component. This may be in the form of statistical measures such as precision and recall, reduction of a performance envelope required for a mission objective, or in a violation of the system’s security requirements, such as maintaining the secrecy

Suggested Citation:"5 AI Technical Risks Under Operational Conditions." National Academies of Sciences, Engineering, and Medicine. 2023. Test and Evaluation Challenges in Artificial Intelligence-Enabled Systems for the Department of the Air Force. Washington, DC: The National Academies Press. doi: 10.17226/27092.
×

of the training set for the AI component. The source of the AI corruption may be a deliberate cyber or physical attack (e.g., destroying a critical sensor or breaking into the server running the AI component) or a result of accidental or environmental conditions (heavy rain or fog distorting sensor input or fault of the hardware or software supporting the component).

5.4 ATTACK SURFACES FOR AI-ENABLED SYSTEMS

The attack surface of any system is defined by NIST (NIST SP 800-172 from GAO-19-128) as the set of points on the boundary of a system, a system element, or an environment where an attacker can try to enter, cause an effect on, or extract data from, that system, system element or environment. For all systems, AI-enabled or not, this defines a surface that must be secured against threats and tested during T&E. Thus, all traditional testing for adversarial attacks against the attack surface that has been previously defined is still required. Defensive mechanisms that obviate or limit the capability of an adversary to take advantage of an attack surface are also the first line of defense for AI-enabled components.

The traditional cyberattack surface can be considered a starting point for defining an attack surface for an AI-enabled system. The rationale for starting with the cyberattack surface is that the AI-enabled component is a data-driven software system, so it shares much of the same surface for an adversary to disrupt, deny, or degrade the system containing the AI-enabled component, which could also provide access by the adversary to the data or software in the AI component.

In addition to the cyberattack surface, the AI component may enable an attack surface beyond the traditional cyberattack surface. This access is due to the dependencies on data within the deployed environment and in the backend infrastructure, the supply chain of any AI model in the component, and the potential for retraining and adaptation with an adversary-controlled environment. These attacks, as described below, expand the traditional attack surface and should be considered for the T&E of any AI-enabled system.

Vulnerabilities in AI-enabled components may be addressed by limiting adversarial access to this component through traditional separation and protection mechanisms. Thus, removing traditional vulnerabilities and adding robust protections to systems containing AI components can limit an adversary’s ability to influence, deny, degrade, or corrupt the functions of the AI component. In all cases, the limitation of access by the adversary to the component is considered the first line of defense in preventing AI corruption in operational conditions. These include but are not limited to network protections, authentication, and authorization to system functions, data at rest and data at motion protections, distributed system protections, rate limiting to prevent denial of service attacks, and robust

Suggested Citation:"5 AI Technical Risks Under Operational Conditions." National Academies of Sciences, Engineering, and Medicine. 2023. Test and Evaluation Challenges in Artificial Intelligence-Enabled Systems for the Department of the Air Force. Washington, DC: The National Academies Press. doi: 10.17226/27092.
×

sensor and actuator protection. The committee can address the white box-black box considerations in adversarial attacks and defenses.

An AI-enabled system consists of many components—those that are specific to AI and components that are part of a more traditional system supporting other functional components. For example, a system that uses a model for object detection may be part of a larger fix and target system supporting a weapons platform. So, the attack surface of a system that includes AI-enabled components also includes the traditional attack surface of the classic system supporting the AI component. This is especially important in the T&E of the system as a whole, as the details of how the AI component is integrated with the traditional system may expand or limit the entire attack surface, which might lead to AI corruption.

While this definition is relevant for AI-enabled systems, some additional vectors should be considered in the life cycle of the AI component. For example, an AI component relies on sensor data free from adversary manipulation, which may be difficult in operational environments. An adversary may alter the environment to cause the AI component to miss-classify an object, thus attacking the system without going through the traditional attack surface.

Another way an AI-enabled system may be attacked beyond the traditional vulnerabilities is by forcing the AI component to respond to many sensor inputs by exposing some of the data used to train the AI model (known as an inversion attack against the ML system). These attacks are effective even with limited access to the ML model and can expose the limitations of the AI component to the adversary.

Yet another attack that is usually not part of a traditional attack surface is the manipulation of training data to an ML component to cause the model to learn a response beneficial to the adversary in an operational deployment. This can be accomplished not only in the initial training of the model but during retraining when the model is updated to respond to updated environmental conditional post-deployment.

AI model inversion attacks refer to techniques that aim to reverse engineer the internal workings of an AI model. These attacks are a type of adversarial attack in which an attacker seeks to reconstruct the input data or features used to train a model or to generate synthetic inputs that will produce a desired output from the model.

Model inversion attacks are a potential concern because they could allow an attacker to learn sensitive information about the training data or the training process itself, which could exploit the model’s vulnerabilities or craft adversarial examples that can fool the model. Model inversion attacks can be particularly dangerous when applied to models used in high-stakes situations, such as DoD

Suggested Citation:"5 AI Technical Risks Under Operational Conditions." National Academies of Sciences, Engineering, and Medicine. 2023. Test and Evaluation Challenges in Artificial Intelligence-Enabled Systems for the Department of the Air Force. Washington, DC: The National Academies Press. doi: 10.17226/27092.
×

weapons systems or decision support systems, as they could result in incorrect or biased decisions.

Several methods can be used to defend against model inversion attacks, including techniques such as differential privacy to obscure the training data and designing models to be robust against adversarial examples.

In summary, the attack surface of AI-enabled systems encompasses all traditional attack surfaces inherent in software-intensive systems—especially those of ML-enabled systems—but has the additional considerations of AI corruption through the life cycle of the AI system, including the data used in training, test, and operations, and the access to the details in any configuration or model that is part of the AI component.

5.5 RISK OF ADVERSARIAL ATTACKS

It can be useful to divide the risk of adversarial attacks into the different levels of integration of the AI-enabled system. The AI component is tested as a standalone system at the most basic level. At this level, the primary risk of adversarial attack is in the supply chain of the software and data used to construct the AI component. The software may rely on open-source components, usually modified for the specific needs of the DoD mission. These compounds may contain elements contributed by the adversary that have not previously been identified by the open-source community. The adversary may also have identified underlying vulnerabilities in the open-source components that have not yet been publicly released, and these may be incorporated into the delivered DoD component. It is important at the DT&E stage to utilize a well-resourced red team with the most up-to-date attack knowledge to expose any potential vulnerabilities that have become part of the software. This case is no different from any other software component that must be tested, but the complexity of open-source AI solutions may be difficult to thoroughly test. Other types of analysis, such as static and dynamic software testing, would be appropriate to augment the red team approach. Modern DevSecOps software pipelines include such tools and should be considered for complex AI incorporation.

It should also be noted that some legacy languages and software stacks may contain numerous vulnerabilities that enable adversarial access to the key data and software elements of an AI component. The use of modern type-safe languages can help to mitigate some of these potential vulnerabilities. Requirements and subsequent T&E for languages that are resilient to attack are an important mitigation technique for AI-enabled systems. In addition, code generation using large language

Suggested Citation:"5 AI Technical Risks Under Operational Conditions." National Academies of Sciences, Engineering, and Medicine. 2023. Test and Evaluation Challenges in Artificial Intelligence-Enabled Systems for the Department of the Air Force. Washington, DC: The National Academies Press. doi: 10.17226/27092.
×

models such as ChatGPT may produce vulnerable code with unsafe languages such as C. Generation of code in a type safe language can help to mitigate some of these risks.3

During the development of an AI component, especially those related to machine learning (ML), the data used for training and testing the component is an important and critical element of exposure to potential adversarial manipulation. When open-source data are used, even if augmented with specific mission data, the potential for adversarial manipulation of the open datasets to force specific ML behaviors can be difficult to detect and can effectively compromise the ML-enabled system in deployment. Protection of the training and test datasets from adversarial manipulation is an important and specific need for ML components. Note that even exposing the training and test datasets to the adversary without their manipulation can provide the adversary with an important tool to discover operational manipulation techniques that can force the ML-enabled system to fail during deployment, even if these components function properly in DT&E and OT&E. Typically this system is tuned and tested by a data scientist, part of the development team for the ML software component.

One recent advance in defense of ML components is using adversarial training. This augmented dataset specifically attacks the function of the ML component, which can then be used as a training set in the component to drive down that specific behavior. An iterative process of continuing to use adversarial examples and train the component to behave appropriately even in these conditions can add to the robustness of the model. Note, however, that over-training in the adversarial space could create vulnerabilities in the inversion attack of the resulting model, thus exposing which attacks and training sets are used to an adversary with limited access to the resulting deployed system.

Recognizing the planned mission objectives is also important when testing these components. Often the theoretical maximum performance of an ML component is tuned to some imagined optimal point, but operational requirements may dictate a different point to tune the trade-offs inherent in machine learning. For example, it may be necessary to have a high-precision result at the expense of recall for a fix and target application, but for a surveillance mission, a high-recall result may better fit operational needs. This should be recognized during DT&E, shown in Figure 5-1, so that by the time this component is integrated into the overall system, the OT&E will have the optimal mission benefit to the ML component.

___________________

3 J. He and M. Vechev, 2023, “Controlling Large Language Models to Generate Secure and Vulnerable Code,” arXiv:2302.05319, https://doi.org/10.48550/arXiv.2302.05319.

Suggested Citation:"5 AI Technical Risks Under Operational Conditions." National Academies of Sciences, Engineering, and Medicine. 2023. Test and Evaluation Challenges in Artificial Intelligence-Enabled Systems for the Department of the Air Force. Washington, DC: The National Academies Press. doi: 10.17226/27092.
×
Image
FIGURE 5-1 A model development and testing environment. SOURCES: I. Ozkaya, 2021, “What Is Really Different in Engineering AI-Enabled Systems?” Pittsburgh, PA: Carnegie Mellon University Software Engineering Institute. Images re-used with permission from Carnegie Mellon University. First publication: Grace A. Lewis, Stephany Bellomo, Ipek Ozkaya: “Characterizing and Detecting Mismatch in Machine-Learning-Enabled Systems.” WAIN@ICSE. 2021:133–140.

General advice published on testing ML-enabled components consists of the following:4

  • What are you intending to test (and learn)?
  • What logistical challenges might you encounter during testing?
  • What are your biggest sources of risk?
  • What is the meaning behind your metrics?
  • How are you dealing with the scale and level of complexity of your system?
  • How are you evaluating for bias and other unintended behaviors?

Once the systems are integrated into an operational component, there will be additional context for the attack surface. These concern the application interfaces to this component and any user interface mechanism and integrated sensor and

___________________

4 V. Turri, R. Dzombak, E. Heim, N. VanHoudnos, J. Palat, and A. Sinha, 2022, “Measuring AI Systems Beyond Accuracy,” paper presented at AAAI Spring Symposium Series Workshop on AI Engineering: Creating Scalable, Human-Centered and Robust AI Systems, https://doi.org/10.48550/arXiv.2204.04211.

Suggested Citation:"5 AI Technical Risks Under Operational Conditions." National Academies of Sciences, Engineering, and Medicine. 2023. Test and Evaluation Challenges in Artificial Intelligence-Enabled Systems for the Department of the Air Force. Washington, DC: The National Academies Press. doi: 10.17226/27092.
×

actuator components that will interface with the component. At this phase, the software engineer must work with the data scientist to assure that the ML-enabled component is sufficiently protected from these interfaces to assure the adversary will not have a path to manipulate the operation of the ML component. During the life cycle, whether DevOps or traditional waterfall, isolation of the model, test data, and testing tools is necessary to prevent exposure of vulnerabilities specific to the trained model that might be exercised during deployment. In particular, if the adversary manipulates the test data, some important operational scenarios may not be tested properly, and the subsequently deployed system may not function properly during these scenarios. This step is demonstrated in Figure 5-2.

At the level of integration of the ML component into the operational system, the addition of software components and data paths expands the attack surface to include many elements that are likely exposed to the adversary, including the sensor input, data streams, and specific APIs of the system to other systems. In this case, many of the more traditional test procedures for an attack surface are relevant and appropriate with the addition of some specific control over these attack surface elements (e.g., sensor input) by the adversary for the express purpose of causing the ML component to fail. In addition, network security, data protection, encryption, and other classic defenses must be integrated at this point into the operational system and tested as part of the OT&E process.

Finally, the risk of adversarial attack during deployment is increased over the traditional software-intensive systems by creating an additional attack surface

Image
FIGURE 5-2 The operational environment for ML-enabled systems. SOURCE: I. Ozkaya, 2021, “What Is Really Different in Engineering AI-Enabled Systems?” Pittsburgh, PA: Carnegie Mellon University Software Engineering Institute. Images re-used with permission from Carnegie Mellon University. First publication: Grace A. Lewis, Stephany Bellomo, Ipek Ozkaya: “Characterizing and Detecting Mismatch in Machine-Learning-Enabled Systems.” WAIN@ICSE 2021:133–140.
Suggested Citation:"5 AI Technical Risks Under Operational Conditions." National Academies of Sciences, Engineering, and Medicine. 2023. Test and Evaluation Challenges in Artificial Intelligence-Enabled Systems for the Department of the Air Force. Washington, DC: The National Academies Press. doi: 10.17226/27092.
×

in the operational data (sensor and other inputs to the AI component), the life cycle of the AI-enabled component, and the additional vulnerability of model inversion.

5.6 NETWORK SECURITY AND ZERO TRUST IMPLICATIONS

Modern network security within DoD relies on the principles of zero trust (ZT). The framework and approach taken by DoD must be directly applied to all AI-enabled systems during the entire life cycle of these components to address the risks identified above. While not all aspects of AI-enabled systems may be amenable to a ZT approach, the use of ZT where appropriate can decrease the risk of adversarial manipulation of the data or the software of the AI-enabled system. Areas where this may not be applicable might be cases where the sensor network and data collection infrastructure is outside of the ZT boundary for the system (e.g., open data for training purposes). However, in this section, ZT will be discussed where the approach is appropriate and within the system boundary for deployed AI-enabled systems.

Zero trust is the term for an “evolving set of cybersecurity paradigms that move defenses from static, network-based perimeters to focus on users, assets, and resources.” Zero trust uses continuous multi-factor authentication, micro-segmentation, advanced encryption, endpoint security, analytics, and robust auditing, among other capabilities, to fortify data, applications, assets, and services to deliver cyber resiliency. DoD is evolving to become a more agile, more mobile, cloud-supported workforce, collaborating with the entirety of the DoD enterprise, including federal and non-federal organizations and mission partners working on various missions. The zero trust framework will reduce the attack surface, reduce risk, offer opportunities to manage the full range of risks (e.g., policy, programming, budgeting, execution, cybersecurity-specific, and others), and enable more effective data sharing in partnership environments. It will also ensure that any adversary damage is quickly contained and remediated if a device, network, user, or credential is compromised.

As the Zero Trust Strategy states:

Trusted Interoperability Data for Warfighters: Military targeteers need secure access to data at the speed of relevance they can use and trust. Warfighters need to target the right adversaries accurately while minimizing civilian and other casualties. Today, DoD data is often siloed, in impractical formats, and not fully vetted or secured from the point of origin to use. The execution of Zero Trust provides targeteers trusted, tagged, and labeled data so they can confidently employ and share it with trusted partners, assured that the data is protected, secure, and accessed by only the people who need it when they need it, using least privilege principles.5

___________________

5 DoD, 2022, “Zero Trust Strategy.”

Suggested Citation:"5 AI Technical Risks Under Operational Conditions." National Academies of Sciences, Engineering, and Medicine. 2023. Test and Evaluation Challenges in Artificial Intelligence-Enabled Systems for the Department of the Air Force. Washington, DC: The National Academies Press. doi: 10.17226/27092.
×

The benefits of securing AI based on the ZT strategy include user, device, application, data, network, automation, and analytic approaches to securing operational systems. Appropriate implementation of zero-trust capabilities across the life cycle of an AI-enabled system covers much, if not all, of the AI attack surface. This is primarily because the nature of supply chain attacks on the training, test, and validation data for AI models require the same level of authentication and authorization as access to the algorithms and models themselves. This is a fundamental principle of zero trust that all elements of the system, data, network, software, and interfaces be authenticated and authorized by the role of the user or software with access to these capabilities. Attacks against the deployed system are also largely addressed by authentication and authorization of all elements, including sensors and other inputs to the AI-enabled system, to prevent adversarial manipulation of any element in the pipeline of the AI-enabled system.

Stages in securing the network to support operational AI-enabled systems include:

  • Data flow mapping. Define granular control access rules and policies. Support least privilege access through a full survey of IT assets, including all AI components, the data they rely on for training, test, and operation, and the trained models designed and deployed.
  • Macro segmentation. Define software-defined networking (SDN) APIs. Use software-defined networks to isolate network assets and traffic. This will ensure that any network or component corruption is contained in a single segment and does not spread to other networks and functions. Separating ML training and test datasets from operational networks will reduce the attack surface for operational AI-enabled systems and mitigate several specific AI attacks. In addition, using specific APIs to control data flow and access to the AI life cycle and deployed environments will limit the ability of the adversary to propagate attacks to multiple operational systems. Note this is also a trade-off to a dynamic approach to DevSecOps that would enable near real-time updates to operational models based on retraining and updates on changing deployed environments as well as the use of operational data in the development and training of new models. Note that this approach may increase the risk of adversary manipulation of the retraining data to drift the model to be more beneficial to the adversary once the retrained model is deployed. This is an inherent trade-off in retraining with operational data that includes adversarial input.
  • Software-defined networking. Assure that all network transport is tightly controlled and adaptable to conditions that may be under adversarial control. Thus, the use of dynamic network controls, bandwidth, routing, and assurance can be monitored and dynamically adjusted based on any changing conditions.
Suggested Citation:"5 AI Technical Risks Under Operational Conditions." National Academies of Sciences, Engineering, and Medicine. 2023. Test and Evaluation Challenges in Artificial Intelligence-Enabled Systems for the Department of the Air Force. Washington, DC: The National Academies Press. doi: 10.17226/27092.
×
  • Datacenter segmentation. Assuring that access to datacenter resources (including access to data repositories and microservices) is not based on network address or reusable tokens but instead on the continuous authentication and authorization of principles of the access to datacenter services.

Finding 5-1: Existing research on attacks on AI-enabled systems and strategies for mitigating them consider attacks that require unimpeded access to an underlying AI model. These attacks are unlikely to be practical with traditional protections and mitigations inherent in deployed DAF systems.

Finding 5-2: Ongoing research on adversarial attacks on AI-enabled systems focus on performance on benchmark datasets which are inadequate for simulating operational attacks. It appears that as robustness to adversarial attacks is improved, the performance often goes down. Even on benchmark datasets, the trade-off between potential performance reduction and improved robustness is not understood. More importantly, the defenses are designed to thwart known attacks. Such pre-trained defenses are not effective for novel attacks.

Finding 5-3: The impact of adversarial attacks on human-AI enabled systems has not been well understood.

At present, the DoD Zero Trust Strategy is only being implemented on enterprise systems. However, it should also be implemented on all DAF AI-enabled systems. This overarching goal may be done in a series of steps.

Recommendation 5-1: The Department of the Air Force (DAF) should fund research activities that investigate the trade-offs between model resilience to adversarial attack and model performance under operational conditions. This research should account for a range of known and novel attacks whose specific effects may be unknown, but can be postulated based on state-of-the-art research. The research should explore mitigation options, up to and including direct human intervention that ensures fielded systems can continue to function even while under attack. The DAF should also simulate, evaluate and generate defenses to known and novel adversarial attacks as well as quantitatively determine the trade-off between potential loss of performance and increased robustness of artificial intelligence–enabled systems.

Recommendation 5-2: The Department of the Air Force (DAF) should apply the DoD Zero Trust Strategy to all DAF artificial intelligence–enabled systems.

Suggested Citation:"5 AI Technical Risks Under Operational Conditions." National Academies of Sciences, Engineering, and Medicine. 2023. Test and Evaluation Challenges in Artificial Intelligence-Enabled Systems for the Department of the Air Force. Washington, DC: The National Academies Press. doi: 10.17226/27092.
×

The DoD Zero Trust Strategy concludes:

To achieve the DoD Zero Trust Strategic Vision, the Department must pursue the strategic goals outlined above as an enterprise. While this is an enormous task, DoD has already made significant progress. Dating over a decade, DoD has advanced cybersecurity through initiatives such as continuous monitoring, multifactor authentication, and others. The technologies and solutions that create ZT, and the benefits it provides, must become a part of the Department’s lexicon and be accounted for in every plan and operation.

Cybersecurity in the world today is, by definition, a moving target, and while it may move, the concept and the culture will remain the same, even as the Department adapts and refines the strategy. Ongoing and open communication and coordination, underpinned by proper funding and resourcing, are key to the strategy’s success.

The Department’s ability to protect, and by extension, DoD personnel against the array of increasingly sophisticated cybersecurity threats depends on it.

5.7 ROBUST AND SECURE AI MODELS

A common approach for increasing the robustness and security of AI models is the incorporation of monitoring or watchdog systems that compare the output of an AI-enabled system to pre-defined operational limits. Should the AI system stray from these operations limits, the external monitoring will take control and prevent the system from drifting beyond these predefined limits. This is similar to guardrails that OpenAI has placed on its ChatGPT system to prevent this system from abuse or offensive results during its use.

Recent studies have shown that the incorporation of guardrails on large language models and similar neural-network AI systems may lead to inaccurate results. For the most recent GPT–4 release, a comparison of the calibration curve of the model prior to the guardrails versus after leads to a significant reduction in the correctness of the results, as shown in Figure 5-3.6 The figure demonstrates the trade-off of the accuracy of the base model (first chart) where provided answers are correct as they become more available. The second chart demonstrates that with the application of guardrails within the models, the availability of answers [P(answer)] have a lower probability of correctness within the critical range of 0.4–0.8 P(answer). This indicates that with the guardrails, the likelihood of incorrect answers (associated with hallucination) is much higher with the same availability of the answer than the base model without the guardrails. The feasibility of guardrails and monitoring of AI-enabled systems is an area of ongoing research, including monitoring to detect attacks and ensure recoverability of the system.

___________________

6 OpenAI, 2023, “GPT-4 Technical Report,” arXiv:2303.08774, https://arxiv.org/abs/2303.08774.

Suggested Citation:"5 AI Technical Risks Under Operational Conditions." National Academies of Sciences, Engineering, and Medicine. 2023. Test and Evaluation Challenges in Artificial Intelligence-Enabled Systems for the Department of the Air Force. Washington, DC: The National Academies Press. doi: 10.17226/27092.
×
Image
FIGURE 5-3 A comparison of the calibration curve of GPT-4 prior to and after the incorporation of guardrails. SOURCE: OpenAI, 2023, “GPT–4 Technical Report,” arXiv:2303.08774, https://arxiv.org/abs/2303.08774.

Robustness is the property that a software component (that includes AI-enabled components) can meet mission requirements with variations in the operational environment.7 Robustness is measured and tested during T&E with the introduction of deliberate perturbations of the environment beyond the initial configuration and training of the software component. This may be an introduction of environmental variations (weather, background, noise, etc.) or variations in detected signals (new objects, other sensory inputs, or variations on decision support). If the SUT passes these variations, it is deemed to be robust. Note the expectation is that this is similar to stress testing hardware systems to determine the performance envelope of the actual delivered system. This testing also enhances justified confidence (see below) as it can generate a more specific set of operational environments and constraints that is communicated to the operator during deployment.

5.8 RESEARCH IN T&E TO ADDRESS ADVERSARIAL AI

To enable DAF T&E, it will be important to distinguish between practical attacks by near-peer adversaries and academic attacks that would be impractical in deployed systems. In particular, attacks that require unimpeded access to an underlying AI model are unlikely to be practical with traditional protections and mitigations inherent in deployed DAF systems. Nevertheless, as stated

___________________

7 Variations regarding adversarial attacks were discussed in Section 3; in this section robustness also includes both adversarial and non-adversarial variations in the environment.

Suggested Citation:"5 AI Technical Risks Under Operational Conditions." National Academies of Sciences, Engineering, and Medicine. 2023. Test and Evaluation Challenges in Artificial Intelligence-Enabled Systems for the Department of the Air Force. Washington, DC: The National Academies Press. doi: 10.17226/27092.
×

above, the use of cybersecurity vulnerabilities to reach AI components will continue to be a primary attack vector for the foreseeable future, and research in the mitigation of these vulnerabilities will be increasingly important to AI-enabled systems.

However, there are other AI-specific attacks that do not require unimpeded access to the AI model, its data, or software. These may include model inversion attacks and environmental manipulation attacks that exploit the mechanisms of the AI component without direct access. Even attacks that require unimpeded access to the AI model may be successful due to the transferability of adversarial attacks. Attacks that can be implemented on more accessible models may then be used to attack the target model. Future research to identify and mitigate these attacks should be a priority for DAF T&E.

For identified attacks against AI systems either through cyber vulnerabilities or through manipulation of the data input and model behavior, research to identify these attacks as they are happening in near real time may allow mitigation through response actions. In addition, this traditional approach to intrusion detection and response can include specific attack characterizations of adversarial AI. Research along these lines continues and will be important to include in T&E activities.

Other mitigations, such as external observation and fencing of AI behavior, can also be used to identify adversarial AI and must also be a part of DAF T&E.

Robustness (i.e., graceful degradation) and resilience (countering the effects when detected) against both natural and adversarial corruptions and performance losses are vibrant areas of academic and industry research and constitute an integral part of the OUSD(R&E) trusted AI thrust. Hence, we should see accelerated future progress that the Air Force can exploit. This is discussed briefly in Section 6.1.

The DAF should not just be a fast follower of private sector research and development (R&D) in this area but should prototype advanced applications for DAF-specific situations and systems and should pursue a few key research vectors that are perhaps not as important in academic and industrial settings. Based on the information gathered by the committee, the following R&D thrusts are particularly important:

First, general red teaming R&D of potential adversary attacks for a few distinct scenarios:

  1. The scenario where the adversary does not compromise our cyber security but can use camouflage, concealment, and denial to influence model training and achieve model evasion or performance degradation.
  2. The scenario where the adversary attempts to compromise our cyber security and then uses a variety of adversarial attacks.
Suggested Citation:"5 AI Technical Risks Under Operational Conditions." National Academies of Sciences, Engineering, and Medicine. 2023. Test and Evaluation Challenges in Artificial Intelligence-Enabled Systems for the Department of the Air Force. Washington, DC: The National Academies Press. doi: 10.17226/27092.
×

Second, blue teaming of detection and mitigation of the above, and counter AI R&D. Since the adversary may be vulnerable to similar attacks, we need to keep this work at appropriate levels of classification. We should also consider “battle reserve” models as an approach.

There are several promising areas of research that will improve the mitigation of adversarial AI including:

  • Techniques for data sanitization. In many cases, sensitive data would need to be sanitized to prevent training data from exposure during model inversion or when testing new types of AI-enabled systems. However, current approaches to sanitization do not effectively support the trade-off between the effective training of the system and the possibility of leaking sensitive data. research into new techniques for data sanitization is necessary to resolve this trade-off. The same research may be used in areas such as data privacy, where specific personally identifiable information (PII) or other sensitive data may be used in the training of an AI-enabled system.
  • Quantifiable uncertainty (QU). DAF systems and operations models should be inherently capable of reporting QU. Through thorough testing, QU metrics should be sufficiently documented to be used confidently in operational contexts or have external monitors or guardrails of performance integrated into their deployed systems. Research into approaches to model-inherent QU is a rich area of enquiry.
  • Certifiable robustness (CR). The main issue with CR in the recent past is that techniques that work only apply to rather restrictive cases, tend to degrade performance, and often require an inordinate amount of computation. However, recent innovations are showing progress. As Salman et al. (2021), write:

Certified patch defenses can guarantee robustness of an image classifier to arbitrary changes within a bounded contiguous region. But, currently, this robustness comes at a cost of degraded standard accuracies and slower inference times. The committee demonstrates how using vision transformers enables significantly better-certified patch robustness that is also more computationally efficient and does not incur a substantial drop in standard accuracy. These improvements stem from the inherent ability of the vision transformer to gracefully handle largely masked images.8

___________________

8 H. Salman, J. Saachi, E. Wong, and A. Madry, 2022, “Certified Patch Robustness via Smoothed Vision Transformers,” Pp. 15137–15147 in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA: IEEE Computer Society and CVF Computer Vision.

Suggested Citation:"5 AI Technical Risks Under Operational Conditions." National Academies of Sciences, Engineering, and Medicine. 2023. Test and Evaluation Challenges in Artificial Intelligence-Enabled Systems for the Department of the Air Force. Washington, DC: The National Academies Press. doi: 10.17226/27092.
×

Conclusion 5-1: Promising areas of research that will improve the mitigation of adversarial AI include techniques for data sanitization, quantifiable uncertainty, and certifiable robustness.

Additionally, Chapter 6 discusses additional emerging AI technologies and promising areas of research. Thus, the DAF should invest in further R&D at both the foundational level and at the applied level, in particular the DAF in the use of these techniques to DAF AI models/AI-enabled systems.

Suggested Citation:"5 AI Technical Risks Under Operational Conditions." National Academies of Sciences, Engineering, and Medicine. 2023. Test and Evaluation Challenges in Artificial Intelligence-Enabled Systems for the Department of the Air Force. Washington, DC: The National Academies Press. doi: 10.17226/27092.
×
Page 102
Suggested Citation:"5 AI Technical Risks Under Operational Conditions." National Academies of Sciences, Engineering, and Medicine. 2023. Test and Evaluation Challenges in Artificial Intelligence-Enabled Systems for the Department of the Air Force. Washington, DC: The National Academies Press. doi: 10.17226/27092.
×
Page 103
Suggested Citation:"5 AI Technical Risks Under Operational Conditions." National Academies of Sciences, Engineering, and Medicine. 2023. Test and Evaluation Challenges in Artificial Intelligence-Enabled Systems for the Department of the Air Force. Washington, DC: The National Academies Press. doi: 10.17226/27092.
×
Page 104
Suggested Citation:"5 AI Technical Risks Under Operational Conditions." National Academies of Sciences, Engineering, and Medicine. 2023. Test and Evaluation Challenges in Artificial Intelligence-Enabled Systems for the Department of the Air Force. Washington, DC: The National Academies Press. doi: 10.17226/27092.
×
Page 105
Suggested Citation:"5 AI Technical Risks Under Operational Conditions." National Academies of Sciences, Engineering, and Medicine. 2023. Test and Evaluation Challenges in Artificial Intelligence-Enabled Systems for the Department of the Air Force. Washington, DC: The National Academies Press. doi: 10.17226/27092.
×
Page 106
Suggested Citation:"5 AI Technical Risks Under Operational Conditions." National Academies of Sciences, Engineering, and Medicine. 2023. Test and Evaluation Challenges in Artificial Intelligence-Enabled Systems for the Department of the Air Force. Washington, DC: The National Academies Press. doi: 10.17226/27092.
×
Page 107
Suggested Citation:"5 AI Technical Risks Under Operational Conditions." National Academies of Sciences, Engineering, and Medicine. 2023. Test and Evaluation Challenges in Artificial Intelligence-Enabled Systems for the Department of the Air Force. Washington, DC: The National Academies Press. doi: 10.17226/27092.
×
Page 108
Suggested Citation:"5 AI Technical Risks Under Operational Conditions." National Academies of Sciences, Engineering, and Medicine. 2023. Test and Evaluation Challenges in Artificial Intelligence-Enabled Systems for the Department of the Air Force. Washington, DC: The National Academies Press. doi: 10.17226/27092.
×
Page 109
Suggested Citation:"5 AI Technical Risks Under Operational Conditions." National Academies of Sciences, Engineering, and Medicine. 2023. Test and Evaluation Challenges in Artificial Intelligence-Enabled Systems for the Department of the Air Force. Washington, DC: The National Academies Press. doi: 10.17226/27092.
×
Page 110
Suggested Citation:"5 AI Technical Risks Under Operational Conditions." National Academies of Sciences, Engineering, and Medicine. 2023. Test and Evaluation Challenges in Artificial Intelligence-Enabled Systems for the Department of the Air Force. Washington, DC: The National Academies Press. doi: 10.17226/27092.
×
Page 111
Suggested Citation:"5 AI Technical Risks Under Operational Conditions." National Academies of Sciences, Engineering, and Medicine. 2023. Test and Evaluation Challenges in Artificial Intelligence-Enabled Systems for the Department of the Air Force. Washington, DC: The National Academies Press. doi: 10.17226/27092.
×
Page 112
Suggested Citation:"5 AI Technical Risks Under Operational Conditions." National Academies of Sciences, Engineering, and Medicine. 2023. Test and Evaluation Challenges in Artificial Intelligence-Enabled Systems for the Department of the Air Force. Washington, DC: The National Academies Press. doi: 10.17226/27092.
×
Page 113
Suggested Citation:"5 AI Technical Risks Under Operational Conditions." National Academies of Sciences, Engineering, and Medicine. 2023. Test and Evaluation Challenges in Artificial Intelligence-Enabled Systems for the Department of the Air Force. Washington, DC: The National Academies Press. doi: 10.17226/27092.
×
Page 114
Suggested Citation:"5 AI Technical Risks Under Operational Conditions." National Academies of Sciences, Engineering, and Medicine. 2023. Test and Evaluation Challenges in Artificial Intelligence-Enabled Systems for the Department of the Air Force. Washington, DC: The National Academies Press. doi: 10.17226/27092.
×
Page 115
Suggested Citation:"5 AI Technical Risks Under Operational Conditions." National Academies of Sciences, Engineering, and Medicine. 2023. Test and Evaluation Challenges in Artificial Intelligence-Enabled Systems for the Department of the Air Force. Washington, DC: The National Academies Press. doi: 10.17226/27092.
×
Page 116
Suggested Citation:"5 AI Technical Risks Under Operational Conditions." National Academies of Sciences, Engineering, and Medicine. 2023. Test and Evaluation Challenges in Artificial Intelligence-Enabled Systems for the Department of the Air Force. Washington, DC: The National Academies Press. doi: 10.17226/27092.
×
Page 117
Suggested Citation:"5 AI Technical Risks Under Operational Conditions." National Academies of Sciences, Engineering, and Medicine. 2023. Test and Evaluation Challenges in Artificial Intelligence-Enabled Systems for the Department of the Air Force. Washington, DC: The National Academies Press. doi: 10.17226/27092.
×
Page 118
Suggested Citation:"5 AI Technical Risks Under Operational Conditions." National Academies of Sciences, Engineering, and Medicine. 2023. Test and Evaluation Challenges in Artificial Intelligence-Enabled Systems for the Department of the Air Force. Washington, DC: The National Academies Press. doi: 10.17226/27092.
×
Page 119
Suggested Citation:"5 AI Technical Risks Under Operational Conditions." National Academies of Sciences, Engineering, and Medicine. 2023. Test and Evaluation Challenges in Artificial Intelligence-Enabled Systems for the Department of the Air Force. Washington, DC: The National Academies Press. doi: 10.17226/27092.
×
Page 120
Next: 6 Emerging AI Technologies and Future T&E Implications »
Test and Evaluation Challenges in Artificial Intelligence-Enabled Systems for the Department of the Air Force Get This Book
×
 Test and Evaluation Challenges in Artificial Intelligence-Enabled Systems for the Department of the Air Force
Buy Paperback | $42.00 Buy Ebook | $33.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

The Department of the Air Force (DAF) is in the early stages of incorporating modern artificial intelligence (AI) technologies into its systems and operations. The integration of AI-enabled capabilities across the DAF will accelerate over the next few years.

At the request of DAF Air and Space Forces, this report examines the Air Force Test Center technical capabilities and capacity to conduct rigorous and objective tests, evaluations, and assessments of AI-enabled systems under operational conditions and against realistic threats. This report explores both the opportunities and challenges inherent in integrating AI at speed and at scale across the DAF.

READ FREE ONLINE

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!