National Academies Press: OpenBook
« Previous: Summary
Suggested Citation:"1 Introduction." National Academies of Sciences, Engineering, and Medicine. 2023. Test and Evaluation Challenges in Artificial Intelligence-Enabled Systems for the Department of the Air Force. Washington, DC: The National Academies Press. doi: 10.17226/27092.
×

1

Introduction

1.1 A CENTRAL QUESTION: HOW TO ACHIEVE SUFFICIENT CONFIDENCE IN AI-ENABLED SYSTEMS?

The Department of the Air Force (DAF) is in the early stages of incorporating modern artificial intelligence (AI) technologies into its systems and operations. The integration of AI-enabled capabilities across the DAF will accelerate over the next few years. As demonstrated by experiences in commercial industry, the DAF will face new opportunities and challenges in integrating AI at scale.

AI is different from aircraft, missiles, and other weapons and support systems, with which the DAF has decades of experience in testing and evaluation. Existing T&E processes and procedures do not translate directly to software capabilities, especially AI’s data-centric, black-box, self-learning, adaptive, and probabilistic characteristics. As a result, it is harder to gain buy-in from the DAF, DoD, public, and international communities for and sufficient confidence in AI-enabled capabilities absent the same kind of testing policies and processes for AI implementations that have guided flight testing for the past 70 years. While similarities between traditional and AI T&E mean that the DAF is not starting from scratch, the substantial differences between them make it imperative that the test community develop and promulgate AI-specific T&E policies and procedures as soon as possible.

The complexity of AI T&E is amplified by the inevitability of a future in hybrid weapons systems, including a combination of legacy non-AI systems, new non-AI systems, current or legacy systems with AI that are “bolted on,” and AI that is “baked-in”—all of which may be operating together simultaneously.

Suggested Citation:"1 Introduction." National Academies of Sciences, Engineering, and Medicine. 2023. Test and Evaluation Challenges in Artificial Intelligence-Enabled Systems for the Department of the Air Force. Washington, DC: The National Academies Press. doi: 10.17226/27092.
×

Moreover, the T&E of AI-enabled systems must account for the cascading effects of multiple AI-enabled systems interacting across weapon systems, C2 architectures, and cyber networks.

One may argue that “the purpose of test, evaluation, verification, and validation (TEVV) . . . is the activity that produces the evidence that completes the needed assurance arguments.”1 Thus, while considering AI T&E requirements and the above factors, a central question becomes clear: How to achieve sufficient confidence in AI-enabled systems?

Similarly, what level of T&E is necessary and sufficient throughout an AI-enabled system’s entire life cycle to ensure the delivery of effective, suitable, reliable, predictable, sustainable, secure, safe, trustworthy, and resilient capabilities?

As described in this report, the answer to this question will likely be considerably different for AI-enabled systems than for T&E of traditional hardware systems. It will be context-dependent, reflecting a combination of factors such as the degree of urgency; end-user requirements or operational imperatives; technology and human readiness levels (TRLs/HRLs); risks, such as threats, opportunity costs, and potential unintended consequences; scope; scale, and required levels of predictability, reliability, explainability, and transparency. While considering these factors, the DAF should be guided, though not unduly constrained by the precautionary principle—introducing a new product or process whose ultimate effects are disputed or unknown should be approached using caution, pause, and review.

Ultimately, the answer to how much testing is necessary and sufficient is defined as much by the end-user or operator as by the developers and the responsible DAF T&E organization. In all cases, end-users will assess the performance of AI-enabled capabilities relative to a given system’s baseline (pre-AI) performance. This report focuses on three main tasks that the study committee was tasked with, which will help the DAF address the fundamental question of how much testing is enough.2

1.2 STUDY QUESTIONS TO BE ADDRESSED

The study committee was tasked with conducting this consensus study to examine the Air Force Test Center’s (AFTC’s) technical capabilities and capacity to conduct rigorous and objective tests, evaluations, and assessments of artificial

___________________

1 D.M. Tate, 2021, “Trust, Trustworthiness, and Assurance of AI and Autonomy,” Institute for Defense Analysis, https://apps.dtic.mil/sti/trecms/pdf/AD1150274.pdf, p. iv.

2 As well as what kind of testing is necessary—see, for example, R. Burnell, W. Schellaert, J. Burden, et al., 2023, “Rethink Reporting of Evaluation Results in AI,” Science 380:136–138, https://doi.org/10.1126/science.adf6369.

Suggested Citation:"1 Introduction." National Academies of Sciences, Engineering, and Medicine. 2023. Test and Evaluation Challenges in Artificial Intelligence-Enabled Systems for the Department of the Air Force. Washington, DC: The National Academies Press. doi: 10.17226/27092.
×

intelligence (AI)-enabled systems under operational conditions and against realistic threats. Specifically, the committee was asked to address three tasks:

Task 1 asks the committee to evaluate and contrast current testing and assessment methods employed by the DAF and in commercial industry. This is discussed in more detail in Chapters 3 and 4.

Task 2 asks the committee to consider examples of AI corruption under operational conditions and against malicious cyberattacks. This is discussed in more detail in Chapter 5.

Task 3 asks for recommendations promising areas of science and technology that may lead to improved detection and mitigation of AI corruption. This is discussed further in Chapter 6.

While the committee set out to maintain a narrow scope driven by the specified tasks, as the study progressed, it became clear that the impact of the findings and recommendations was more significant than anticipated. Central to the study’s assessment of AFTC’s technical capabilities is the capacity to deploy these new systems with the same rigor and discipline they have applied historically with traditional systems. To ensure the same level of trust that the test community has rightly earned from the space, cyber, and air forces, the processes, requirements, and culture of the test community and the DAF, in general, will need to evolve and adapt. These adjustments will be necessary to accommodate the developmental differences in AI-enabled systems.

1.3 WHAT DO WE MEAN BY “ARTIFICIAL INTELLIGENCE”?

Artificial intelligence (AI) is a broad term that means different things to different people. For example, AI can be broadly defined as a computing system that can perform tasks that are normally associated with human intelligence, such as conversing in a natural language, solving problems, and recognizing types of objects in a scene, etc. Generally, AI is defined to include all such tasks that a computing system can perform at human or near human proficiency levels. The ability to learn is also an important aspect of any intelligent system. Recently, major advances have occurred in the field of machine learning, leading to an increase in proficiency across virtually all current AI tasks. Thus, to many, AI and ML are often used synonymously today, although machine learning is just one, albeit very important subset of AI implementation.

For the purposes of discussing test and evaluation (T&E), the committee divides AI implementations into three broad categories: element, independent system, or joint cognitive system.

  • The implementation may be an element of a program or system that uses an artificial intelligence algorithm or knowledge structure. Examples are a route planner, ground avoidance, machine learning for target detection.
Suggested Citation:"1 Introduction." National Academies of Sciences, Engineering, and Medicine. 2023. Test and Evaluation Challenges in Artificial Intelligence-Enabled Systems for the Department of the Air Force. Washington, DC: The National Academies Press. doi: 10.17226/27092.
×
  • It may be an independent system, sometimes called a stand-alone, turnkey, or engineering system, that uses one or more intelligent elements coupled with other components (e.g., user interfaces) to produce results for a well-specified problem domain. Examples include Project Maven, logistic, and recommender systems.
  • An implementation might be a joint cognitive system, where one or more elements or systems are coupled with additional elements supporting human-AI interaction solving a joint problem or task. Neither the human nor the engineered system is capable of individually achieving the desired outcomes. An example is the aircraft itself, where neither the pilot and nor the engineered components can with multiple heterogenous AI elements achieve the mission without the other.

In each of the three cases, the breadth of the intelligence is bounded by the types of problems of interest. Creating an optimal route planner is similar to a savant—someone that is extraordinarily smart, but only about some specialized field. The intelligence for an independent system is likewise specialized for the purpose, though the AI elements and the integration may be more sophisticated than a single element and constitute a systems-level form of intelligence. A joint cognitive system may be focused on only one problem or mission but solving that problem or executing that mission requires true interaction with a human operator to specify objectives, dynamically delegate and reacquire authority, supervise, coordinate, etc. A joint cognitive system must include social intelligence to support the interaction with the human. For example, the introduction of natural language chatbots to a system interface may lead the human to erroneously expect the system to behave as a joint cognitive system, further complicating T&E.

The system complexity of each of the three categories varies as well. An element is not a system in the traditional NASA engineering system sense and thus, while the algorithm may be quite sophisticated and pose its own T&E challenges, there are few complications for testing and evaluation due to concomitant hardware, user interfaces, other software, etc. The enabling intelligence in an independent system is much harder to discern because it is embedded in a larger engineering construct. The joint cognitive system category presents the hardest case as the system is a system-of-systems with the human operator as one of those systems.

A third dimension distinguishing the categories is the type of user interaction. An element would typically have minimal user interaction, as it exists as an embedded component of a system. A pilot may have a user interface for an independent system but might only be able to turn off or ignore the output from an element, assuming its contribution was obvious and accessible—for example, turning off the ground avoidance function. Joint cognitive systems differ from independent systems in that the user interaction for an independent system is generally through

Suggested Citation:"1 Introduction." National Academies of Sciences, Engineering, and Medicine. 2023. Test and Evaluation Challenges in Artificial Intelligence-Enabled Systems for the Department of the Air Force. Washington, DC: The National Academies Press. doi: 10.17226/27092.
×

user interfaces or fixed protocols; a joint cognitive system involves give-and-take interactions between the human and the computer, and introduces many of the human-machine teaming considerations discussed in Section 2.4.

These categories, summarized in Table 1-1, illustrate why testing and evaluation of AI is not a one-size-fits-all endeavor. Although the categories appear to restate decades of work in function (or unit), systems, and systems-of-systems testing, the breadth of intelligence, systems complexity, user interaction, and potential for both engineering and human error illustrates why AI imposes new demands on T&E. For example, a route planning algorithm element is straightforward to prove correctness as well as time and memory constraints but a convolutional neural

TABLE 1-1 Categories of Artificial Intelligence Implementation

Intelligence System Complexity End-User Interaction Examples Ramifications for Testing
Element Computation produces an exceptional, but narrow, skill or result Little, generally producing a binary output or exists as a unit within a system None, beyond either accepting or rejecting output Route, ground avoidance, target detection, Bayesian models Still involves functional or unit testing in isolation but depends on the unique vulnerabilities of the algorithms
Independent System One or more elements coupled with other components to produce results for a well-specified problem domain Generally, a function of the hardware, software, and application; involves combinations of different AI elements Restricted; humans work through an interface or rigid protocols Project Maven, logistics, recommender systems Builds on systems testing principles but with multiple heterogenous AI elements
Joint Cognitive System One or more systems coupled with additional social intelligence supporting human-AI interaction for a well-specified problem domain System includes true human interaction Fluid, the interaction mechanisms and partitioning of roles must dynamically support team behavior Pilot-aircraft System-of-systems, which includes human-machine cognitive testing
Suggested Citation:"1 Introduction." National Academies of Sciences, Engineering, and Medicine. 2023. Test and Evaluation Challenges in Artificial Intelligence-Enabled Systems for the Department of the Air Force. Washington, DC: The National Academies Press. doi: 10.17226/27092.
×

network element for target detection has much different structure and vulnerabilities. An independent system may involve intelligence to coordinate the intelligent components, complicating the already challenging systems testing landscape, as well as the issues such as human trust. A joint cognitive system is vulnerable to subtle mismatches between human and computer capabilities which are difficult to anticipate and simulate.

Broader applications of AI, such as artificial general intelligence, lie beyond the scope of this study. Furthermore, while some sections of the report focus on machine learning or large language models as prominent examples of AI, these are just examples—they are not, and should not be considered, the only target of this report.

Given the diversity of implementations, meanings, categories, and scope of applicability of the term AI, this study has used the general term AI without applying a series of qualifications each time the term is used. The report uses AI to refer variously to AI elements, independent systems, and joint cognitive systems. The report does not always call out the specific scope of the AI to which it refers in every instance, but the committee trusts that the context will make the meaning clear to the reader.

1.4 CURRENT STATE OF THE ART OF AI

The foundations for AI were laid through a seminal paper by Alan Turing and a 1956 summer workshop held at Dartmouth attended by some of the best-known researchers in the country. In the 1960s and 1970s, AI was focused on developing efficient search algorithms such as A* and playing games such as checkers, chess, etc. In the mid-1980s, uncertainty models were introduced as Bayesian networks and associated inference algorithms based on enumeration and variable elimination. In the 1980s, researchers pursued neural networks, which, although promising, did not lead to significant progress in AI then. Domain knowledge was the key contributor to the development of AI in the 1970s and 1980s. Starting in the 1990s, an explosion of automated data collection and computational processing power helped to seed a new age of data-driven AI. Since 2012, with the reemergence of deep learning algorithms (an expansion of the neural net concept), much of what is known as AI is based on learning from data using supervised, unsupervised, semi-supervised, or weakly supervised techniques, or reinforcement learning. More recently, generative AI models such as generative adversarial networks, diffusion models, and neural radiance fields have been used for generating synthetic data that can be used for deep learning. Over the decade, the main applications of AI have been in game playing, computer vision, natural language processing, advertising and marketing, and robotics. Figure 1-1 describes these developments.

Suggested Citation:"1 Introduction." National Academies of Sciences, Engineering, and Medicine. 2023. Test and Evaluation Challenges in Artificial Intelligence-Enabled Systems for the Department of the Air Force. Washington, DC: The National Academies Press. doi: 10.17226/27092.
×
Image
FIGURE 1-1 The history of AI.

1.5 CURRENT STATE OF THE PRACTICE OF AI IN THE DAF

The DAF is in the early stages of incorporating modern AI technology into its systems and operations. Through interviews with Air Force T&E leadership (full meeting agendas are available in Appendix C), the committee ascertained that, possibly apart from classified programs,3 the Air Force has not yet acquired any modern AI capability within the standard acquisition processes for a major defense acquisition program (MDAP) or major automated information system (MAIS). AI-related projects to date have been research and development initiatives or proof-of-concept demonstrations or have been integrated into existing systems as upgrades or prototypes.

According to the Government Accountability Office (GAO) February 2022 report Artificial Intelligence: Status of Developing and Acquiring Capabilities for Weapon Systems (Table 1-2), as of April 2021, the Air Force had funded 80 projects that incorporated AI technology. Of those, research, development, test, and evaluation (RDT&E) funded 74, and 6 were acquisition procurement.

DAF AI-Based Prototypes and Demonstrations

The DAF has used proof-of-concept demonstrations and prototypes to motivate the value of AI and to increase its understanding of the processes, infrastructure needs, and specific challenges accompanying the integration of AI capability into its systems.

___________________

3 The committee is unaware of classified programs that incorporate AI technology, but that does not mean that such activities have not taken place.

Suggested Citation:"1 Introduction." National Academies of Sciences, Engineering, and Medicine. 2023. Test and Evaluation Challenges in Artificial Intelligence-Enabled Systems for the Department of the Air Force. Washington, DC: The National Academies Press. doi: 10.17226/27092.
×

TABLE 1-2 Current DoD AI Efforts

# of AI Projects
DoD Component R&D Funding Procurement Funding Total
DAFa 74 6 80
Army 209 23 232
Marine Corps 26 7 33
Navy 176 38 215
Other DoD Entitiesb,c 117 8 126

a DoD’s methodology combined AI projects from the Air Force and Space Force.

b Other DoD entities include combatant commands and other unspecified DoD components included in the JAIC’s methodology.

c DoD’s initial inventory does not include classified AI projects of those funded through operations and maintenance.

SOURCES: GAO analysis of Department of Defense (DoD) information: GAO-22-104765, Appendix II.

For example, in 2020, the Air Force conducted a demonstration at Beale Air Force Base that integrated machine learning into a test aircraft. In a training flight, the AI algorithm controlled the sensor and navigation systems on a U-2 Dragon Lady spy plane. A test pilot oversaw the AI operation but did not intervene, although flight control always remained in the hands of the pilot. According to an interview4 with Dr. William Roper, 13th Assistant Secretary of the Air Force for Acquisition, Technology, and Logistics, “Roper said the AI was trained against an opposing computer to look for oncoming missiles and missile launchers. The AI got the final vote for the initial test flight on where to direct the plane’s sensors.”

As another example, the Defense Advanced Research Projects Agency’s (DARPA’s) Air Combat Evolution (ACE) program has also been advancing the use of AI in DAF systems. As described by DARPA:5

The ACE program seeks to increase trust in combat autonomy by using human-machine collaborative dogfighting as its challenge problem. This program also serves as an entry point into complex human-machine collaboration. ACE will apply existing artificial intelligence technologies to the dogfight problem in experiments of increasing realism. In parallel, ACE will implement methods to measure, calibrate, increase, and predict human trust in combat autonomy performance. Finally, the program will scale the tactical application of autonomous dogfighting to more complex, heterogeneous, multi-aircraft, operational-level

___________________

4 A. Gregg, 2020, “In a First, Air Force Uses AI on Military Jet,” The Washington Post, December 16, https://www.washingtonpost.com/business/2020/12/16/air-force-artificial-intelligence.

5 R. Hefron, “Air Combat Evolution (ACE),” Defense Advanced Research Projects Agency, https://www.darpa.mil/program/air-combat-evolution.

Suggested Citation:"1 Introduction." National Academies of Sciences, Engineering, and Medicine. 2023. Test and Evaluation Challenges in Artificial Intelligence-Enabled Systems for the Department of the Air Force. Washington, DC: The National Academies Press. doi: 10.17226/27092.
×

simulated scenarios informed by live data, laying the groundwork for future live, campaign-level Mosaic Warfare experimentation.

An early ACE success was the AlphaDogfight simulation contest. In this contest, an AI agent based on deep reinforcement learning beat a seasoned Air Force F-16 pilot 5-0 in a set of simulated one-on-one dogfights between two F-16 aircraft.6

AI prototype capability has also been integrated into the DAF Common Mission Control Center (CMCC).7 The CMCC has incorporated a system called APIGEE (Automated Pipeline for Imagery Geospatial Enhancement and Enrichment), which does auto-mensuration of intelligence, surveillance, and reconnaissance (ISR) imagery to reference imagery to generate target-quality coordinates. APIGEE uses AI and deep learning to do image matching. The system currently performs electro-optical (EO)-to-EO mensuration, and additional multi-model capabilities are being tested for IR-to-EO and SAR-to-EO. CMCC has also incorporated a prototype Dynamic Electronic Order of Battle (EOB) capability that uses machine learning to develop patterns-of-life from electronic intelligence (ELINT) and identifies anomalies based on these normal behavior patterns. The project teams are migrating the EOB capability to the enterprise cloud as part of a CMCC “Common EOB” project.

Similarly, work has been ongoing with the Machine Assisted GEOint Exploitation (MAGE) program out of Air Combat Command (ACC) to incorporate AI into ISR processing, exploitation, and dissemination (PED) systems for geospatial intelligence (GeoINT) exploitation at defense geospatial service (DGS) sites. MAGE uses AI models to automate object detection workflows in various GeoINT products to support intelligence production. While actively under development, it still is not a fully deployed system and remains in development and evaluation.

The Air Force Collaborative Combat Aircraft (CCA) program pulls developed capabilities from other DAF programs such as Skyborg, AlphaDogfight, ACE, Variable In-Flight Simulator Aircraft (VISTA), and others. As the 96th Operations Group Commander briefed the committee, different aspects of each program, along with autonomy, data, and AI experimentation (ADAX),8 will be used to help formulate AI T&E policies and processes across the DAF. Early observations include the challenges of building new or heavily modifying existing air vehicles—such as the XQ-58 Valkyrie and the Viper Experimentation and Next-Gen Operations

___________________

6 E. Tegler, 2020, “AI Just Won a Series of Simulated Dogfights Against a Human F-16 Pilot, 5-0. What Does That Mean?” Forbes, August 20, https://www.forbes.com/sites/erictegler/2020/08/20/ai-just-won-a-series-of-simulated-dogfights-against-a-human-f-16-pilot-5-to-nothing-what-does-that-mean/?sh=7025a447235d; P. Tucker, 2020, “An AI Just Beat a Human F-16 Pilot in a Dogfight—Again,” Defense One, August 20, https://www.defenseone.com/technology/2020/08/ai-just-beat-human-f-16-pilot-dogfight-again/167872.

7 Private correspondence with Paul Metzger, MIT Lincoln Laboratory.

8 Autonomy, Data, and AI Experimentation.

Suggested Citation:"1 Introduction." National Academies of Sciences, Engineering, and Medicine. 2023. Test and Evaluation Challenges in Artificial Intelligence-Enabled Systems for the Department of the Air Force. Washington, DC: The National Academies Press. doi: 10.17226/27092.
×

Model (VENOM) project, respectively—while incorporating extensive autonomy and, eventually, AI-enabled autonomy. At this stage, few AI-specific standalone capabilities are associated with these projects. However, the goal of Project Venom is to demonstrate autonomous capabilities on manned (and at some point unmanned) F-16s. The maturation of the CCA program will require a commensurate comprehensive AI T&E strategy and implementation plan.

Based on the committee’s site visit to Eglin Air Force Base (AFB) and what various DAF representatives told the committee, it is evident from these prototype demonstrations that there is no well-established DoD-wide or DAF-wide set of standards for AI-based systems development or T&E. That is not to say that rigorous testing does not occur but that each project must develop its own T&E approaches and impose its own standards. However, what was also clear is that, due to the nature of the AI life cycle, early user involvement and continual T&E were essential elements of success.

A Case Study of Transition from Prototype to Initial Operating Capability

As the DAF seeks to transition prototypes to operational use, key aspects of AI-based systems acquisition have emerged (see Section 1.6 for a more in-depth case study). For example, Massachusetts Institute of Technology Lincoln Laboratory (MIT LL) transferred a Global Synthetic Weather Radar (GSWR) prototype to a software company called NextGen Federal Systems (NFS) for inclusion as part of the DAF weather forecasting system. The GSWR provides radar-like analyses and forecasts over regions not observed by actual weather radars by compiling lightning data, satellite imagery, and numerical weather models. The T&E process implemented to achieve the GSWR initial operating capacity (IOC) is instructive and underscores a few key points in acquiring AI-based capability. The process, in summary, was:9

  • MIT LL developed a prototype capability based on prior work for the Federal Aviation Administration (FAA) and performed extensive testing using its access to global weather data. Several modifications to adapt the FAA model were required to tune and retrain GSWR to work in various geographic regions around the globe.
  • Before officially transferring the software, MIT LL hosted a baseline implementation in the Amazon Web Services (AWS) government cloud. In addition, the contractor was given access to facilitate technology transition through a detailed assessment and refinement phase using an agile development process.

___________________

9 From private correspondence with Dr. Mark Veillette, MIT Lincoln Laboratory.

Suggested Citation:"1 Introduction." National Academies of Sciences, Engineering, and Medicine. 2023. Test and Evaluation Challenges in Artificial Intelligence-Enabled Systems for the Department of the Air Force. Washington, DC: The National Academies Press. doi: 10.17226/27092.
×
  • After this initial phase, NFS accepted the software, re-hosted the baseline as-is, and used the MIT LL prototype as a reference system to verify their implementation. In addition, MIT LL collaborated with NFS on the USAF cloud portal to update the software and make it compliant with DoD cyber security requirements. Much of this work entailed fixing code quality issues identified by scanners (e.g., SonarQube) and developing unit and integration tests to support future development operations (DevOps). During this time, NFS also integrated the baseline into Kubernetes10 for improved cloud deployment.

The DAF is developing an in-house platform for tracking curated datasets, machine learning (ML) model training, and experimentation. Because GSWR has significant ML components, the datasets and training processes were also integrated into this platform for NFS to replicate MIT LL results.

This case study underscores a few key points that manifest in virtually all AI-based projects:

  • Subject-matter experts (SMEs) in machine learning and weather forecasting were needed throughout all phases, from initial concept to IOC, and have remained involved beyond the IOC phase to facilitate a rapid and flexible DevOps process that integrates security requirements (DevSecOps).
  • Extensive data was needed in the early research and development and deployment phases, and retraining for new geographic regions with new datasets was required. Curation, protection, and integration of data into the overall DevSecOps process were recognized as a necessary part of the engineering process. Retraining using operational data and the infrastructure to support this retraining were also key elements of success.

DAF AI Research and Development

The DAF, principally through the Air Force Research Laboratory (AFRL), is conducting or funding several research and development projects to advance AI implementations for the DAF. The projects span the following AI-related areas:11

  • Basic AI research in the mathematics, information sciences, and life sciences
  • AI applied to materials for structures, propulsion, and subsystems

___________________

10 Kubernetes is an open-source container orchestration platform that automates many of the manual processes involved in deploying, managing, and scaling containerized applications.

11 Department of Defense, 2022, Fiscal Year (FY) 2023 Budget Estimates, Office of the Secretary of Defense, Vol. 3 of 5 in Defense-Wide Justification Book, Washington, DC, https://comptroller.defense.gov/Portals/45/Documents/defbudget/fy2023/budget_justification/pdfs/03_RDT_and_E/OSD_PB2023.pdf. Some smaller RDT&E projects have been excluded.

Suggested Citation:"1 Introduction." National Academies of Sciences, Engineering, and Medicine. 2023. Test and Evaluation Challenges in Artificial Intelligence-Enabled Systems for the Department of the Air Force. Washington, DC: The National Academies Press. doi: 10.17226/27092.
×
  • AI, automation, and autonomy for sensory evaluation and decision science
  • AI for EO sensors and countermeasures technology
  • AI applied to sensor fusion
  • AI applied to C4I12 dominance, battlespace development, and demonstration
  • AI-enhanced life-cycle management
  • Skyborg integrated technology demonstration

One technology demonstration of note is the AFRL-Air Force Life Cycle Management Center (AFLCMC) Skyborg project, one of the DAF’s four Vanguard programs and a component of the DAF’s overarching CCA project. Skyborg is an autonomous aircraft teaming architecture designed to increase the number of mission sorties while lowering costs. The program is investigating how AI-operated drones can team with human-piloted aircraft. Skyborg has established an open approach to autonomy architecture, building a scalable system designed to be portable across aircraft platforms and modular in its design to accommodate multiple software applications. Skyborg is intended to become a program of record in 2023 or 2024, depending on budget constraints. In 2022, the program executive officer (PEO) for AFLCMC’s fighters and advanced aircraft directorate called Skyborg “wildly successful in terms of what we got out of it, what we continue to get out of it, and how we use that to present decision space to our leaders on how we set up programs of record.”13

The AFRL-funded DAF-MIT AI Accelerator (AIA 1.0) is an example of a research and development project that has elements of core AI, enabling AI, and AI-enabled capability. The project’s website (https://aia.mit.edu/about) provides project details and a succinct introduction to AIA 1.0. The latter is excerpted below:

In February 2019, the President of the United States signed Executive Order 13859 announcing the American AI Initiative—the nation’s strategy on AI. He wrote, “Continued American leadership in Artificial Intelligence is paramount to maintaining the economic and national security of the United States.”

The DAF subsequently signed a cooperative agreement with the Massachusetts Institute of Technology (MIT) to create a joint artificial intelligence Accelerator hosted at MIT. The effort, known as the DAF-MIT AI Accelerator (AIA), leverages the combined expertise and resources of MIT and the DAF. The AIA conducts fundamental research to enable rapid prototyping, scaling, and the ethical application of AI algorithms and systems to advance the DAF and society. A multidisciplinary team of embedded officers and enlisted airmen join MIT faculty, researchers, and

___________________

12 Command, control, communications, computers, and intelligence.

13 G. Hadley, 2022, “‘Wildly Successful’ Skyborg Will Become Program of Record But Won’t Stop Developing S&T,” Air and Space Forces Magazine, August 16, https://www.airandspaceforces.com/wildly-successful-skyborg-program-of-record-developing-st.

Suggested Citation:"1 Introduction." National Academies of Sciences, Engineering, and Medicine. 2023. Test and Evaluation Challenges in Artificial Intelligence-Enabled Systems for the Department of the Air Force. Washington, DC: The National Academies Press. doi: 10.17226/27092.
×

students to tackle some of the most difficult challenges facing our nation and the air force, ranging from technical to humanitarian.

In January 2020, the AI accelerator launched ten interdisciplinary projects involving researchers from the MIT campus, MIT LL, and the DAF, as seen in Table 1-3. The 3-year projects, which encompass 15 research workstreams, advance AI research in various areas, including weather modeling and visualization, optimization of training schedules, and autonomy for augmenting and amplifying human decision-making.

While a major goal of the AIA is to develop core AI relevant to societal benefit and air force needs, the program is also developing tools, techniques, processes, and infrastructure that pioneer new DAF approaches to AI technology acquisition. Examples include the following:

  • Computational support for AI. The “Fast AI” and “ML-Enhanced Data Collection, Integration, and Outlier Detection” projects focus on providing

TABLE 1-3 DAF-MIT AI Accelerator Projects

# AIA Project Name Project Type
1 Guardian Autonomy for Safe Decision Making AI Core
2 Fast AI AI Enabling
3 ML-Enhanced Data Collection, Integration, and Outlier Detection AI Enabling/AI Core
4 Transferring Multi-Robot Learning Through Virtual and Augmented Reality for Rapid Disaster Response AI Core
5 Conversational Interaction for Unstructured Information Access AI Core
6 AI for Personalized Foreign Language Education AI Core
7 Multimodal Vision for Synthetic Aperture Radar AI Core
8 AI-Assisted Optimization of Training Schedules AI Core
9 The Earth Intelligence Engine AI Core
10 Continual and Few-Shot Learning: Transferring Knowledge to New Low Resource Domains and Tasks AI Core/AI enabling
11 Explainable Machine Learning for Decision Support AI Core/AI Enabling
12 AI Education Research: Know-Apply-Lead AI Enabling
13 RAIDEN (Robust AI Development Environment) AI Core/AI Enabling
14 Objective Performance Prediction and Optimization Using Physiological and Cognitive Metrics AI Core
15 Robust Neural Differential Models for Navigation and Beyond AI Core
16 AI-Enhanced Spectral Awareness and Interference Rejection AI Core
17 Application of Coevolutionary Algorithms for DoD Complex Enterprises AI Core
18 Space Domain Awareness AI Core
19 Better Networks via AI-Enabled Hierarchical Connection Science AI Core
Suggested Citation:"1 Introduction." National Academies of Sciences, Engineering, and Medicine. 2023. Test and Evaluation Challenges in Artificial Intelligence-Enabled Systems for the Department of the Air Force. Washington, DC: The National Academies Press. doi: 10.17226/27092.
×
  • efficient computational and data management capabilities to enable scalable AI development. The latter project also uses machine learning in its support algorithms (“AI for AI”).
  • AI tools. The “RAIDEN,”“Continual and Few-Shot Learning,” and “Explainable Machine Learning for Decision Support” projects focus on foundational AI advances to support T&E and responsible AI development.
  • Small, agile teams. All projects by design include MIT researchers, MIT LL application and AI specialists, and DAF airmen and guardians working together as a team from project conception to transition. Transition partners are identified at the outset of projects and interact throughout the project life cycle. This diverse team composition encourages technology transition to the fir Force and air force user feedback to the researchers.
  • Challenge problems. The open release of labeled datasets such as ImageNet has spurred the advancement of commercial and academic machine learning technology worldwide. Each AIA project defines a set of challenge problems and releases curated and labeled datasets to engage the broader AI research community. The datasets are unclassified but representative of key AI technology challenges in each project’s research domain. Also, to facilitate challenge participation, the AIA developed a DoD-tailored data-sharing approach based on a University of California agreement that has been used for decades.14
  • Educational outreach. Recognizing the importance of educating the DAF workforce, the “AI Education Research: Know-Apply-Lead” project was established to explore how to shape curricula and create course-ware to customize AI education for various learners with different needs and responsibilities.

Many AIA projects will seek to transition their technologies to air force stakeholders in the next few years. These transitions should provide opportunities to explore the efficacy of small-team AI development processes, including DevSecOps processes and continual T&E.

Summary

The DAF is only beginning to pursue AI technology for its systems and operations. To the committee’s knowledge, no major DAF acquisition program (MDAP or MAIS) has incorporated modern AI technology beyond prototype capabilities and advanced concept demonstrations. In the absence of DAF and DOD AI-specific

___________________

14 See University of California, 2022, “University of California Research Data Policy,” VP-Research and Innovation, https://policy.ucop.edu/doc/2500700/ResearchData; Department of the Air Force-Massachusetts Institute of Technology Artificial Intelligence Accelerator, 2023, “Challenges Supplemental Resources,” http://aia.mit.edu/challenges-supplemental.

Suggested Citation:"1 Introduction." National Academies of Sciences, Engineering, and Medicine. 2023. Test and Evaluation Challenges in Artificial Intelligence-Enabled Systems for the Department of the Air Force. Washington, DC: The National Academies Press. doi: 10.17226/27092.
×

standards, the acquisition and T&E processes adopted by these prototyping projects have been ad hoc, although they emulate sound commercial practices. Several AI projects are in the RDT&E pipeline, which motivates the need to address AI T&E for DAF as these technologies mature and become incorporated into DAF systems. Similarly, the 2022 establishment of the Autonomy, Data, and AI Experimentation proving ground at Eglin AFB as a joint venture between DAF CDAO and AFWERX is an encouraging initial step. Additionally, information is not always shared between the different pockets of AI work throughout the DAF.

This report aims to provide timely recommendations to help the Air Force establish effective T&E infrastructure and processes in anticipation of increased use of AI, especially applying AI technology to safety critical systems.

1.6 ALGORITHMIC WARFARE CROSS-FUNCTIONAL TEAM (PROJECT MAVEN) CASE STUDY

In April 2017 then-Deputy Secretary of Defense Robert Work established the Algorithmic Warfare Cross-Functional Team (AWCFT), or Project Maven, which reported to the deputy secretary through the under secretary of defense for intelligence (USDI). The AWCFT was the Department of Defense’s first program to operationalize AI/ML at speed and scale. The AWCFT’s primary objective was to accelerate the Department of Defense’s integration of big data and machine learning and to “turn the enormous volume of data available to DoD into actionable intelligence and insights at speed.”15

The AWCFT’s first specified task was to field AI capabilities to augment, accelerate, and automate the processing, exploitation, and dissemination (PED) of full-motion video (FMV) from tactical and medium-altitude unmanned aerial systems (UAS). During the first year, Project Maven adopted and tailored commercially-developed computer vision (CV) algorithms for object detection, classification, and tracking. Its work subsequently expanded to include natural language processing (NLP) for exploitation of hard copy and digital materials collected during combat operations in the Middle East and East Africa, as well as machine translation, facial recognition, and SAR. Maven was also tasked to consolidate existing AI algorithm-based technology projects across the defense intelligence enterprise (DIE), including initiatives that developed, employed, or fielded AI, automation, machine learning (ML), deep learning (DL), and computer vision algorithms.

The initial cadre of Project Maven personnel lacked AI T&E experience, forcing them to rely extensively on T&E support from outside organizations. The primary participants were the Johns Hopkins University Applied Physics Laboratory

___________________

15 Department of Defense, 2017, “Establishment of an Algorithmic Warfare Cross-Functional Team (Project Maven),” Deputy Secretary of Defense, https://www.govexec.com/media/gbc/docs/pdfs_edit/establishment_of_the_awcft_project_maven.pdf.

Suggested Citation:"1 Introduction." National Academies of Sciences, Engineering, and Medicine. 2023. Test and Evaluation Challenges in Artificial Intelligence-Enabled Systems for the Department of the Air Force. Washington, DC: The National Academies Press. doi: 10.17226/27092.
×

(JHU APL), the Army and Air Force Research Laboratories, and several commercial companies with AI model fielding experience.

In the first few months of operations, the Project Maven team learned what has been noted elsewhere in this study, to wit, the substantial differences between AI T&E and T&E for traditional hardware systems. There was no DoD-wide AI T&E “playbook” for Maven to rely on. And because no other extant DoD-wide AI projects were dedicated to fielding AI-enabled solutions at scale, the OSD director of operational test and evaluation had not yet developed a standardized DoD AI T&E template or established AI T&E best practices.16 Instead, individual users and organizations, primarily within the service research laboratories, had continued to develop boutique T&E processes, procedures, practices, and technical solutions tailored to their unique AI projects, the vast majority of which were research and development initiatives at relatively low technology readiness levels (TRLs).

The Maven team developed model performance benchmarks and other T&E criteria for each algorithm purchased from a commercial vendor (and subsequently trained against DoD data to become a DoD-licensed AI model). For computer vision algorithms, these included precision, recall, f-scores, intersection over union—more of a parameter than a metric in and of itself—and mean average precision. While each commercial vendor provided its own internal testing results, Maven insisted on reinforcing commercial testing results with additional, DoD-led tests and evaluation of each algorithm and trained model, using withheld test data to which the vendors were not exposed.

Since there were few examples of AI T&E within DoD apart from small-scale research laboratory projects, Maven adopted and adapted AI T&E best practices from the private sector and academia. These practices include setting aside sufficient representative, quality data for training, test, and validation or assessment; building T&E harnesses;17 evaluating fielded models as part of ongoing operational assessments; evaluating model boundary conditions and AI failure modes; and developing T&E processes for each subsequent update to fielded models through normal CI/CD processes. The extent of T&E required for each subsequent model version depended on the breadth and depth of the changes included in each update. In most cases, later versions required a shorter T&E process than required during the first several updates to fielded models. In all cases, the Maven T&E team gained enough experience to accelerate T&E timelines. Maven also coordinated with commercial AI companies to establish contractual requirements for AI algorithm

___________________

16 As of December 2022, OSD DOT&E had not developed AI T&E templates or promulgated AI T&E best practices.

17 A T&E harness is “a software that tests model accuracy and other metrics.” See JAIC, 2020, “JAIC Spotlight: The JAIC’s Test Evaluation and Assessment Team Shapes Future AI Initiatives,” CDAO blog, May 27, https://www.ai.mil/blog_05_27_20-jaic_spotlight_test_evaluation_and_assessment_team.html.

Suggested Citation:"1 Introduction." National Academies of Sciences, Engineering, and Medicine. 2023. Test and Evaluation Challenges in Artificial Intelligence-Enabled Systems for the Department of the Air Force. Washington, DC: The National Academies Press. doi: 10.17226/27092.
×

performance and detail intellectual property (IP) protections (although neither of these was entirely resolved under Maven; given the unique circumstances of each AI project, performance requirements and IP protections must be addressed separately for every AI development project—as discussed in more detail in the requirements section that follows).

In addition to developing an AI T&E “playbook,” the Maven team worked closely with operational end-users, none of whom had any previous experience using AI-enabled systems and were unfamiliar with establishing requirements for or interpreting the metrics associated with AI T&E. Maven personnel “translated” T&E metrics into terms most relevant to operational end-users. Because formal requirements had not been established for AI model performance, once the Maven team had completed data quality assurance, T&E on each model, integration testing in the Maven Integration Lab, and live-fly testing; user acceptance of each trained model, and follow-on updates to those fielded models, was based primarily on an agreement between the Maven team and operational users that models had demonstrated adequate performance under operational conditions. As acknowledged elsewhere in this study, this process underscored the importance of defining future T&E requirements for all AI capabilities and AI-enabled platforms, sensors, and tools in ways that reflect consensus between developers and end-users at every stage of the AI life cycle.

After the first year of operations, Dr. Yevgeniya (Jane) Pinelis, who worked for the Institute for Defense Analyses (IDA) as a technical advisor to OSD DOT&E, moved to JHU APL to serve as their on-site representative to the AWCFT. As the Project Maven T&E team lead, Dr. Pinelis led the developmental and operational testing of AI algorithms, including computer vision, machine translation, facial recognition, natural language processing, and human-machine teaming. In addition, Dr. Pinelis relied on existing policies and standards from outside the department, particularly those established by the International Organization for Standardization (ISO), the Institute of Electrical and Electronics Engineers (IEEE), and the National Institute of Standards and Technology (NIST), to develop DoD-specific AI T&E policies, processes, procedures, and best practices in this role.18

Based on T&E lessons learned from Project Maven, the inaugural Director of the DoD Joint AI Center (JAIC), Lieutenant General Jack Shanahan,19 established a test and evaluation directorate within the JAIC as part of the initial organizational structure. Dr. Pinelis served as the JAIC’s inaugural chief of test and evaluation, and subsequently served as the chief of AI assurance in the OSD CDAO until her departure in early 2023. Dr. Pinelis extended her previous Maven T&E work to

___________________

18 Once the deputy secretary of defense issued the “DoD AI Ethical Principles” in February 2020, the Maven AI T&E cadre was tasked with testing AI algorithms and models for operational effectiveness, robustness, resiliency, and alignment with those principles.

19 Lieutenant General Shanahan (USAF, Ret.) served as a committee member for this study.

Suggested Citation:"1 Introduction." National Academies of Sciences, Engineering, and Medicine. 2023. Test and Evaluation Challenges in Artificial Intelligence-Enabled Systems for the Department of the Air Force. Washington, DC: The National Academies Press. doi: 10.17226/27092.
×

develop an AI T&E template for the JAIC, which became the accepted standard for defining T&E requirements, evaluating algorithm performance during model training and testing, and performing T&E on updates to fielded models.

In her role at the JAIC, Dr. Pinelis formed an AI T&E Community of Interest (CoI) across DoD and with academia and other government agencies, including the National AI Initiative Office (NAIIO), NIST, the Office of the Director of National Intelligence (ODNI), OSD DOT&E, the Test Resource Management Center (TRMC), the OSD Under Secretary for Research and Engineering (OUSD[R&E]), the military services, DARPA, Federally Funded Research and Development Centers (FRDC) and University-Affiliated Research Centers (UARC), and representatives from academia and industry. The CDAO has since published AI T&E playbooks and best practice guides, which are available to all government agencies and organizations, and launched a first-of-its-kind AI T&E bulk purchasing agreement that allows government components to access leading AI T&E commercial vendors.

A Project Maven vignette from 2019 to 2020 underscored the importance of rigorous and disciplined AI T&E and the need for government agencies to rely on in-house or disinterested third-party T&E to validate test results provided by commercial vendors. When evaluating the performance of a later version of a fielded AI computer vision model, T&E results indicated a decrease in performance compared to the previous model version. This was an unexpected result since all other earlier updates to the model demonstrated steadily-improving performance across all T&E metrics. Unfortunately, the results did not improve, despite repeated testing with additional test data. As a result, the team faced a decision of whether to field an update that was needed immediately by the operational end-users, under the assumption that there were unknown flaws in the T&E process rather than the model itself or delaying fielding until the unexpected results could be explained. They elected to delay fielding the updated version of the model.

After a detailed analysis of contributing factors, the team discovered that the commercial vendor responsible for the CV algorithm had lost several key data scientists over several months. Their replacements were not as familiar with the fielded model and provided an updated version of the algorithm that had been insufficiently tested. The performance of the updated model was not operationally acceptable—exactly as Maven’s T&E results had indicated. The algorithm and model were improved, retrained, retested, and fielded. The Maven case study highlights how many AI T&E issues are technically feasible but organizationally challenging.

Suggested Citation:"1 Introduction." National Academies of Sciences, Engineering, and Medicine. 2023. Test and Evaluation Challenges in Artificial Intelligence-Enabled Systems for the Department of the Air Force. Washington, DC: The National Academies Press. doi: 10.17226/27092.
×
Page 17
Suggested Citation:"1 Introduction." National Academies of Sciences, Engineering, and Medicine. 2023. Test and Evaluation Challenges in Artificial Intelligence-Enabled Systems for the Department of the Air Force. Washington, DC: The National Academies Press. doi: 10.17226/27092.
×
Page 18
Suggested Citation:"1 Introduction." National Academies of Sciences, Engineering, and Medicine. 2023. Test and Evaluation Challenges in Artificial Intelligence-Enabled Systems for the Department of the Air Force. Washington, DC: The National Academies Press. doi: 10.17226/27092.
×
Page 19
Suggested Citation:"1 Introduction." National Academies of Sciences, Engineering, and Medicine. 2023. Test and Evaluation Challenges in Artificial Intelligence-Enabled Systems for the Department of the Air Force. Washington, DC: The National Academies Press. doi: 10.17226/27092.
×
Page 20
Suggested Citation:"1 Introduction." National Academies of Sciences, Engineering, and Medicine. 2023. Test and Evaluation Challenges in Artificial Intelligence-Enabled Systems for the Department of the Air Force. Washington, DC: The National Academies Press. doi: 10.17226/27092.
×
Page 21
Suggested Citation:"1 Introduction." National Academies of Sciences, Engineering, and Medicine. 2023. Test and Evaluation Challenges in Artificial Intelligence-Enabled Systems for the Department of the Air Force. Washington, DC: The National Academies Press. doi: 10.17226/27092.
×
Page 22
Suggested Citation:"1 Introduction." National Academies of Sciences, Engineering, and Medicine. 2023. Test and Evaluation Challenges in Artificial Intelligence-Enabled Systems for the Department of the Air Force. Washington, DC: The National Academies Press. doi: 10.17226/27092.
×
Page 23
Suggested Citation:"1 Introduction." National Academies of Sciences, Engineering, and Medicine. 2023. Test and Evaluation Challenges in Artificial Intelligence-Enabled Systems for the Department of the Air Force. Washington, DC: The National Academies Press. doi: 10.17226/27092.
×
Page 24
Suggested Citation:"1 Introduction." National Academies of Sciences, Engineering, and Medicine. 2023. Test and Evaluation Challenges in Artificial Intelligence-Enabled Systems for the Department of the Air Force. Washington, DC: The National Academies Press. doi: 10.17226/27092.
×
Page 25
Suggested Citation:"1 Introduction." National Academies of Sciences, Engineering, and Medicine. 2023. Test and Evaluation Challenges in Artificial Intelligence-Enabled Systems for the Department of the Air Force. Washington, DC: The National Academies Press. doi: 10.17226/27092.
×
Page 26
Suggested Citation:"1 Introduction." National Academies of Sciences, Engineering, and Medicine. 2023. Test and Evaluation Challenges in Artificial Intelligence-Enabled Systems for the Department of the Air Force. Washington, DC: The National Academies Press. doi: 10.17226/27092.
×
Page 27
Suggested Citation:"1 Introduction." National Academies of Sciences, Engineering, and Medicine. 2023. Test and Evaluation Challenges in Artificial Intelligence-Enabled Systems for the Department of the Air Force. Washington, DC: The National Academies Press. doi: 10.17226/27092.
×
Page 28
Suggested Citation:"1 Introduction." National Academies of Sciences, Engineering, and Medicine. 2023. Test and Evaluation Challenges in Artificial Intelligence-Enabled Systems for the Department of the Air Force. Washington, DC: The National Academies Press. doi: 10.17226/27092.
×
Page 29
Suggested Citation:"1 Introduction." National Academies of Sciences, Engineering, and Medicine. 2023. Test and Evaluation Challenges in Artificial Intelligence-Enabled Systems for the Department of the Air Force. Washington, DC: The National Academies Press. doi: 10.17226/27092.
×
Page 30
Suggested Citation:"1 Introduction." National Academies of Sciences, Engineering, and Medicine. 2023. Test and Evaluation Challenges in Artificial Intelligence-Enabled Systems for the Department of the Air Force. Washington, DC: The National Academies Press. doi: 10.17226/27092.
×
Page 31
Suggested Citation:"1 Introduction." National Academies of Sciences, Engineering, and Medicine. 2023. Test and Evaluation Challenges in Artificial Intelligence-Enabled Systems for the Department of the Air Force. Washington, DC: The National Academies Press. doi: 10.17226/27092.
×
Page 32
Suggested Citation:"1 Introduction." National Academies of Sciences, Engineering, and Medicine. 2023. Test and Evaluation Challenges in Artificial Intelligence-Enabled Systems for the Department of the Air Force. Washington, DC: The National Academies Press. doi: 10.17226/27092.
×
Page 33
Suggested Citation:"1 Introduction." National Academies of Sciences, Engineering, and Medicine. 2023. Test and Evaluation Challenges in Artificial Intelligence-Enabled Systems for the Department of the Air Force. Washington, DC: The National Academies Press. doi: 10.17226/27092.
×
Page 34
Next: 2 Definitions and Perspectives »
Test and Evaluation Challenges in Artificial Intelligence-Enabled Systems for the Department of the Air Force Get This Book
×
 Test and Evaluation Challenges in Artificial Intelligence-Enabled Systems for the Department of the Air Force
Buy Paperback | $42.00 Buy Ebook | $33.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

The Department of the Air Force (DAF) is in the early stages of incorporating modern artificial intelligence (AI) technologies into its systems and operations. The integration of AI-enabled capabilities across the DAF will accelerate over the next few years.

At the request of DAF Air and Space Forces, this report examines the Air Force Test Center technical capabilities and capacity to conduct rigorous and objective tests, evaluations, and assessments of AI-enabled systems under operational conditions and against realistic threats. This report explores both the opportunities and challenges inherent in integrating AI at speed and at scale across the DAF.

READ FREE ONLINE

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!