Summary
The military is investing in artificial intelligence (AI) as a tool that can potentially play a critical role in supporting command and control for future multi-domain operations (MDO) by boosting the processing rate of a wide variety of data inputs, automating mission planning, and creating faster predictive targeting and systems maintenance. Achieving this goal requires that the AI system be both reliable and robust across a wide variety of potential future missions, and that it works well as a teammate with humans. This report examines the factors that are relevant to the design and implementation of AI systems with respect to human operations, and it recommends needed research for achieving successful performance across the human-AI team.
Although AI has many potential benefits, it has also been shown to suffer from a number of challenges for successful performance in complex, real-world environments such as military operations, including brittleness, perceptual limitations, hidden biases, and lack of a model of causation important for understanding and predicting future events. These limitations mean that, for the foreseeable future, AI will remain inadequate for operating on its own in many complex and novel situations for the foreseeable future, and that humans will need to carefully manage AI systems to achieve their desired utility.
Research over the past 30 years has demonstrated, however, that people are significantly challenged in performing as successful monitors of complex automation, including AI systems. People can suffer from poor understanding of what the systems are doing, high workload when trying to interact with AI systems, poor situation awareness (SA) and performance deficits when intervention is needed, biases in decision making based on system inputs, and degradation of manual skills. These numerous challenges will continue to create problems in terms of human performance, even with more capable AI-based automation.
Therefore, effective human-AI teams capable of taking advantage of the unique abilities of both humans and AI, while overcoming the known challenges and limitations of each team member, need to be developed. An effective human-AI team ultimately augments human capabilities and raises performance beyond that of either entity. To this end, the committee has developed an interrelated set of research objectives that are presented to focus around the development of effective human-AI teams, based on improvements in models and metrics for human-AI teams (Chapter 2), team processes (Chapter 3), SA (Chapter 4), AI transparency and explainability (Chapter 5), human-AI interaction approaches (Chapter 6), trust (Chapter 7), reduction of human and AI bias (Chapter 8), and training (Chapter 9), supported by a foundation of human-systems integration (HSI) processes (Chapter 10). The report ends with a concluding chapter summarizing human-AI teaming research objectives aligned along near-, mid-, and far-term objectives. Each chapter has a summary of research objectives that have been developed by the committee for the sponsor.
HUMAN-AI TEAM MODELS
The committee finds that there is significant value in considering the human and AI as a team. This team construct fosters a recognition of the need to consider the interrelated roles of each team member, and it places emphasis on the value of team interactions, including communication and coordination, for boosting their combined performance. In such team arrangements, the committee believes that, in general, the human should have authority over the AI system, for both ethical and practical reasons. Improved computational models of human-AI teams are called for that consider the interrelated, dynamically evolving, distributed, and adaptive collaborative tasks and conditions that are also needed for networked command and control systems for MDO, and that are predictive within the design trade space. Improved metrics for human-AI teaming are needed that consider the team’s ability to manage interdependencies and dynamic role assignments, that reduce uncertainty, and that improve the ability of the AI system to deliver capabilities that are in line with expectations of warfighters.
While it is assumed that human-AI teams will be more effective than either humans or AI systems operating alone, in the committee’s judgment this will not be the case unless humans can (1) understand and predict the behaviors of the AI system (see Chapters 4 and 5); (2) develop appropriate trust relationships with an AI system (see Chapter 7); (3) make accurate decisions based on input from the AI system (see Chapter 8); and (4) exert control over the system in a timely and appropriate manner (see Chapter 6).
HUMAN-AI TEAM PROCESSES
Supporting humans and AI systems as teammates relies on a carefully designed system with the capability for both taskwork and teamwork. Along this line, research is needed to improve team effectiveness in long-term, distributed, and agile human-AI teams through improved team assembly, goal alignment, communication, coordination, social intelligence, and the development of a new human-AI language (see Chapter 3). This research can leverage the large body of existing work on human-human teaming, but the committee recognizes that new research is needed to better understand and support effective team processes between humans and AI systems. In addition, the committee believes that research should examine the potential for an AI system to boost team performance by serving as a team coordinator, orchestrator, or human-resource manager.
SITUATION AWARENESS
It is widely recognized that human situation awareness (SA) is critical for effective MDO performance, including for the oversight of AI systems. Methodologies for supporting individual and team SA in command and control operations need to be extended to MDO, and methods for using AI to support information integration, prioritization, and routing across the joint battle space are needed, as well as for improving resilience to adversarial attacks on SA. Methods for improving human SA of AI systems need to be developed that consider diverse types of applications, timescales of operations, and the changing capabilities associated with machine learning (ML)-based AI systems (see Chapter 4). In addition, research directed at creating shared SA within the human-AI team deserves attention. The degree to which AI systems need to have both self-awareness and awareness of their human teammates needs to be explored, to determine the benefit for overall team performance. Finally, future AI systems will need to possess integrated situation models to appropriately understand the current situation and to project future situations for decision making. AI models of the dynamic task environment will be needed that can work with humans to align or deconflict goals and to synchronize situation models, decisions, function allocations, task prioritizations, and plans to achieve coordinated and approved actions.
AI TRANSPARENCY AND EXPLAINABILITY
Improved AI system transparency and explainability (see Chapter 5) are key to achieving improved human SA, as well as trust. Real-time transparency is critical for supporting understanding and predictability of AI systems and has been found to significantly compensate for out-of-the-loop performance deficits. Research is needed to
better define the information requirements and methods for achieving transparency of ML-based AI systems, as well as to define when such information should be provided to meet SA needs without overloading the human. Improved visualization of explanations from ML-based AI systems needs further exploration, as well as research on the value of machine personae. Further, the relationship between AI explainability and trust would benefit from further research, to inform improved, multi-factor models of how explanations can foster trust and trust-influenced decisions. Effective mechanisms to adapt explanations to receivers’ needs, prior knowledge and assumptions, and cognitive and emotional states need to be developed. The committee also suggests that research be directed at determining whether explanations of human reasoning could likewise improve AI system and human-AI team performance.
HUMAN-AI TEAM INTERACTION
Interaction mechanisms and strategies within the human-AI team are critical to team effectiveness, including the ability to support flexible assignments of levels of automation (LOAs) across functions over time. Research is needed to determine improved methods for supporting collaboration between humans and AI systems in shared functions, to support human operators working with AI systems at multiple LOAs, and to determine methods for maintaining or regaining SA when working with AI systems at high LOAs (i.e., on-the-loop control). Research is also needed to determine new requirements to support dynamic functional assignments across human-AI teams, and to determine the best methods for supporting dynamic transitions in LOAs over time, including when such transitions should occur, who should activate them, and how they should occur, to maintain optimal human-AI team performance. The committee suggests that research also be conducted on Playbook control methodologies (defined in Chapter 6), extending it to MDO tasks and human-AI teaming applications. Finally, research directed at a better understanding and prediction of emergent human-AI interactions, and on a better understanding of the effects of interaction design decisions on skill retention, training requirements, job satisfaction, and overall human-AI team resilience would be beneficial.
TRUST
Trust in AI is recognized as a foundational factor associated with use of AI systems. It would benefit future research to better document the decision context and goals involved in the teaming environment (detailed in Chapter 7), to advance understanding of how broader sociotechnical factors affect trust in human-AI teams. Interaction structures that extend beyond supervisory control arrangements would also benefit from further study, particularly to understand the effect of AI directability on the trust relationship. The team lens is useful here for identifying novel interaction structures with AI teammates. Improved measures of trust are needed that draw on the importance of cooperation, and that separate the concept of distrust from trust. Finally, dynamic models of trust are needed that capture how trust evolves and affects performance outcomes in various human-AI team contexts. This research would do well to examine trust-enabled outcomes that emerge from dyadic team interactions and extend that work into how trust evolves in larger teams and multi-echelon networks.
BIAS
The potential for bias in AI systems, often hidden, can be introduced through the development of its algorithms as well as through systemic biases in training sets, among other factors (see Chapter 8). Further, humans can suffer from several well-known decision biases. Of particular import, human decision making can be directly affected by the accuracy of the AI system, creating a human-AI team bias; therefore, humans cannot be viewed as independent adjudicators of AI recommendations. Research is needed to better understand the interdependencies between human and AI decision-making biases, how these evolve over time, and methods for detecting and preventing bias with ML-based AI. Research is also needed to detect and prevent potential adversarial attacks that may attempt to take advantage of these biases.
TRAINING
Training of the human-AI team will be needed to develop the appropriate team constructs and skills necessary for effective performance (see Chapter 9). Directed research is needed to determine what, when, why, and how to best train human-AI teams, taking into consideration various team compositions and sizes. Existing training methodologies can be explored to see whether they can be adapted to human-AI teaming. In addition, training may be needed to better calibrate human expectations of AI teammates and to foster appropriate levels of trust. Specific platforms will be necessary to develop and test human-AI teamwork procedures.
HSI PROCESSES AND MEASURES
Finally, achieving the successful development of an AI system that can function as a good teammate will require advances in HSI processes and measures (see Chapter 10). Good HSI practice will be key to the design, development, and testing of new AI systems, particularity with respect to system development based on agile or DevOps practices. New HSI design and testing methods for effective human-AI teams will also be needed, including an improved ability to determine requirements for human-AI teams, particularly those that involve ML (see Chapter 10). Improved approaches for multi-disciplinary AI development teams are needed that include human factors engineers, sociotechnical researchers, systems engineers, and computer scientists. New teams, methods, and tools centered around AI lifecycle testing and auditability, as well as AI cyber vulnerability, will also be needed. Methods for testing and verification of evolving AI systems need to be developed to detect AI system blind spots and edge cases and to consider brittleness. New human-AI testbeds to support research and development activities by these new teams will also be important. Finally, improved metrics for human-AI teaming may be needed, specifically regarding matters of trust, mental models, and explanation quality.
CONCLUSION
In total, 57 research objectives are presented to address the many challenges for effective human-AI teaming. These research objectives are divided into near term (1–5 years), mid-term (6–10 years), and far term (10–15 years) priorities. This integrated set of research objectives will achieve significant advances in human-AI teaming competence. These objectives are fundamental prerequisites to the safe introduction of AI into critical operations such as MDO, and they create a framework for better understanding and supporting the effective use of AI systems.
