National Academies Press: OpenBook

2019-2020 Assessment of the Army Research Laboratory (2021)

Chapter: 4 Human Sciences

« Previous: 3 Computational and Atmospheric Sciences
Suggested Citation:"4 Human Sciences." National Academies of Sciences, Engineering, and Medicine. 2021. 2019-2020 Assessment of the Army Research Laboratory. Washington, DC: The National Academies Press. doi: 10.17226/26325.
×

4

Human Sciences

The Panel on Human Factors Sciences at the Army Research Laboratory (ARL) conducted its review of selected research and development (R&D) projects of ARL’s human sciences (HS) core competency area at the Aberdeen Proving Ground, Maryland, on May 29-31, 2019. The HS project areas reviewed were human-autonomy team interactions and humans understanding autonomy; autonomy understanding humans and estimating human-autonomy team outcomes; human interest detection; cyber science and kinesiology; neuroscience, training effectiveness, and strengthening teamwork for robust operations with novel groups (STRONG).

ARL’s HS core competency is focused on identifying, creating, and transitioning scientific discoveries and technological innovations underlying three research areas: cognitive dominance,1 readiness for technological complexity, and teaming overmatch.2 These areas are critical to the U.S. Army’s future technological superiority. This core competency area concentrates on high-risk and high-payoff transformational basic research with potential for having revolutionary impacts on the Army’s warfighting capabilities. The ultimate goal is to contribute to the creation of disruptive and game-changing soldier-centric technologies for the Army, while also anticipating technological surprises from potential adversaries.

___________________

1 As described by ARL, the cognitive dominance research area involves the following: Foreign powers are exploiting approaches to enhance soldiers’ mental capabilities that the U.S. military has determined are outside the ethical standards of operations. Without an alternative approach, the U.S. military risks losing its advantage in battlefield cognition. Merging expertise from DoD’s leading nonmedical neuroscience R&D group with expertise in machine learning, artificial intelligence, cognitive sciences, and learning sciences, CCDC ARL is leading the development of the neuro-technologies to ensure that soldiers cognitively dominate the battlefield. Technology has capabilities that even “super-humans” will not be able to achieve. Thus, CCDC ARL’s work focuses on mixed human-technology approaches that optimally exploit human and technology capabilities to create faster, more effective cognitive processing in the field. Innovations in this area are expected to seamlessly integrate soldiers and technology to produce more accurate situational awareness and situational understanding; faster, more effective decision making; and greater agility to handle threats.

2 As described by ARL, teaming overmatch refers to the following: From team cohesion and coordination, to organizational communications and shared situation awareness, through collective action within societies, group dynamics are critical to determining success in every operational environment our soldiers face. Foreign powers are exploiting knowledge of how groups of people function as well as the capabilities of artificial intelligence to shape the environment and develop operational advantages. Without countercapabilities focused on optimally exploiting group dynamics in order to shape the environment and create windows of opportunity to provide overmatch, the United States will remain at a disadvantage. In response, CCDC ARL is conceiving of technologies that enable U.S. Army units to predict and outperform their adversaries, including technologies focused on within-group (friendly) dynamics as well as prediction of enemy and civilian group behaviors.

Suggested Citation:"4 Human Sciences." National Academies of Sciences, Engineering, and Medicine. 2021. 2019-2020 Assessment of the Army Research Laboratory. Washington, DC: The National Academies Press. doi: 10.17226/26325.
×

HUMAN-AUTONOMY TEAM INTERACTIONS AND HUMANS UNDERSTANDING AUTONOMY

ARL is preparing for a future where both humans and machines are working together toward operational goals via human-autonomy teaming. ARL’s research is critical for making autonomy work for the Army. The field of human-autonomy teaming is relatively new, and ARL has an opportunity to conceptually and empirically define the trajectory for this domain of interest. There is recognition that for effective human-autonomy teams, the autonomy needs to complement and respond to human needs and responsibilities. Overall goals of ARL human-autonomy teaming work are aimed at greater team resilience with robust, adaptive performance; fast, dynamic team reconfiguration; faster, more informed decision making; and reduced risk to soldiers. Current and future work focused on these goals is likely to lead to more efficient, effective, and high-performing human-autonomy teams.

Now part of the Army Futures Command (AFC), recent changes within ARL allow for new opportunities relating to human-autonomy teaming. ARL is focused on performing basic research that contributes to what is referred to by ARL leadership as “foundational” research—described as the integration of both basic and applied work. This work encompasses both sides of the human-autonomy teaming paradigm, autonomy understanding humans and humans understanding autonomy. There is a clear understanding at multiple levels within ARL that it is necessary to focus on both the technology and the human, yet the emphasis is rightly on human performance and outcomes (especially within the HS core competency area). Specific to the HS core competency within ARL, the human-autonomy teaming group operates under the teaming overmatch thread of work, and this includes two substreams of work termed “understanding and predicting group dynamics” and “creating group synergies.” Work being conducted within these areas includes understanding variability in humans, variability in human-related technologies, human-artificial intelligence (AI) teamwork, human-robot interaction, and crew enhancement.

Accomplishments and Advancements

There are many crosscutting human-autonomy teaming projects occurring throughout ARL that are bettering the human experience in human-autonomy teaming and are working toward the development of real human-autonomy teams, where the autonomy is a fully functioning technology.

ARL has developed robust, contextualized human-autonomy teaming research laboratories. ARL has developed state-of-the-art synthetic task environments and data collection platforms through Cyber-Human Integrated Modeling and Experimentation Range Army (CHIMERA) and Information for Mixed Squads Laboratory (INFORMS). INFORMS has the potential to gather data on platoon-size teams (dual 7-person crews) that does not exist anywhere else. This could lead to very interesting science on platoon-size interactions, shared mental models, and attention allocation. In addition, the CHIMERA laboratory has outstanding metrics collection capabilities for cyber-human systems studies. ARL has significant experience and investment in neurophysiological measures to infer human states, as well as instrumented laboratories and simulation capabilities. Such advanced facilities promise to provide the ecological validity and experimental control needed to generate empirical evidence to address research questions and advance the science of human-technology integration.

ARL’s research portfolio includes an ambitious mix of human sciences and enabling autonomy. ARL is taking on an ambitious research portfolio relating to multiple aspects of human-autonomy teaming. As outlined earlier, both humans and technology need to be studied separately and in concert with each other to further human-autonomy teaming, and ARL is doing research that falls within this purview. Key problems are being explored, including autonomy adapting to humans, autonomy understanding humans, trust, spatial mental models, and estimating team outcomes.

ARL has identified expertise gaps in the composition of their research team and has a plan in place to enhance internal expertise while connecting with the broader community via the STRONG program.

Suggested Citation:"4 Human Sciences." National Academies of Sciences, Engineering, and Medicine. 2021. 2019-2020 Assessment of the Army Research Laboratory. Washington, DC: The National Academies Press. doi: 10.17226/26325.
×

STRONG refers to strengthening teamwork for robust operations in novel groups, with the goal of developing the foundation for enhanced teamwork within heterogeneous human-intelligent agent teams. The STRONG program is an innovative way to elicit expert opinion, outside ARL, to inform the human-autonomy teaming work that will occur internal and external to ARL. This unique and highly collaborative method to encourage constant and direct communication between academia and government could be examined as a future model of funding within other departments and agencies of the government. A critical introspective assessment of the successes and failures of the STRONG program will need to be conducted to ensure that the model is adjusted for maximum productivity.

Challenges and Opportunities

An Approach to Foundational Research with Unique Access to Contexts

ARL leadership has a clear and correct understanding that the scientific questions to be pursued may be shaped by or result in changes to its long-standing doctrine. The purpose of this scientific approach is not just to serve the outbuilding of technological capabilities, which may be co-opted by competitors or enemy combatants, but also to inform the overall doctrine, organization, and strategy of the Army. As previously noted, ARL is tasked with doing foundational research involving the interaction of basic and applied research. This seems an admirable goal, but the concept needs to be better defined, as both basic and applied research meld into each other, making empirically setting and testing specific goals of each project difficult.

The most difficult questions to answer are also those that are most difficult to measure appropriately, least likely to be co-opted, and most likely to provide advantage against adversaries. These include questions that will help clarify how the myriad sociotechnical factors in real-world environments shape human decision making with adversaries, where superiority of doctrine relies on emergent outcomes that derive from values, strategy, and creativity, rather than predetermined processes and mechanistic outcomes. ARL is investigating these types of questions and could beneficially conduct more work on adversarial human-autonomy teaming research. Researchers with expertise in the HS, especially in human factors, cognitive psychology, industrial and organizational psychology, and systems engineering, are best able to address these important questions, and ARL has shown leadership by building a staff with these scientific capabilities.

ARL faces extremely challenging human-autonomy problems and has unique access to the operational environments and subject matter experts to best understand the novel problems that will arise in the most complex human-autonomy teaming environments. These operational environments are a platform for providing real-world context and evaluation of human-autonomy teaming initiatives.

Continuing to Build In-House Expertise in Team Science

There is an opportunity to continue to build team science expertise in house. In order to successfully study human-autonomy teaming and provide cutting-edge applications, it is necessary to have a deep knowledge of traditional and nontraditional teaming approaches. Here nontraditional teaming is meant to indicate human-autonomous teaming. There is some expertise residing in ARL specific to human-autonomy teaming, but there is a significant need for more such expertise. Specifically, there seems to be a lack of expertise relating to human-human teaming that is needed to inform human-autonomy teaming. It is necessary to have experts in house who understand the fundamentals of teamwork to advance the domain of human-autonomy teaming.

Suggested Citation:"4 Human Sciences." National Academies of Sciences, Engineering, and Medicine. 2021. 2019-2020 Assessment of the Army Research Laboratory. Washington, DC: The National Academies Press. doi: 10.17226/26325.
×

Continuing to Build Team Science Expertise Externally

ARL’s IEEE Brain publication3 could be leveraged to continue to develop specific areas of research such as the human-machine interface and team communication within the paradigm of human-autonomy teaming. These team science-specific areas could be defined in detail to provide an organizational landscape for the future trajectory of human-autonomy teaming. These areas will also help to specifically outline short- and long-term goals. It is also necessary to keep sight of the importance of human-human teaming in studying the paradigm of human-autonomy teaming. It is expected that the STRONG program will help bolster external team science expertise.

Directing and Using Unique Access for Human-Autonomy Teaming

To achieve its stated goals, ARL is in an advantageous position with unique access to the U.S. Army’s warfighter environment, soldiers as study participants and subject matter experts, and standardized training programs to provide controlled baselines for scientific research. ARL thus has strong potential to lead the research of leveraging autonomy and AI in high-criticality complex work environments.

ARL could make more use of this access to ensure that its researchers immerse themselves and become more familiar with actual operations. In addition, it would be beneficial if there were continued interactions between customers, users, and researchers throughout projects to realize possible need for mid-course corrections. It would also be beneficial if more military personnel with combat experience were integrated into these research teams. Much of the research presented was conducted using college students as participants, instead of military personnel. ARL could either use more military personnel for experimental studies or use college students with greater knowledge of the military problem space. College Reserve Officer Training Corps programs are a much better resource for these studies than college students in other departments are.

Unbiasing Data for Human-Autonomy Teaming

The main challenge will be to carefully design unbiased experimental scenarios for ARL’s data collection environments that are not an echo chamber for proving current Army strategy or technology (although it may also be used for these purposes). The research needs to be geared for rapid advancement rooted in research questions that will provide the most return on investment for the current science.

Addressing this challenge will involve starting with fundamental science questions and then figuring out how they apply to the Army needs as opposed to the other way around. To do disruptive work, ARL needs to start at the basic science level instead of the applied problem. The latter may be easier to convey to Army stakeholders but is likely to lead to myopic results for the scientific community.

Addressing this challenge will also require input from experts in human subjects’ research with synthetic task environments. The challenge is not just to advance science and technology but also to advance understanding of whether or not the U.S. Army has the superior doctrine underlying how the technology is used. Understanding how to study and integrate critical work environment factors with knowledge of human capabilities and limitations requires specific expertise and leadership from human factors and systems engineering experts.

___________________

3 See IEEE Brain, “Enhancing Human-Agent Teaming with Individualized, Adaptive Technologies: A Discussion of Critical Scientific Questions,” https://brain.ieee.org/brain-storm/enhancing-human-agent-teaming, accessed May 11, 2020.

Suggested Citation:"4 Human Sciences." National Academies of Sciences, Engineering, and Medicine. 2021. 2019-2020 Assessment of the Army Research Laboratory. Washington, DC: The National Academies Press. doi: 10.17226/26325.
×

Therefore, basic or fundamental science questions and theories needs to be pursued by ARL, but this could be done in the context of applied or “actual” operations. Such fundamental research, perhaps on nonmilitary study participants, might be pursued first, and then be followed up with further tests in military personnel.

Balancing Near-Term Problems with Long-Term Vision

A challenge is matching the ambitious long-term vision communicated by ARL leadership with the demands of addressing near-term problems of human-autonomy teaming. The researchers seem to face a tension between addressing problems on human-autonomy teaming that affect warfighters using near-term robotic and AI technologies on one hand, and doing the basic research that leads to the kind of scholarly work and to foundational breakthroughs in human-autonomy teaming on the other hand. There seems to be a tension between trying to do long-term foundational work but at the same time having to support the development of many short-term deliverables.

AUTONOMY UNDERSTANDING HUMANS AND ESTIMATING HUMAN-AUTONOMY TEAM OUTCOMES

ARL is responsible for a wide range of interdisciplinary research projects that bring state-of-the-art research to bear on Army-unique problems. Within the HS core competency area, many projects contain an autonomy component that is designed to assess or predict human-human or human-autonomy teaming. The current portfolio focuses on specific aspects of these problems. Machine learning (ML) application focuses on fusion of multimodal sensory input for online learning or off-line modeling. Experimental and simulated frameworks leverage in situ data collection to evaluate theoretical models in real-world application. Projects in this domain include human AI interactions for intelligent squad weapons, cycle of learning for autonomous systems, and information for mixed-squad laboratories. The project on human-AI interactions for intelligent squad weapons uses advanced sensing algorithms to perform automated target recognition, which is then displayed on a scope or a heads-up display. The cycle of learning for the autonomous systems project uses Learning from Demonstration (LfD) to program and refine a quadrotor-landing algorithm. INFORMS is a dual 7-person crew station facility that simulates auditory, tactile, haptic, and visual input during an operational context. It is networked to other ARL facilities at the Aberdeen Proving Ground, for distributed operations simulation. INFORMS enables the study of human-agent teaming during operations.

Additional projects under way incorporate either state-of-the-art or black box ML. Researchers interested in human performance prediction and human interest detection (HID) leverage ML techniques in sensor fusion for human understanding. These projects include convolutional neural networks used to measure human performance through prediction of p300 variability,4 soldier gaze tracking and neural activity used to passively detect salient environment features at a team level, and methods for understanding the quality of human communications tools for annotating human conversations.

___________________

4 p300 is an event-related potential component elicited in the process of decision making. It is considered to be an endogenous potential, as its occurrence links not to the physical attributes of a stimulus, but to a person’s reaction to it. More specifically, the p300 is thought to reflect processes involved in stimulus evaluation or categorization.

Suggested Citation:"4 Human Sciences." National Academies of Sciences, Engineering, and Medicine. 2021. 2019-2020 Assessment of the Army Research Laboratory. Washington, DC: The National Academies Press. doi: 10.17226/26325.
×

Accomplishments and Advancements

The goals of the HS core competency area is unique in that research challenges lie in the application of innovations. The ability to apply autonomy in the presence of real-world noise at real-time time scales is challenging. The oversimplification and constraint removal traditionally employed by academic researchers misaligns their research to the problems faced by the military. All the HS researchers could consider the challenges and complexities of autonomy in real-world tasks.

Infrastructure, which is key in evaluating research in the Army context, is one of the biggest strengths of the HS research teams. INFORMS is a valuable and unique resource. It simulates platoon-level crew stations with haptics, auditory, display, and tactile interactions. It was conceived as a platform for testing crew interactions with intelligent agents. It provides an additional opportunity to study crew interactions and leverage insights into innovations in rapid prototyping. The Innovations Commons is an instrumented, distributed sensing environment that allows for the capture of more realistic data sets.

The neuroimaging laboratory enables research on neural input during physical activities (as would be required in this context). Several prototype electrodes are being constructed with different signal and noise characteristics. The project included development of a “phantom head” for generating controllable synthetic data, as well as development of new electrodes (using materials science) that are robust to noise. This is an important step forward for the Army because it will need to solve these challenges if it is to use electroencephalogram (EEG) sensing in the field. The phantom head allows the generation of ground truth data that can be used to validate models.

Autonomy and computation are key to research that allows for prediction in noisy, real-world conditions. Human motion is captured during walking or running and can be replayed on a hexapod for simulating human walking motions, which allows for simulation of real-world noise and the study of the motion artifact.

The project on human-AI interactions for intelligent squad weapons seeks to use advanced sensing algorithms to perform automated target recognition. Such algorithms could leverage video and other multimodal input to identify targets on the heads-up display targets and infer whether these targets are friend or foe. The algorithms would not be limited to target recognition and could grow as new capabilities and needs unfold. The research team’s key contribution is a system that could continuously acquire and store real field data for labeling and training of improved perception algorithms. Rather than try to develop better algorithms, this team seeks to fill a significant void by having a ubiquitous platform collect the right relevant data for humans to annotate at a desk or in the field in real time. The project has the right mix of AI and human science-focused researchers. The team understands well its competencies and is focused properly on an exciting opportunity to design a potentially game-changing system to acquire, curate, and leverage powerful-targeted data for training off-the-shelf computer vision algorithms. The intent is to collect data at large scale and under more complex and naturalistic task environments with novel sensors. The researchers have a solid research plan and are well-poised to deliver on a project that they are just ramping up. If the team succeeds, it will have the evidence it needs to bring support to investment in the underrepresented and overlooked area of data acquisition and curation mechanisms to feed ML algorithms. While intelligence analysts seek data for their own purposes, the process of acquiring and labeling data for training ML algorithms is different and requires its own skill set and infrastructure. This project could revolutionize the way the Army views AI/ML and, in turn, the latent power that could be unlocked by investing in such projects.

The project on cycle of learning for autonomous systems leverages learning by demonstration for robot control. The researchers take the realistic position that achieving ubiquitous robots in the hands of end-users will require a multiphased, multimodal approach to situated learning. This project presents a cyclic architecture at four levels in which (1) a robot is taught a skill first by a human demonstrating said task; (2) the human shifts to intervening in the robot attempting that task (e.g., via direct control) when failure occurs or is about to occur; (3) the human provides a grade (i.e., a reinforcement signal) to guide the robot toward better behavior; and (4) the robot refines its own policy based upon a human-encoded

Suggested Citation:"4 Human Sciences." National Academies of Sciences, Engineering, and Medicine. 2021. 2019-2020 Assessment of the Army Research Laboratory. Washington, DC: The National Academies Press. doi: 10.17226/26325.
×

grading system.5 The robot moves from level 1 to 4 as it gains proficiency, ultimately becoming autonomous.

A significant achievement includes the collection and management of large data sets (here, large means something significantly bigger than average). Ambitious data collection activities—over time, between and within subjects, in the field—with target populations have created a number of large data sets that will be used to drive ML and simulation. The team has access to good information technology infrastructure to store and protect these organized and time-stamped infrastructure components. These researchers are pioneering something new.

Researchers involved in the project on brain dynamics of driver-passenger communication presented an interesting methodology. The goal is to identify neural mechanisms that indicate effective verbal communication. The researchers observe EEG changes during communication with the goal of identifying physiological markers that relate to successful communication. Such information can perhaps be used to appropriately adjust autonomy based on human performance.

The research project on p300-passive detection of situation awareness for the battlefield environment is a good opportunity for the HS core competency area. The team demonstrates the relevant expertise to attack the problem and understanding of the limitations of current methods (e.g., task specificity, subject specificity, noise, and requirements for large data sets). The neural encoding approach shows great promise in this area, and is methodologically sound. The task selected for testing (stimuli with human targets with and without weapons) was Army-relevant.

Challenges and Opportunities

Researchers are expanding the scope of autonomy understanding individuals to include predictions of human-autonomy team outcomes. This effort is less mature, and researchers are focused on what data to collect to evaluate team effectiveness and outcomes. Initial efforts involved looking at communication effectiveness and resulted in the creation of a tool to annotate human conversations. A study of human-to-human communication under higher cognitive workloads provided an opportunity to evaluate the usefulness of the tool. The goal is to understand how to evaluate mixed team effectiveness, possibly utilizing methods identified for evaluating individual or exclusively human team performance. The choice of this approach to measure communication effectiveness is not well motivated. No evidence is presented that supports this as the most promising measure—physiological or otherwise. Rather, this approach seems to be selected based on the researcher’s interest and background.

INFORMS is a valuable and unique resource. Because of the fidelity that can be achieved, there is a danger that this laboratory could be devoted to training. It is important that this facility be reserved for research use.

Although the approach demonstrated in the cycle of learning is appropriate, none of this research is particularly novel. Each phase of interaction has been considered explicitly, repeatedly, and thoroughly in prior literature. More than 10 years ago, researchers started exploring multiphased interactions for robot learning architectures that consider learning from both observation and demonstration. The deep-learning algorithm Deep Training an Agent Manually via Evaluative Reinforcement (Deep TAMER) itself is not particularly game changing. Over the past 5 to 10 years, the field of AI has seen numerous reboots of algorithms that leverage neural networks as a preprocessing step or in substitution for a different function approximation technique. These steps are important, but they are nonetheless incremental.

With respect to the project on cycle of learning, the team’s primary demonstration mechanism was teaching an unmanned aerial vehicle (UAV) to land on top of an armored troop transport. UAVs today

___________________

5 N.R. Waytowich, V.G. Goecks, and V.J. Lawhern, 2018, Cycle-of-learning for autonomous systems from human interaction, https://arxiv.org/abs/1808.09572.

Suggested Citation:"4 Human Sciences." National Academies of Sciences, Engineering, and Medicine. 2021. 2019-2020 Assessment of the Army Research Laboratory. Washington, DC: The National Academies Press. doi: 10.17226/26325.
×

typically ship with autopilot controls and could easily be programmed through nonlearning techniques to find a landing zone and land without collision just as commercial aircraft can leverage instrument landing systems. It became apparent that the genesis of trying to apply LfD to this landing problem was from the military, because soldiers were struggling to manually land a UAV on the back of their vehicle, and wanted this research team to solve that problem. The research team’s strength is developing LfD algorithms. Thus, the team is left with either trying to develop a solution outside its research focus or trying to use LfD algorithms for a problem that does not warrant such techniques. The program would benefit by management direction to emphasize human science-focused problems, while better matching open source ML or engineering approaches to the specific research problems. Although this issue was primarily observed in one project, all projects that leverage black box ML techniques or those that require novel ML techniques could create a distraction around the existing dedicated focus on human sciences research.

HUMAN INTEREST DETECTION

The specific objective of the human interest detection (HID) program is to use passively collected data from human sensory systems in the field to make conclusions about the elements of a sensed scene that are of interest to viewers (e.g., discovering blind spots and focusing attention). In team situations, this effort may involve combining diverse views into a single model of the scene.

The wider objective is to develop the infrastructure for real-world neuroscience. The infrastructure will allow both a broader range of sensing inputs and analytics to be incorporated, and it will support future applications built on the infrastructure. HID was listed as a future program, but the current effort emanates from the Cognition and Ergonomics Collaborative Technology Alliance (CTA), which is an approximately 10-year effort.6 Outputs from that CTA—for example, a convolutional network for EEG-based brain-computer interfaces (EEGNet)—are being leveraged for this new program.

Examples of work that was presented include detection of objects of interest from multiple streams of data collected from the visual system of soldiers and EEG, detection of instances of target detection by a human using EEG (p300 visual-evoked potentials), and improved EEG signal acquisition.

Accomplishments and Advancements

HID is an important area in the ARL research program. The research is uniquely applying real-world neuroscience and AI to the area of attention. Building on a 10-year or so neuroscience program, the research team is taking the kinds of risks that are appropriate. These efforts and risks include the following: taking a plausible but rich use-case set and demonstrating feasibility, understanding and modeling HID, and exploring alternative inputs and algorithmic approaches over time to find solutions that will scale. There has already been successful technology transfer of some of the HID team’s work. The new HID program is leveraging innovative tools developed from this past work. The HID effort is performed in partnership with external laboratories and contractors. The studies and results in the HID area are interesting and constitute an exciting start on the journey toward the new HID program. The HID research team’s platform approach is an effective way to manage the risk of failure in a particular line of exploration by ensuring that work that was developed and documented can be reused if the project needs to pivot to take another direction.

___________________

6 See CaN CTA, “The Cognition and Neuroergonomics Collaborative Technology Alliance,” https://cancta.net, accessed May 11, 2020.

Suggested Citation:"4 Human Sciences." National Academies of Sciences, Engineering, and Medicine. 2021. 2019-2020 Assessment of the Army Research Laboratory. Washington, DC: The National Academies Press. doi: 10.17226/26325.
×

The HID team on EEGNet and studies on detection of p300 for target detection in humans did significant feasibility work. The convolutional neural network implemented by the HID team for EEGNet is a useful tool for this research and other studies.7 The merit of the effort transcends the specific project shown, in that it paves the way for similar future real-world neuroscience.

Challenges and Opportunities

ARL is in the early days on the HID program, so it is not too surprising that the specific projects presented would require several technical obstacles be overcome. These obstacles include the need for more efficient and convenient collection of data in the field (especially EEG signals), overcoming motion artifacts in the data, and the use of adaptive decision fusion for decision making.

There are several projects in other programs that are not officially under the umbrella of HID but could be included in the exploration of this area. As the HID research team grows the program, leveraging links to related projects could be advantageous. HID researchers are already coordinating across some other programs and teams.

Members of the HID research team observed, rightly, that deployment of their algorithms and methods will depend on leveraging lines of research undertaken by external companies, and on emerging commercial technical solutions and tools. For example, IBM, Facebook, Microsoft, and others are working on a variety of EEG and other wireless sensing devices. The HID team understands that there is an opportunity to find better ways to influence industry to ensure that technologies that will be needed to deploy HID solutions will be available when needed. This understanding may lead to the development of a new model of interaction with industry in the area of HID.

In the future it will be important to explore and prioritize other inputs for HID, such as speech between members of soldier teams, auditory input (which influences visual attention), and situational and automatic emotional states and responses. These inputs can be used to identify the most effective set of data and analytics and help create useful applications.

The basic idea of using passive measurements from individual and collections of warfighters in order to improve situation awareness and identify blind spots is attractive. It also offers a test-bed for a variety of other important application areas such as agent assistants and future human-agent teaming scenarios. The principal challenges include implementation of the EEG sensing mechanisms in a manner that would be minimally inconvenient to the subjects; ensuring that the signal processing module can detect the signal of interest with sufficient reliability (e.g., with sufficiently high signal-to-noise ratio especially in the face of the inevitable motion artifacts); facilitating automatic or near-automatic operation in the field (i.e., without need for a human guide to focus the attention of the system when operating in the field, establishing a common view by subjects and targets of interest); developing a data and decision fusion framework for scene analysis from multiple inputs including identification and isolation of nonparticipatory players and scene-change detection (this task would involve adaptive decision fusion); and scaling (i.e., the work presented so far involved a single subject or a small group of subjects in laboratory settings, and implementation of the algorithms is required in more realistic environments and in larger scales).

The success of an HID system depends on a higher-level fusion module that would interpret the results of the detection subsystems. Devising this module is challenging because it requires a priori definition of targets (versus clutter and interference) as well as a high level of agreement between the human sensors on their classification of potential targets (what is important and what is threatening). Data

___________________

7 For example, see V.J. Lawhern, A.J. Solon, N.R. Waytowich, S.M. Gordon, C.P. Hung, and B.J. Lance, 2018, EEGNet: A compact convolutional network for EEG-based brain-computer interfaces, arxiv.org/abs/1611.08024.

Suggested Citation:"4 Human Sciences." National Academies of Sciences, Engineering, and Medicine. 2021. 2019-2020 Assessment of the Army Research Laboratory. Washington, DC: The National Academies Press. doi: 10.17226/26325.
×

fusion or decision fusion would be needed, with the ability to adapt (e.g., tune out a participant who is not observing the same scene that one’s team members observe).

The HID project team was open to considering other inputs that could help interpret the EEG and gaze data. One potential area is collecting and leveraging the conversations between soldiers in the field as they communicate about what they are seeing and its potential importance. Several projects currently under way elsewhere in ARL are focusing on how to analyze conversations such as these, and they could potentially benefit the HID program.

Given that decision making in the field would depend on many other inputs and signals, it would also be desirable to qualify the expected contribution of the human interest detector system to scene analysis quality—that is, how inclusion of input from the human interest detector may affect the probability of detection, probability of false alarm, and the receiver operating characteristic (ROC) curve.

The potential of real-world neuroscience to disrupt current situation awareness technology is high, given that real-world neuroscience may provide measurements and human reasoning about sensed scenes that are not available at present from other sources. Realization of this potential requires overcoming technical obstacles—among them developing high-quality wearable sensing data collection systems that are robust to motion and noise. The availability of signals that are easier to collect, such as communication between soldiers, competes with this HID technology.

CYBER SCIENCE AND KINESIOLOGY

ARL’s cyber science group focuses on the human aspects of cyber in support of three thrusts: training assessment, informed technology selection, and development of cyber operational plans (OPLANs). The goal of the kinesiology group is to improve the understanding of physical human performance in order to optimize soldier performance.

Accomplishments and Advancements

The cyber team has made good progress in the 2 years since their work was previously reviewed by the ARLTAB. Cyber science is getting considerable focus and funding in commercial industry and across the DoD and Services. The ARL cyber team has done a good job of finding an important research niche when most focus elsewhere is on building tools.

With its performance assessment suite for the Cyber Mission Force (CMF), ARL is tackling the challenging problem of assessing the performance of the CMF. Assessing effectiveness of these teams and their training is more challenging for this mission than for more tangible, kinetic missions.

The cyber science group has done a good job of identifying and establishing partnerships with organizations including the Persistent Cyber Training Environment, the National Guard’s CyberShield exercises, USCYBERCOM, and the Cyber Center of Excellence.

Their concept of understanding operators before building tools is unlike the approach taken by most tool developers in the space. This concept for supporting insertion of automation dovetails well with ARL’s larger focus on human-agent teaming while focusing on how this applies specifically to cyber.

The continued focus that the cyber team is placing on developing new behavioral measures and metrics is commendable. Throughout its work, there is thoughtful consideration in developing and integrating different measures in their attempt to understand human behavior while engaged in demanding cyber tasks. With questionnaires, eye tracking, successful cyber task completion, location tracking, and other methods, the team is continuing to make advancements that improve understanding of human-cyber behavior. Quality behavioral measures and metrics are foundational to advancing scientific progress in this field, and the cyber team is making important contributions in developing this science at the individual level and at the team level. It is not clear how this work could support ARL’s objective of supporting cyber operational planning.

Suggested Citation:"4 Human Sciences." National Academies of Sciences, Engineering, and Medicine. 2021. 2019-2020 Assessment of the Army Research Laboratory. Washington, DC: The National Academies Press. doi: 10.17226/26325.
×

The kinesiology group is developing methods for objective, quantitative characterization of the ways in which soldiers move and need to be able to move in order to perform a range of tasks that could be supported by physical augmentation systems such as an exoskeleton. The current focus is close combat operations. This will allow movements to be objectively compared between conditions where soldiers are and are not wearing augmentation. The group plans to determine which kinematic variables are most informative. This quantitative approach is an important advancement over prior qualitative ways in which task performance has been evaluated. It is important to provide this type of information to exoskeleton designers so that they fully understand these important requirements of soldiers’ equipment. This work can have important implications for military and civilian applications of exoskeletons. The capacity for performing quantitative assessment of effectiveness can also be applied to other means of enhancing soldier physical performance.

Of particular note is the project on team performance in a series of cybersecurity defense competitions: generalizable effects of training-type and functional role specialization, with the team actively engaging with collegiate cyber teams and collecting data. The active, deep engagement of collegiate cyber teams, as a source of input data and as test subjects for the results of other projects, is particularly creative given their skill sets and the availability challenges of operational cyber protection teams (CPTs).8 The progress ARL is making in building relationships with the National Cyber Collegiate Defense Competition teams will serve it well as it continues this promising line of research.

Challenges and Opportunities

Being disruptive in the cyber domain overall is challenging given the level of funding spent by commercial industry and the long lead time of the government acquisition system. There appears to be a tension between ARL’s stated mission of fundamental research and the push to meet the near- to midterm requirements of the Army’s cross-functional teams. The project on team performance in a series of cybersecurity defense competitions could be extended to include collection of team training history to build a more complete set of metrics that might affect team performance in addition to the individual team member training history.

The Army (and the whole of DoD) is in the uniquely challenging position of having deep and broad requirements for cyber warriors, while not having the funding to compete with industry for the most qualified individuals. Nothing suggests that this situation will change. As such, the Army needs to fill the gaps itself. The cyber science group has the opportunity to help the Army grow better cyber warriors by ensuring that the Army’s training and the resulting teams are effective. ARL needs to leverage what is being done by industry as well as use open source tools for such leveraging.

To align more with ARL’s human-agent teaming goal, the project on explainable AI has the opportunity to apply AI/ML to improve the quality of the knowledge base by applying feedback from the user rather than just informing the user based on an extant knowledge base. More generally, the project has an ambitious scope and needs to be tracked carefully to ensure that project milestones are met.

New projects that the kinesiology group is planning blend neuroimaging with biomechanics to understand how the body perceives constraints imposed by exoskeletons. This could lead to new training techniques for soldiers learning to work with an exoskeleton and eventually to smart exoskeletons that could adapt and respond to and even perceive the intent of the user. This is consistent with the overall goal of the Army, which is that technology needs to adapt to the user.

This research could also contribute and lead to other types of human-technology teaming, such as human-robot soldier teaming and the training of robot soldiers to perceive intent of enemy combatants.

___________________

8 Although not within the mission of ARL, this could also represent an opportunity for the Army to recruit future CPT members.

Suggested Citation:"4 Human Sciences." National Academies of Sciences, Engineering, and Medicine. 2021. 2019-2020 Assessment of the Army Research Laboratory. Washington, DC: The National Academies Press. doi: 10.17226/26325.
×

The key challenge for the group is in the uncertainty of who or what the future soldier will be, human or robot or a blend. The group needs to continue to identify ways of supporting the physical performance of tasks that will continue to be allocated to human soldiers, and the group can play an important role in identifying those tasks at which human soldiers (with or without augmentation) will excel over robotic soldiers.

NEUROSCIENCE, TRAINING EFFECTIVENESS, AND STRONG

The role of the HS core competency research is to create individualized, adaptive technologies to augment or enhance the cognitive functions of the warfighter. Within this broader goal, the HS research portfolio includes a variety of neuroscience research programs aimed at understanding factors affecting warfighter performance in a variety of domains related to performance of the individual and the group as a collective unit. The HS core competency area comprises a team of psychologists, neuroscientists, and biomedical engineers, and their competencies include measurement and analysis of electrophysiological and behavioral signals in humans immersed in complex, real-world testing scenarios.

Accomplishments and Advancements

The group has established a unique and valuable niche for engineering advances for neurophysiological monitoring in dynamic tasks, such as ambulatory EEG and eye-gaze tracking. Specific contributions include novel algorithms for achieving accurate, reliable, and online detection of evoked response signals in the EEG while subjects are engaged in complex, operationally relevant activities. The group has also made novel contributions to the hardware and software technologies for EEG monitoring, particularly in mobile scenarios, which are prone to contamination from motion and environmental noise sources. The group has taken a rigorous approach to addressing these problems, developing a novel testbed for isolating and eliminating sources of noise through innovative electrode and signal processing strategies.

The group has also created unique testing platforms for studying human performance in complex tasks involving teaming among groups of humans and autonomous vehicles in ecologically valid settings. These platforms are generating data sets that are unique and exceedingly rich in measuring physiological and behavioral aspects of human performance, spanning multiple time scales and modalities. In addition to supporting immediate questions, these data sets could be leveraged to support secondary analyses within and beyond ARL.

Challenges and Opportunities

The researchers do not think of themselves primarily as engineers building innovative systems, but as researchers who plan to use the systems that they are building now to expand the bounds of cognitive science, especially as it can be applied to the learning and training of teams and individuals. There are three important scientific purposes that this work might address: machine modeling as it relates to brain areas, contributions to cognitive theory as it relates to joint actions between pairs of team members (human-human or human-robot pairs) and to integration of perception with action in predictive processing, and archiving the nearly unique set of individual and team performance data that this work is and will be generating.

Suggested Citation:"4 Human Sciences." National Academies of Sciences, Engineering, and Medicine. 2021. 2019-2020 Assessment of the Army Research Laboratory. Washington, DC: The National Academies Press. doi: 10.17226/26325.
×

Machine Modeling of Human Cognition

The past 10 years have witnessed a merger of machine learning methods with the modeling of brain functions. The discussion here is limited to two efforts: adaptive control of thought—rational (ACT-R) and NENGO (a Python tool for building large-scale functional brain models).

Although ACT-R has existed since 1993, the neuro-version of ACT-R is much more recent. Carnegie Mellon University (CMU) has recently applied ACT-RN (where “N” is added for “neuro”) to examine brain processes via three highly varied types of technologies: magnetoencephalography, EEG, and functional magnetic resonance imaging.9 For magnetoencephalography, a combination of hidden semi-Markov models and multivariate pattern analysis is used to locate brief bumps in the sensor data that marked the onset of different stages of cognitive processing. For EEG, CMU developed a method that identifies on a trial-by-trial basis where brief sinusoidal peaks (i.e., bumps) are added to the ongoing electroencephalographic signal. It was proposed that these bumps mark the onset of critical cognitive stages in processing. For functional magnetic resonance imaging, the CMU researchers combined multivoxel pattern analysis to identify cognitive stages and hidden semi-Markov models to identify their durations.10 When applied to a problem-solving task, this method identifies four distinct stages of cognitive processing—encoding, planning, solving, and responding.

A neuroengineering framework was provided to explore how cognitive processes can be implemented in a biological substrate.11 This work has been influential, with publications in Science12 and other journals.13,14

In addition to this group’s many publications, both the ACT-RN and NENGO projects have provided tutorials at annual conferences and summer schools, and they freely share their software. HS scientists interested in developing applications for soldiers whether as individuals or as teams (human-human or human-robot) would be welcomed into these communities of researchers.

Advancing Cognitive Theory

A side effect of ignoring the cognitive modeling community may be the lack of awareness of changes in areas of research developed over the past 15 years as well as the concomitant changes in basic cognitive theory. At least two of these changes are directly relevant to ongoing HS research—joint action and predictive processing.

Joint action refers to the interactions of two individuals (human-human usually, but human-robot increasingly) at the level of eye movements, actions, and general coordination.15,16 A joint action is an

___________________

9 J.R. Anderson, J.P. Borst, J.M. Fincham, A.S. Ghuman, C. Tenison, Q. and Zhang, 2018, The common time course of memory processes revealed, Psychological Science 29(9):1463-1474, PMID: 29991326.

10 J.R. Anderson, A.A. Pyke, and J.M. Fincham, 2016, Hidden stages of cognition revealed in patterns of brain activation, Psychological Science, https://doi.org/10.1177/0956797616654912.

11 T. Bekolay, J. Bergstra, E. Hunsberger, T. DeWolf, T.C. Stewart, D. Rasmussen, X. Choo, A.R. Voelker, and C. Eliasmith, 2014, NENGO: A Python tool for building large-scale functional brain models, Frontiers in Neuroinformatics 7, https://doi.org/10.3389/fninf.2013.00048.

12 C. Eliasmith, T.C. Stewart, X. Choo, T. Bekolay, T. DeWolf, C. Tang, and D. Rasmussen, 2012, A large-scale model of the functioning brain, Science 338(6111):1202-1205.

13 T. DeWolf, T.C. Stewart, J.-J. Slotine, and C. Eliasmith, 2016, A spiking neural model of adaptive arm control, Proceedings of the Royal Society B-Biological Sciences 283(1843), https://doi.org/10.1098/rspb.2016.2134.

14 T.C. Stewart, T. Bekolay, and C. Eliasmith, 2012, Learning to select actions with spiking neurons in the basal ganglia, Frontiers in Neuroscience 6, https://doi.org/10.3389/fnins.2012.00002.

15 N. Sebanz and G. Knoblich, 2009, Prediction in joint action: What, when, and where, Topics in Cognitive Science 1(2):353-367.

Suggested Citation:"4 Human Sciences." National Academies of Sciences, Engineering, and Medicine. 2021. 2019-2020 Assessment of the Army Research Laboratory. Washington, DC: The National Academies Press. doi: 10.17226/26325.
×

event grounded by two or more agents’ actions.17 In this definition, it does not matter what the “joint” reaction or action is. It could be fish swimming together or arms swinging, but the cues are all nonverbal communication during teaming. When two people swing their arms in synchrony, the event of their swinging their arms is a joint action. Likewise, if fish are agents, then the movements of a shoal are joint actions. Well beyond fish and arms, joint action focuses on nondeliberate actions such as the coordination of eye movements when two humans work as a team on a physical task wherein the nonverbal, point-of-gaze of one human attracts the point-of-gaze of the other human without deliberate thought or conscious awareness by either team member. A critical review of the emerging action simulation theories in the wide-ranging embodied cognition and motor cognition literatures provides an integrative neuro-computational account of action simulation that links it to the neural substrate and to the components of a computational architecture that includes internal modeling, action monitoring, and inhibition mechanisms.18

A second and related omission is predictive processing (PP).19,20 PP builds on a series of studies begun in the 1990s that showed that action simulation and imagery could be neurally realized by the same brain mechanisms that control the execution of overt actions. It has taken about 15 years for the implications of this work to be fully understood. Acceptance of this work has contributed to blurring the traditional separation between perceptual, cognitive, and motor domains and has resulted in assigning sensorimotor simulation a prominent role in higher cognition.

The insights of PP have been codified as internally generated sequences of structured, multineuron firing patterns forming components of goal-directed decision systems that implement a type of sampling-based inference engine that optimizes goal acquisition at multiple time scales of online choice, action control, and learning. Although much of the evidence for PP is brain-based, its implications are applicable to the external world. An excellent introduction to PP is a tutorial paper by an international group of primarily transportation researchers, which introduces PP in the context of automobile driving.21 The mix of humans and robots, tools, and data available to HS researchers is much richer and could be leveraged into a productive line of research that furthers both basic and applied goals.

Data Archiving

The behavioral and physiological data collected by the HS researchers are considered a national treasure. HS researchers could archive those data using modern data archiving procedures. Examples of existing archives include the German Chess databases,22 the archives of the Open Science Foundation,23 and the archive established and endorsed by the American Psychological Association.24

___________________

16 G. Knoblich, S. Butterfill, and N. Sebanz, 2011, “Psychological Research on Joint Action: Theory and Data,” in B.H. Ross, editor, Psychology of Learning and Motivation: Advances in Research and Theory, volume 54, Burlington, MA, Academic Press, pages 59-101.

17 S.A. Butterfill, 2017, “Coordinating Joint Action,” in The Routledge Handbook of Collective Intentionality, London, Routledge, pages 68-82.

18 G. Pezzulo, M. Candidi, H. Dindo, and L. Barca, 2013, Action simulation in the human brain: Twelve questions, New Ideas in Psychology 31(3):270-290.

19 A. Clark, 2013, Whatever next? Predictive brains, situated agents, and the future of cognitive science, Behavioral and Brain Sciences 36(3):181-204.

20 K. Friston, 2018, Does predictive coding have a future? Nature Neuroscience 21(8):1019-1021.

21 J. Engstrom, J. Bargman, D. Nilsson, B. Seppelt, G. Markkula, G.B. Piccinini, and T. Victor, 2018, Great expectations: A predictive processing account of automobile driving, Theoretical Issues in Ergonomics Science 19(2):156-194.

22 N. Vaci and M. Bilalic, 2017, Chess databases as a research vehicle in psychology: Modeling large data, Behavior Research Methods 49(4):1227-1240.

Suggested Citation:"4 Human Sciences." National Academies of Sciences, Engineering, and Medicine. 2021. 2019-2020 Assessment of the Army Research Laboratory. Washington, DC: The National Academies Press. doi: 10.17226/26325.
×

OVERALL QUALITY OF THE WORK

Human-Autonomy Team Interactions and Humans Understanding Autonomy

The technical quality of some of the research, observed primarily through one-on-one technical conversations with individual ARL researchers (e.g., during the poster sessions, and a select few studies presented), appeared to be high and on par with quality research being conducted at other leading research institutions and universities. In general, there is a wide variance in the quality of research being conducted.

Scientific communication at the programmatic level seems to be geared toward front-line stakeholders of the research, as opposed to the broader scientific community. A rigorous understanding of the human sciences concepts and applications being pursued, definitional fidelity, and identifying the conceptual gaps at the forefront of human-autonomy teaming was missing. At times, it was difficult to evaluate both the potential science and innovativeness in much of the research programs presented owing to these missing elements.

Much of the work on human-autonomy teaming specifically appeared to echo past research using different terminology but similar concepts, or to iterate on research coming from top laboratories in this area, rather than driving innovation in this space and asking novel questions that will shape the next decade of research in this area.

ARL researchers are ambitiously engaged in learning and applying state-of-the-art AI/ML approaches in the effort to create advances in autonomy that enable human-autonomy teaming. For some of the researchers, the AI/ML work appears to be a second language, meaning that their formal education only tangentially relates to the algorithms that need to be developed. Consequently, some of the AI/ML work is more replication of existing work than innovative research that is likely to lead to major advancements.

Autonomy Understanding Humans and Estimating Human-Autonomy Team Outcomes

Many researchers demonstrate an in-depth understanding of the unique challenges in their research. Researchers on the brain dynamics of driver-passenger communication, human-AI interactions for intelligent squad weapons, and trust in automation efforts demonstrate a broad understanding of the state of the art. However, a broad understanding of the state of the art was not uniformly demonstrated.

The research group is highly interdisciplinary, including sociologists, neuroscientists, mathematicians, psychologists, and researchers comfortable with the application of black box ML techniques. The project on human-AI interactions for intelligent squad weapons is an example of how an appropriate mix of AI and human science-focused researchers can tackle an interdisciplinary problem.

The researchers demonstrate the ability to identify the civilian research questions that can be studied. This is evidenced by the breadth of publications styles and venues, which is what would be expected of high-quality research. Publications are a mix of human factors, neuroengineering, and other area specific journals but include domain-specific publications like American Journal of Infection Control. Projects are an appropriate mix of low-level, single-investigator work and large-scale team efforts (e.g., the driving simulator project had issues involving integration of neuroscience hardware with data collection and synthesis). Researchers are a mix of strong experts, including experimentalists, human factors practitioners, experts in application of neuroscience techniques to human problems, and sociological research methodologists (quantitative and qualitative methods).

___________________

23 See COS, “About the Center for Open Science,” https://osf.io, accessed May 11, 2020.

24 See American Psychological Association, “Access to Archives of Scientific Psychology Data,” https://www.apa.org/pubs/journals/arc/data-access, accessed May 11, 2020.

Suggested Citation:"4 Human Sciences." National Academies of Sciences, Engineering, and Medicine. 2021. 2019-2020 Assessment of the Army Research Laboratory. Washington, DC: The National Academies Press. doi: 10.17226/26325.
×

Human Interest Detection

Significant progress was made toward feasibility, but substantial challenges remain. These include robust collection of sensory information in the field, overcoming motion artifacts, and attention focusing and decision fusion architectures that have not yet been developed. The work has contributed key tools for improved EEG measurement techniques and improved signal processing tools (e.g., the convolutional neural network used in EEGNet). The work has not yet characterized fully the quality of binary target detection (the trade-off between probability of detection versus probability of false alarm and the development of the receiver operating characteristic [ROC] curve).

Cyber Science and Kinesiology

Evaluation of the technical quality of the cyber science work may be somewhat premature given that this effort is relatively new. The team established its goal as developing quality human-cyber metrics 4 years ago, and it is continuing to pursue that goal. The kinesiology group has state-of-the-art facilities, equipment, and software to support this work. The project on performance assessment suite for the cyber mission force and generalizable effects of training-type and functional role specialization appear to be on the right track to collect the data necessary to support the stated research goals. The project on explainable AI applied as a junior threat and vulnerability analyst has identified several good, authoritative sources of data to support the project, but it could have made more progress in 1.5 years. The project on participation shifts explaining degree distributions in a human communications network does not appear to be well-conceived with respect to Army-specific problems in the social network domain, nor does it appear to be doing work that advances the state of the art.

Given the time scale at which progress is made in the commercial and adversary cyber domain, it would be nearly impossible for ARL to be disruptive in this space considering the inherent time scales of research and DoD funding cycles. Most work in the cyber domain is focused on building tools and execution with less focus on evaluating the effectiveness of either.

Neuroscience, Training Effectiveness, and STRONG

Most of the research work was described as building the infrastructure required to conduct a variety of studies under conditions as close as possible to actual combat. While the group used various mathematical and ML-style modeling techniques, it seems largely unaware of the progress being made in the neuromodeling of human behavior, especially by the ACT-R and the NENGO communities. This may reflect a lack of group background in brain studies and cognitive neuroscience. Likewise, two of the most innovative approaches to integrating cognition, perception, and action are predictive processing theory and joint action, with joint action being an extension of predictive processing to the cognitive, perceptual, and action interactions between two intelligent beings. The group has access to whatever equipment it believes it needs. This access extends to custom-built nonmobile vehicles for future studies of mixed mobile teams of human and robot vehicles. The facilities and equipment are state of the art. The current research approach is not theoretical in that data are collected and analyzed but with a very task-oriented focus. This style of work is common among human factors groups in which finding a solution to a current, well-specified problem is more important than contributing to the development of theory. The main emphasis of the work was on computation and experimentation, not theory.

Suggested Citation:"4 Human Sciences." National Academies of Sciences, Engineering, and Medicine. 2021. 2019-2020 Assessment of the Army Research Laboratory. Washington, DC: The National Academies Press. doi: 10.17226/26325.
×

RECOMMENDATIONS

ARL needs to continue to focus on its long-term vision for advancing basic science research in human sciences. Its leadership needs to engage in more dialogue regarding its research strategies with its front-line researchers as well as the greater scientific community. An approach that incorporates both bottom-up and top-down approaches to science would strengthen the program and allow ARL to advance its position to the forefront of basic research.

To inform the quantitative work, ARL needs to focus its efforts on more rigorous understanding of the qualitative concepts and in rigorous qualitative analysis. The former captures the forest, and the latter grows the trees. ARL researchers need to experience the environments that its stakeholders experience firsthand, to help enrich their appreciation of the emotional and physical complexity of the work environment constraints, and to elucidate the generalizability of their potential research questions. This experience would provide the needed complexity for grounding and generating innovative science within the environment that the science is meant to impact.

Recommendation: The Army Research Laboratory (ARL) should consider formal mentoring mechanisms that can assist junior developers with designing methodologies that appropriately place work in the context of the state of the art. ARL should examine the mechanism by which individual projects are selected. ARL should ensure that individual research projects collectively contribute to research questions identified on the roadmap. ARL should select individual projects based on existing evidence that outcomes will likely contribute to the open research questions. ARL should determine the appropriate mechanism for conducting research that leverages the state of the art in both human sciences research and machine learning research.

Human-Autonomy Team Interactions and Humans Understanding Autonomy

ARL needs to continue to collaborate with universities through its state-of-art data collection facilities and through the STRONG program. Doing so would develop the top talent needed among the next generation of researchers to realize ARL’s long-term research vision for advancing U.S. Army capabilities and to broaden the pool of trainees familiar with the unique challenges the Army faces. As with the ARL Open Campus Initiative, the richness of data that could be collected through ARL facilities is an inexpensive way to collaborate and if strategically planned could support several different laboratories asking the same questions from different angles to more robustly inform the research in these multifactorial environments with emergent outcomes.

It is unclear how the processes for collaboration and coordination with external researchers through the STRONG program will be measured and precisely how this program will lead to innovative advancement over incremental science. The processes through which collaboration occurs is critical. ARL needs to more clearly define the expected outcomes and to conduct a rigorous and unbiased self-study of this program to determine whether this mechanism and its processes are effective for innovative research.

Human-autonomy teaming needs to include many team phases, including mutual training, planning, execution, and after-action reviews. “Training” in the previous sentence is not used in the sense of procedures that are formally specified in another part of ARL, but rather highlights the need for most teams to develop shared understanding and expectations through shared experiences. Algorithms designed through ML based on a preconceived notion of team behavior may not lead to resilient and adaptive human-autonomy teams, so future research needs to consider how mutual adaptation of human to autonomy and vice versa leads to resilient and adaptive human-autonomy teams.

ARL needs to identify and work with traditional well-founded researchers of teamwork and newer researchers focused on this area. In general, it is important to work with experts beyond the long-

Suggested Citation:"4 Human Sciences." National Academies of Sciences, Engineering, and Medicine. 2021. 2019-2020 Assessment of the Army Research Laboratory. Washington, DC: The National Academies Press. doi: 10.17226/26325.
×

established and funded experts. It is also important to identify interdisciplinary researchers who have background in the many disciplines relating to human-autonomy teaming.

The continued success of ARL and the human sciences core competency is critical to ensuring that human-autonomy teaming does not end up being solely technology centered.

Understanding the communications and interactions between human-machine agent teaming is critical for the operational success of teams and system performance. In addition, if such systems are to become disruptive, it is imperative that designers of such systems understand the various dimensions of human-machine interactions such as intent, trust in automations, and so on. Similarly, designers of such systems need to anticipate how the machine will respond to the variability in human responses. Therefore, a sustained research concentration on human-autonomy teaming is critical.

Recommendation: The Army Research Laboratory (ARL) should consider giving priority for the interaction of multiagent teaming in human sciences research efforts. Such projects should provide input (insights, understanding, data, etc.) that results in a better understanding of human-autonomy interactions.

Autonomy Understanding Humans and Estimating Human-Autonomy Team Outcomes

Many researchers demonstrate an in-depth understanding of the unique challenges in their research. Researchers on the brain dynamics of driver-passenger communication, human-AI interactions for intelligent squad weapons, and trust in automation efforts demonstrate a broad understanding of the state of the art. However, a broad understanding of the state of the art was not uniformly demonstrated.

Researchers are expanding the scope of autonomy understanding individuals to include predictions of human-autonomy team outcomes. This effort is less mature, and researchers are focused on what data to collect to evaluate team effectiveness and outcomes. Initial efforts involved looking at communication effectiveness and resulted in the creation of a tool to annotate human conversations. A study of human-to-human communication under higher cognitive workloads provided an opportunity to evaluate the usefulness of the tool. The goal is to understand how to evaluate mixed team effectiveness, possibly utilizing methods identified for evaluating individual or exclusively human team performance. The choice of this approach to measure communication effectiveness is not well motivated.

Human Interest Detection

Presentation of ARL’s current HID research efforts in context is needed. ARL needs to provide past work found in the literature, including previous work in attention focusing and ARL-supported work (especially in the Cognition and Ergonomics CTA).

A framework is needed for binary target detection using the human interest measurements (probability of detection versus probability of false alarm and ROC); this framework is likely to require adaptive decision fusion.

A clear roadmap is needed, including potential dependency on related research from other programs, metrics for success, milestones, and timelines. Milestones could include 3-5 realistic test scenarios specifying how many subjects would be using the HID system in a field test, what tasks and targets will be looked at, what form factor and other wearability constraints would need to be met by the HID system, and what ROC or probability of detection/probability of false alarm performance points on the ROC curve would be achieved.

Suggested Citation:"4 Human Sciences." National Academies of Sciences, Engineering, and Medicine. 2021. 2019-2020 Assessment of the Army Research Laboratory. Washington, DC: The National Academies Press. doi: 10.17226/26325.
×

Software development has advanced at a tremendous pace over the past few years. Much of the reason for this rapid development is the increasingly common practice to employ open source software to build software platforms. Because of this rapid pace, it will be difficult for ARL to remain competitive within the software development space. ARL needs to be a member of GitHub25 if it is not already.

Recommendation: The Army Research Laboratory (ARL) should develop a mechanism for collaboration between ARL and industry on software development. Specifically, ARL should use and develop software platforms in collaboration with open source software libraries that will enable ARL to keep up to date and to rapidly develop human interest detection (HID) software. In addition, given the difficulty of attracting top talent in the HID space, ARL should make a focused effort to attract high-quality talent through sabbaticals, postdoctoral fellowships, and workshops.

Cyber Science and Kinesiology

The cyber science group has made good progress in 2 years, including identifying areas where the Army has unique challenges. The cyber science group could continue to extend its work on individuals to teams in the areas of performance assessment and tool selection. Additionally, ARL could continue its progress in innovating and integrating new human-cyber behavioral measures and metrics, because that research is fundamental to advancing the science. The cyber science group could extend its external teaming efforts to identify and engage with Army organizations responsible for development of cyber operational plans so that its thrust in this area has a transition target. The cyber science group brings a unique skill set to the Army that could be left intact or allowed to grow to advance the science. The cyber science group possesses a skill set that could be attractive to other groups—for example, training or computer science. If that skill set were siphoned away into other groups within the Army, the human science that the group is advancing will suffer.

The kinesiology research strives to ensure that physical agents such as exoskeletons are designed based on how humans need to move so that they can team well with the warfighter. For the future work that the group envisions, studying smart exoskeletons is an interesting path to pursue. However, this research effort needs to better understand the musculoskeletal “cost” of the technology. Supporting the physical performance of tasks that will continue to be allocated to human soldiers is an important contribution of this group. The group can play an important role in identifying those tasks at which human soldiers (with or without augmentation) will excel over or be able to keep pace with robotic soldiers, and vice versa, as robotic technology advances.

Recommendation: The kinesiology researchers should consider the top-level question of what is different about kinematics for the Army rather than being pulled solely by ARL’s strong focus on human-robot teaming.

Neuroscience, Training Effectiveness, and STRONG

Changes made in the personnel, organization, and structure of the HS core competency area have provided the tools and opportunities needed to make it a national laboratory in the area of cognitive science and robotics for individual human performance as well as for human teams and mixed teams of humans and robots. The funding directed to this group by the Army has enabled the researchers to build

___________________

25 GitHub is a major open source group; see https://github.com/.

Suggested Citation:"4 Human Sciences." National Academies of Sciences, Engineering, and Medicine. 2021. 2019-2020 Assessment of the Army Research Laboratory. Washington, DC: The National Academies Press. doi: 10.17226/26325.
×

the next generation of high-technology research tools with which to study human performance in combat simulations both with and without robotic teammates.

Recommendation: The human sciences core competency area should build on its strengths by acquiring expertise in those areas of cognitive science that focus on joint actions among individual intelligent entities (i.e., human-human and human-robot). Likewise, the Army Research Laboratory (ARL) should adopt a research strategy that embraces the findings of predictive processing, with particular attention to the dynamics of military operations.

Suggested Citation:"4 Human Sciences." National Academies of Sciences, Engineering, and Medicine. 2021. 2019-2020 Assessment of the Army Research Laboratory. Washington, DC: The National Academies Press. doi: 10.17226/26325.
×
Page 41
Suggested Citation:"4 Human Sciences." National Academies of Sciences, Engineering, and Medicine. 2021. 2019-2020 Assessment of the Army Research Laboratory. Washington, DC: The National Academies Press. doi: 10.17226/26325.
×
Page 42
Suggested Citation:"4 Human Sciences." National Academies of Sciences, Engineering, and Medicine. 2021. 2019-2020 Assessment of the Army Research Laboratory. Washington, DC: The National Academies Press. doi: 10.17226/26325.
×
Page 43
Suggested Citation:"4 Human Sciences." National Academies of Sciences, Engineering, and Medicine. 2021. 2019-2020 Assessment of the Army Research Laboratory. Washington, DC: The National Academies Press. doi: 10.17226/26325.
×
Page 44
Suggested Citation:"4 Human Sciences." National Academies of Sciences, Engineering, and Medicine. 2021. 2019-2020 Assessment of the Army Research Laboratory. Washington, DC: The National Academies Press. doi: 10.17226/26325.
×
Page 45
Suggested Citation:"4 Human Sciences." National Academies of Sciences, Engineering, and Medicine. 2021. 2019-2020 Assessment of the Army Research Laboratory. Washington, DC: The National Academies Press. doi: 10.17226/26325.
×
Page 46
Suggested Citation:"4 Human Sciences." National Academies of Sciences, Engineering, and Medicine. 2021. 2019-2020 Assessment of the Army Research Laboratory. Washington, DC: The National Academies Press. doi: 10.17226/26325.
×
Page 47
Suggested Citation:"4 Human Sciences." National Academies of Sciences, Engineering, and Medicine. 2021. 2019-2020 Assessment of the Army Research Laboratory. Washington, DC: The National Academies Press. doi: 10.17226/26325.
×
Page 48
Suggested Citation:"4 Human Sciences." National Academies of Sciences, Engineering, and Medicine. 2021. 2019-2020 Assessment of the Army Research Laboratory. Washington, DC: The National Academies Press. doi: 10.17226/26325.
×
Page 49
Suggested Citation:"4 Human Sciences." National Academies of Sciences, Engineering, and Medicine. 2021. 2019-2020 Assessment of the Army Research Laboratory. Washington, DC: The National Academies Press. doi: 10.17226/26325.
×
Page 50
Suggested Citation:"4 Human Sciences." National Academies of Sciences, Engineering, and Medicine. 2021. 2019-2020 Assessment of the Army Research Laboratory. Washington, DC: The National Academies Press. doi: 10.17226/26325.
×
Page 51
Suggested Citation:"4 Human Sciences." National Academies of Sciences, Engineering, and Medicine. 2021. 2019-2020 Assessment of the Army Research Laboratory. Washington, DC: The National Academies Press. doi: 10.17226/26325.
×
Page 52
Suggested Citation:"4 Human Sciences." National Academies of Sciences, Engineering, and Medicine. 2021. 2019-2020 Assessment of the Army Research Laboratory. Washington, DC: The National Academies Press. doi: 10.17226/26325.
×
Page 53
Suggested Citation:"4 Human Sciences." National Academies of Sciences, Engineering, and Medicine. 2021. 2019-2020 Assessment of the Army Research Laboratory. Washington, DC: The National Academies Press. doi: 10.17226/26325.
×
Page 54
Suggested Citation:"4 Human Sciences." National Academies of Sciences, Engineering, and Medicine. 2021. 2019-2020 Assessment of the Army Research Laboratory. Washington, DC: The National Academies Press. doi: 10.17226/26325.
×
Page 55
Suggested Citation:"4 Human Sciences." National Academies of Sciences, Engineering, and Medicine. 2021. 2019-2020 Assessment of the Army Research Laboratory. Washington, DC: The National Academies Press. doi: 10.17226/26325.
×
Page 56
Suggested Citation:"4 Human Sciences." National Academies of Sciences, Engineering, and Medicine. 2021. 2019-2020 Assessment of the Army Research Laboratory. Washington, DC: The National Academies Press. doi: 10.17226/26325.
×
Page 57
Suggested Citation:"4 Human Sciences." National Academies of Sciences, Engineering, and Medicine. 2021. 2019-2020 Assessment of the Army Research Laboratory. Washington, DC: The National Academies Press. doi: 10.17226/26325.
×
Page 58
Suggested Citation:"4 Human Sciences." National Academies of Sciences, Engineering, and Medicine. 2021. 2019-2020 Assessment of the Army Research Laboratory. Washington, DC: The National Academies Press. doi: 10.17226/26325.
×
Page 59
Suggested Citation:"4 Human Sciences." National Academies of Sciences, Engineering, and Medicine. 2021. 2019-2020 Assessment of the Army Research Laboratory. Washington, DC: The National Academies Press. doi: 10.17226/26325.
×
Page 60
Next: 5 Materials and Manufacturing Sciences »
2019-2020 Assessment of the Army Research Laboratory Get This Book
×
Buy Ebook | $34.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

The National Academies of Sciences, Engineering, and Medicine's Army Research Laboratory Technical Assessment Board (ARLTAB) provides biennial assessments of the scientific and technical quality of the Army Research Laboratory (ARL). These assessments include the development of findings and recommendations related to the quality of ARL's research, development, and analysis programs. 2019-2020 Assessment of the Army Research Laboratory reviews the following research core competencies of ARL: human sciences, network and information sciences, computational sciences, materials and manufacturing sciences, and propulsion sciences. This biennial report summarizes the findings of the ARLTAB from reviews conducted in 2019 and 2020.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!