7

Human Sciences

The Panel on Human Factors Sciences at the Army Research Laboratory (ARL) conducted its review of selected research and development (R&D) projects of the ARL Human Sciences Campaign and the Analysis and Assessment Campaign at the Aberdeen Proving Ground, Maryland, on July 14-16, 2015, and at the SFC Paul Ray Smith Simulation and Training Technology Center, Orlando, Florida, on June 14-16, 2016. The human sciences project areas reviewed were as follows:

• Humans in multiagent systems. These efforts aim to achieve critical technological breakthroughs needed for future Army multiagent, mixed-agent teams to effectively merge human and agent capabilities for collaborative decision-making and enhanced team performance in dynamic, complex environments. The challenges for human sciences R&D are soldier workload, situation awareness, trust, influence, and cultural cognition.
• Real-world behavior. The objectives of the R&D in this area are to enable the collection, analysis, and interpretation of human behavioral data within dynamic, complex, natural environments. ARL conducts R&D in the following two areas: (1) real-world complexity in human science experimentation and (2) assessing human behavior in the real world. A key focus of this work is the development of novel technology and methodologies and to collect and analyze these data in real-world conditions.
• Toward human variability. The goals of this R&D are to enable high-resolution, moment-to-moment predictions of an individual soldier’s internal and external behavior and performance and the ways in which soldiers interact dynamically in mixed-agent team and social settings in both training and operational environments. Human variability R&D is conducted in the following two areas: (1) multifaceted soldier characterization to develop a comprehensive understanding of the factors influencing human variability and (2) brain structure function coupling

• to create a multiscale understanding of the relationship among the brain’s physical structure, its dynamic neurophysiological functioning, and human behavior.
• Training. The goal of the training program is to discover and develop methods, models, tools, and technologies that will increase soldier readiness by improving training methods and training technologies.
• Integration technologies. The objectives of the integration technology R&D program are to discover and innovate principles and mechanisms for the integration of humans and systems.
• Augmentation. The goals of this R&D program are to develop and enable technological approaches for augmenting fundamental human capabilities that may enhance Army mission-related performance.

The panel also reviewed in 2015 a component of the Analysis and Assessment Campaign portfolio on assessing mission capabilities of systems. The goal of this area, more or less, is engineering and acquisition decision support for current and future Army human systems. Efforts are conducted in two areas, the first of which is human factors, which focuses on integrated human factors engineering (HFE) and system engineering (SE) assessments and analytic techniques to predict human, system, and mission capabilities early in the acquisition cycle. In this first area, HFE applications and tools are developed and refined to lower acquisition costs and improve design. The second area, soldier survivability, is concerned with analysis and engineering to increase the survivability of platforms and soldiers operating in combat environments.

Just prior to the 2015 assessment by the Army Research Laboratory Technical Assessment Board (ARLTAB), ARL reformulated its R&D programs to align with its science and technology (S&T) campaign plans for 2015-2035. This transformation was in its early stages during the 2015 review but appears to have crystalized for the 2016 assessment.

The increased emphasis on understanding and managing real-world complexity within the key campaign initiative (KCIs) is an important and notable development. Increasingly, complex sociotechnical systems that encompass interactions among multiple agents whose behavior is governed by intangible variables (e.g., emotion and culture) will pose many tough challenges for the Army after next. Research in this area will likely promote increased interdisciplinary collaboration with other campaign areas. Conceptual and theoretical breakthroughs from the growing adaptive complex systems engineering literature need to be leveraged to support this work.

ARL responsiveness to prior recommendations from the ARLTAB was fully evident in both of these reviews. For example, the preparatory materials provided in advance of the 2016 review meeting were exceptionally clear, concise, and focused; presentations were generally more uniform and at a higher standard; research hypotheses were clearly stated, and experimental variables were described and discussed in detail; and access by researchers to subject populations with face validity has clearly improved but still remains an issue.

Proactive research management approaches were evident, such as the Human Sciences 6.1 refresh, which has led to a more cohesive program and the Big Idea process for stimulating and supporting innovative leaps forward in research.

In general, gains were evident in publication rates and in the establishment of collaborations and partnerships with relevant peers inside and outside ARL. The ARL human sciences work environments are exceptional in terms of their unique and advanced technological capabilities to support research. ARL has continued to successfully attract clever postdoctoral researchers from a diverse set of universities and disciplines.

Kudos are due the ARL leadership for their success at overcoming government-wide constraints on conference participation by scientists and engineers. This year, presentations at technical and professional meetings increased, coupled with substantial gains in published papers in the peer-reviewed literature. The proactive efforts to enhance leadership in the broader scientific community through data and resource sharing and professional engagement (e.g., editorships and participation in conference organizing committees) is a vital step forward in building long-term credibility and influence. Taken together, these are outstanding accomplishments and mark a visible advance over prior years.

ARL’s vision of the future challenges to be faced by the Army is compelling and provides the foundational basis for investments in innovative R&D. Success in the execution of quality research that may deliver beneficial options to the Army after next faces significant challenges and opportunities. To overcome anticipated challenges and to leverage opportunities, ARL has developed and implemented an impressive array of organizational visions, initiatives, and strategic plans aimed at enabling environments and processes that leverage staff competencies and motivate innovation; encourage mentoring, coaching, and career management; and facilitate opportunities for collaboration with external peers (e.g., through the ARL open campus initiative).

While there was substantive evidence of good progress by ARL in all these areas, there were some areas that may require attention. Opportunities for technical and career mentoring do not appear uniform or consistent across the Aberdeen and Orlando sites. Orlando personnel need better connectivity with Aberdeen (e.g., Y: drive accessibility). The Big Idea initiative is a very good idea, but there was some evidence that bench-level staff in Orlando were not fully cognizant of the opportunities offered. While considerable progress has been made with increased publication in peer-reviewed literature, ARL staff expressed concerns that the public release clearance process is unduly bottlenecked, adding unnecessary delays and overhead in getting research to press. There appears to be a need for systematic feedback mechanisms from stakeholders—principally users/customers—as a basis for refining research and planning.

There appears to be a need for improved project-level planning that systematically considers Army requirements in the context of what others have done, are doing, and where ARL could best contribute. Good project planning that guides what problems researchers select to work on, how the work is executed, and the responsibilities assigned for transitioning the products of that work forward into the value stream is consistent with the management and execution of quality science and innovation. Available project plans would provide the ARLTAB assessment with valuable contextual data regarding research baselines, related advances, and expected unique contributions of the work under way and planned. A well-informed ARLTAB can better assess the quality of the science programs and address, for example, whether experimental designs and analyses are reasonable and appropriate to achieve desired technical objectives. Unfortunately, although project-level plans may exist, they were not provided to the ARLTAB.

A general challenge, noted in past assessments by the ARLTAB, persists with respect to the critical need for access to military-relevant subjects in the work. Many of the real-world research questions that ARL is dealing with urgently require more effort with respect to subject populations and the representation of mission contexts. A number of studies presented drew upon ARL researchers as subjects to a worrisome degree. As another example, researchers at the soldier performance and equipment advanced research (SPEAR) facility waited 8 months to acquire and run a limited number of soldiers as subjects. ARL needs to find a workable solution to this problem that threatens to compromise the credibility and impact of important research addressing vital Army needs.

On a positive note, statistical analysis at ARL has become increasingly rigorous; however, other analytical tools (e.g., mathematical modeling [optimization] and data mining) need further consideration in the program of work.

Several questions arose with respect to gaps in the disciplinary composition of the ARL human sciences workforce. Important core competencies, such as systems and simulation engineering, are absent; hiring is needed to grow these competency areas. There was also an expressed challenge and concern that the scant number of technicians that support the increasingly high-tech R&D within the Human Sciences Campaign is too low.

HUMANS IN MULTIAGENT SYSTEMS

Accomplishments

ARL human sciences work in multiagent systems addresses sociotechnical network operations, focusing on distributed collaboration and decision making in complex operational environments; sociocultural competency skills needed to support, inform, and influence operations in complex environments; human–robot trust; teaming of humans and intelligent systems; and human control of multiple robots. ARL appears primed to make important contributions to this paradigm and demonstrated a sound awareness of the key trends driving the research challenges in this area (i.e., the rise of the networked organization, increasing autonomy, and the need for cultural competency in military operations). The work in the area of human–robotic interactions and trust builds on the extant baseline of substantive theoretical and empirical work dealing with trust in automation. The conceptual organization of data-to-decisions at social cognitive, information, and communication layers allows for effective linkages and human–system integration with other multiagent systems activities. Hence ARL is applying a well-integrated systems approach to the study of purposeful social systems, addressing not only structure but also the role and function of sociotechnical Army systems involving people, information, and technology.

Challenges and Opportunities

The potentially valuable work in this area faces inherent challenges, but overall there does not appear to be a coherent vision for how the research holds together and cumulatively builds to push the state of the art. This may be the result of a forced merger of ongoing activities into the new campaign organization, which is still somewhat immature.

The problem area as scoped is very broad-ranging, including human–technology interactions to human–human interaction (sociocultural interactions), and it is not clear how all of the pieces fit together. For example, the distinction between the human–robot interaction (HRI) research and sociocultural research is an important one, because these groups have different pre-theoretical ideas and histories, concepts, analogies, and terminology and very different subject matter and methodologies. Although there are connections that could be made between these areas, they are generally distinct subdisciplines, and the connection between them needs to be made clear in the present organization.

The emerging work on joint operations of robots and humans is an important area of work with a large body of extant research that needs to be strategically considered as ARL evolves its niche in this area. There are many directions to be considered, including multiple HRI configurations (i.e., multiple humans with multiple robots), threats to security or integrity of robots as potentially compromised agents, and the possibility of coadaptation and coagency, where either the human or agent can take control. The

progress at ARL has been unidirectional (i.e., human to robots) and needs to progress to encompass bidirectional communication—that is, humans trusting robots and robots trusting humans.

Good use is made of modeling and simulation (M&S) in this area, although there is room for some improvement by expanding the toolkit of available techniques. For example, there are instances in which a statistical or event-driven simulation is used instead of physics-based M&S. M&S has emerged as a discipline in itself, as opposed to only an enabling technology, with several universities now offering degree programs. As such, ARL needs to consider recruiting more simulationists onto its research staff.

The emphasis on experimental work by the human factors group is commendable, but good mathematical and computational models of the robots, the interacting humans, and the feedback could save a lot of trial and error, provide insight, allow testing of different control regimes and feedback strategies, quantify sensitivity, help with identifying worst-case scenarios and sources of instability, and provide guidance to the design of experiments. There are modeling opportunities, especially of human operators, of robots, and of human collaborators from the viewpoint of the robot. The development of such models can aid understanding about how humans and robots interact by using inexpensive simulations and sensitivity studies. These studies need to be done before undertaking costly experimentation with humans and physical machines.

The research staff, including postdoctoral researchers and interns with recent experience, appears competent. However, from a staffing perspective, this is an area that would benefit from growing interdisciplinary collaborations as the portfolio matures. Similarly, the modeling and simulation capability to support this work is good, but an expanded toolkit will be needed to enable substantive advances in the future.

The challenges of HRI are inherently interdisciplinary, and drawing upon multiple contributing competencies is necessary to have successful, relevant outcomes. For example, the investigators in this area have, for the most part, backgrounds in psychology, and they could benefit from the knowledge, alternative frameworks, and approaches of experts in robot navigation, control, and modeling and simulation. Multidisciplinary approaches can interrogate issues from multiple perspectives, promote the integration of insights, and facilitate the connection of ideas in novel ways. Over the years, ARL has apparently supported a body of work on robotic control and navigation and already has many experts on staff, including mathematicians, electrical engineers, mechanical engineers, and physicists. There now appears to be a great opportunity for joint development of models and algorithms supporting this work. The description of research in multiagent systems appeared to focus on the human factors aspects of the systems and did not describe collaborative interactions with robotics researchers. Such collaborations are essential for a full understanding of the robotics features of such systems.

There is a wealth of available knowledge to be leveraged on how robots identify where they are, map their environments, plan their movement in autonomous or semiautonomous settings, use beacons and milestones, identify obstacles and hazards, fulfill commands, navigate, and follow trajectories. There is an opportunity to use this knowledge in the design of joint human–machine action. Missing out on this wealth of knowledge means that the algorithms developed by the Human Sciences Campaign may miss out on a variety of behaviors and constraints that are well studied and well modeled by individuals with mathematical, mechanical, electrical engineering, and computer science training.

Another area where ARL research on joint robot–human action can benefit is the significant body of work on hard fusion. Also of increasing significance is the emerging field of soft and hard fusion. Soft and hard fusion appears to be a major opportunity for developing a framework for HRI scene interpretation, because it admits a large suite of heterogeneous inputs, including heat sensors and tweets. There are numerous individuals across ARL with relevant expertise in this area, but there was scant evidence of any collaboration with these individuals.

ARL identified a focus on sociotechnical systems, including data to decisions, decision support systems, human dynamics of cybersecurity, and network team performance. Although ARL identified human dynamics of cybersecurity as a component of ARL’s focus on sociotechnical systems within its Human Sciences Campaign, ARL’s Human Sciences Campaign did not present anything that was germane to cyber research. Given that there exist many opportunities for adversaries to compromise robots or influence people, this would seem a critical area for human sciences research. This work may already be ongoing under a different ARL campaign, but it is important to support proactive collaboration and engagement of the human sciences in this area.

The research on understanding sociocultural influences suffered from some serious methodological design flaws and weaknesses in analysis that completely undercut its potential usefulness and impact. Chief among the problems was the study’s reliance on a subject population that was not representative of the target population. More specifically, subjects were uniformly Christian, white males who did not remotely reflect the religious, ethnic, racial, or gender demographics of the Army personnel targeted by the study, making it difficult and misleading to draw any credible conclusions from the findings of this study. The serious problematic design of this study suggests a need for preapproval quality vetting of proposed research designs at ARL.

REAL-WORLD BEHAVIOR

Accomplishments

The collection and analysis of human behavioral data within dynamic, complex, natural environments is an ambitious and challenging undertaking. Not surprisingly, the accomplishments in this area are only somewhat incremental given the immature state of the art and the groundbreaking challenges to develop needed enabling technology and methodology. Continued strategic investment to push advances in this area would yield significant payoffs for the Army and potential spin-offs of benefit to other government and private sector research and development.

Both the environment for auditory research (EAR) facility and the SPEAR obstacle course are outstanding, world-class facilities that have been brought in to this key campaign initiative. The research team using these facilities is well focused and has effective leadership. Ongoing hardware updates to the EAR facility will ensure that it is state of the art and easy to use for advanced studies of auditory perception. Since the ARLTAB’s 2013-2014 review, the research at the EAR facility has become more general and more relevant to the real world. Specifically, studies appear better designed, are less controlled than traditional psychoacoustic work, and address questions that can impact situations beyond the specific conditions tested. These advances could beneficially be pushed even further to raise the ante on research outcomes to an even higher level of importance and excellence. The collaboration with the neuroscience group to jointly measure behavior and neural signals, using electroencephalogram (EEG) technology, is laudable.

The research presented in this area appears focused on mission-relevant problems and contexts and draws on measures from multiple domains (e.g., biomechanics, cognition, and neurosciences), consistent with the goal of addressing real-world complexity. For example,

• The research using electrocortical activity to distinguish between uphill and level walking is apparently the first study to demonstrate that cortical activity changes while the human walks over different terrains. However, given the limited statistical power of the EEG signal to distinguish the terrain condition (level or incline), it would be necessary to consider more sophisticated

• analytics (e.g., source analyses for EEG studies) or other methods (e.g., signal enhancement methods that can reduce artifacts from electrode movement) to provide feed-forward control signals for exoskeletons. Regardless of outcome, this type of work is a good example of the foundational research needed to support the development of highly mobile sensing systems that could be useful in the field. Advanced measures for evaluating signal quality need to be developed, and the translational neuroscience (TN) group needs to ensure that it is aware of and understands the lessons learned from prior work in this area.
• The work on stretchable conductive elastomers for soldier biosensing was impressive and, if successful, could have applications well outside the military realm. This is also important foundational work, showing encouraging progress, to enable EEG measurement under real-world conditions where soldiers are moving, sweating, and otherwise burdened.
• The effort showing the effects of marching, rucksack load, and heart rate on shooting performance in the field is an applied study with considerable generality, because it is designed to understand the effect of work and fatigue on human performance. The study makes excellent use of the SPEAR facility and is very relevant to real-world Army combat, where soldiers may carry heavy rucksacks for extended periods of time.
• The research dealing with temporal and semantic coherence of sounds has implications for the presentation of multisensory data for training—for example, the possible effect of latency in presentation of initiating event and audio and visual stimuli generated in response to the event.
• The research on a novel measure of driver and vehicle interaction and research on eye movement correlates of behavioral performance in a simulated guard duty task showed interesting use of validating measures other than EEG. The novel driver–vehicle interaction metric and eye-tracking measurement used appeared less complex with respect to the collection and interpretation of data than the EEG measures that seem to dominate ARL’s real-world research paradigm.

Challenges and Opportunities

Measuring and understanding behavior in the real world is an audacious objective given the limits to the current understanding of nonlinear causal propagation and emergence in complex real-world systems and environments. True multidisciplinary projects are generally rare but are essential in order for this area to mature to yield meaningful and valuable outcomes.

Research in the real world requires giving up experimental control for the most part. It requires ecologically valid contexts and tasks and the means to capture and analyze the nonlinear causal interactions of multiple variables that may contribute to variance in performance and behavior. For example, it is not appropriate for real-world studies of auditory localization to assume fixed head and sounds unnaturally short that are turning on or off simultaneously. Expertise in ecological psychology and complex sociotechnical systems engineering is needed and could provide useful insights and research strategies under this initiative.

It may be possible to conduct some tightly controlled studies as a tool for assessing whether a new technique is feasible and then progress to a study in a more ecologically valid setting. Indeed, there may be a continuum of increasing complexity within which it may be possible to effectively derive valid causal assumptions. Therefore, based on findings, the driving–vehicle interaction work might be repeated for validation in more capable simulators before moving out to data collection in real-world driving in the field. Developing a methodology for moving research between the laboratory and real-world settings would be a major step forward and a significant contribution to establishing a methodological advance for real-world research.

The project on systems-based metrics of team states using communications data is somewhat ambitious, envisioning the collection of massive amounts of data on hundreds of individuals on a 24/7 basis on brigade-level mission command training exercises. Big data analytics will then be applied in an effort to explore the extent to which group efficacy is a function of variables such as shared understanding of command intent. The hope is that the data collection will be unobtrusive and that it will permit intervention before an existing or potential problem. The analysis of this massive volume of heterogeneous team communication data appears daunting and might benefit by collaboration with information technology or computer sciences subject-matter expertise resident elsewhere in ARL.

TOWARD HUMAN VARIABILITY

Accomplishments

Understanding and predicting human variability is an important and timely topic of investigation. Current systems are calibrated to the average performance of the average person in challenging circumstances; optimized adaptive systems might enable better use of human capacity when situations and states permit. Advances in this area reflect the availability of increasingly sophisticated behavioral and brain measures and the development of new analytic and statistical methods that may enable adaptive systems in operational settings.

The human variability research at ARL addresses the variability in performance between different individuals and the variability within one individual at different times and particularly in different states. Two components of this research initiative were presented: (1) cognitive neuroscience research and development with a focus on real-world measurement designed to illuminate the connection between brain states and behavior and their potential implications for brain–computer interfaces and (2) a proposed initiative to understand and predict variations in performance with measures of human behavior and physiological state over long periods, using relatively unobtrusive immersive measures interspersed with laboratory-based tests and measures.

Overall, the work in this area seems to be of exceptional quality. The majority of the research efforts used EEG (or in some cases fMRI) indexes of brain responses associated with visual or auditory processing or with motor responses. Several of the newer projects used either EEG sensor data or fMRI activity to identify patterns of coactive brain circuits that are characteristic during different stages of sensory, motor, or cognitive tasks. Novel work was presented aimed at advancing the state of the art of dynamic analysis of brain signals to identify active brain circuits and show how activity within these brain circuits differs within individuals and between individuals across the time course of responses in typical operator-relevant tasks. The research infrastructure in EEG, eye movements, and (through university collaboration) fMRI is superb and well suited to address the target issues.

This group has recruited an exceptionally strong set of researchers, including well-qualified postdoctoral and early-career scientists representing different technical backgrounds, with some gender and ethnic diversity. A number of the presentations featured early-career scientists. Overall, the presentations indicated good mentorship of these early-career researchers.

The quality, productivity, and scope of coverage in journal publications and presentations continue to be exceptional. The recent peer-reviewed publication record indicated between 5 and 10 journal articles per year since 2012, including 6 to date in 2015, with 10 additional articles submitted. The focuses of publications are well distributed over the scope of the initiative and are directed at cutting-edge issues in human variability. For example, a number are focused on EEG and/or survey methods (e.g., the Big Five inventory) and a number evaluate intra-individual gender measures. The majority of the verbal

and poster presentations for current and completed work were focused on brain states and behavior in a variety of operational situations.

An informal network of shared resources and shared knowledge is developing that transcends organizational boundaries. In particular, there appear to be considerable cohesiveness and teamwork that extend across ARL laboratories. The level of awareness and cooperation across the laboratories is impressive, the resources needed to advance the work are available, and the collective expertise can rival that in top academic institutions, where a more insular attitude is commonplace. Cohesiveness can be a powerful tool, and ARL deserves kudos for maintaining this excellent environment.

Challenges and Opportunities

Many of the projects presented appeared at first to be at formative stages at both empirical and theoretical levels, and the range of problems being worked on seemed somewhat vague, diverse, and unconnected. However, the implicit focus was clarified somewhat during small group discussions with the research staff, who described the focus as the identification of potential brain–behavior signatures for brain–computer interface translation.

The development of practical adaptive systems based on assessment of operator state is an important research objective and would benefit from a focused set of research priorities using a well-defined set of high-value operator tasks and operational contexts. The current effort appears devoted to testing and gaining expertise with various biometric measurements (e.g., field testing of EEG and multimodal sensors). However, the diverse array of biosensors and the extent of potential behavioral components require some winnowing of possibilities. In addition to high-fidelity brain measurements examined in the laboratory, a range of other measures such as multiple cue or cue fusion approaches needs to be considered, with the goal of identifying potentially lower fidelity neurocognitive measures of brain state. Fused cue approaches use multiple data modalities (e.g., heart rate measures, eye movement behaviors, simple EEG indicators, and other behavioral indexes) to achieve possible benefits of cue integration in the classification of human states (e.g., attention, vigilance state, fatigue). This could, in turn, be foundational to the notion of precision performance, whereby inputs, decisions, and scheduling demands can be tailored to the individual, including the mental state of the individual.

One caution is that the ARL studies on individual differences focus almost entirely on developing neurophysiological measures, with little or no incorporation of more traditional behavioral and psychological measurements. As a counterexample, ARL’s reported research has shown that a relationship exists between the spatial abilities of operators and their performance on robotic tasks. This finding illustrates how much information on individual differences might be missed if the focus is entirely on biological sensors. Advice on the selection of appropriate psychological measurements depends on the parameters of a particular targeted behavior (e.g., precision), but their incorporation could be given a careful and more complete consideration in the overall research strategy.

The planned new initiative in unobtrusive immersive measurement of behavior is potentially an enormous problem domain. The findings and best practices from other related major data projects need to be leveraged to identify the research niche with the highest potential return on investment for ARL. As an example, there is a program of the U.S. Department of Transportation designed in part to characterize state precursors of traffic events. Other potential opportunities for leverage include advances in detecting and correlating attention or fatigue states with human performance in smart-home projects or monitoring system efforts associated with medical issues or adaptive support for aging populations.

One of the core challenges faced by this work—as in other initiatives addressed earlier—is the availability of suitably representative human subject populations for research testing. These might include

civilian populations where those are appropriate or selected access to military personnel, where data need to be representative of the populations to which the research is expected to be applied. Suitably representative populations are especially important for credible research on interindividual variation.

Another core challenge is the need to engage appropriate information systems and computational resources where needed. The scope of data that could be collected in the immersive measurement for understanding and predicting individual variation in performance has the potential to be massive. The measurements planned in the immersive workplace (possibly including recording facial expression, direction of gaze, all keystrokes in work projects, and some simple physiological measures of heart rate) suggested the generation of large amounts of data. While preliminary and pilot testing of such data in smaller units may be feasible without special arrangements, the large data demands of ongoing immersive projects of this kind are likely to require specialized data management plans and adapting existing algorithms or developing new algorithms for data mining and data analysis. This would necessitate engagement of more expertise in information systems and computational resources. While the information systems and computational resources were not part of the research program under review, it was indicated that arrangements were under way for broader collaboration with other ARL researchers and staff with expertise and resources in this area.

The cognitive neuroscience group has recruited a number of early-career researchers and collaborators from academia. Although there was evidence for strong mentoring of these individuals, the expansion of the group, combined with the fact that key individuals are moving up into administration, suggests a need to develop expanded procedures for mentorship and for the identification of a set of mentors.

For the planned research into characterizing and predicting human variability, a series of workshops is being planned to discuss relevant research topics to enable program development. This is laudable, and a beneficial expansion of this effort (perhaps under ARL’s open campus initiative) could include short-term visitors from academia or industry to broaden intellectual engagement around planned initiatives such as that for large-scale immersive measurement and individual variability.

Research on individual variability is one of the most difficult problems in the field of psychology, and there is much motivation to understand the phenomenon. High-quality and high-impact research in this area, focused on measurement, estimation, and prediction, could position the group for a broadly recognized leadership role. Unique high-impact studies of individual differences and intra-individual variability as it relates to stress, fatigue, and other psychophysiological states in targeted operational environments could also provide significant benefits, beyond those envisioned by ARL, to external university or industry research efforts that generally do not have direct access to these operational contexts.

TRAINING

Accomplishments and Advancements

Effective execution of the ARL Training program is a challenging endeavor. The technology baseline on which it depends cuts across multiple S&T areas and, in turn, requires a diverse, multidisciplinary workforce. Likewise, the relevant basic cognitive and social science research that applies to many training problems is fragmented and somewhat characterized by competing micro-theories. Additionally, the advanced training and simulation technologies are increasingly linked to commercial products and requirements with rapidly evolving capabilities.

The Training program is addressing important scientific and technical challenges that are relevant to both the Army and the larger education and training community. The presented projects are employing appropriate methods and using relevant environments and subject populations.

The program seeks to improve training efficiency and effectiveness through a combination of training technologies and methods. The projects presented addressed a number of challenges, such as how to create affordable intelligent tutoring systems, how to provide simulation-based training for dismounted soldiers, and how to increase the fidelity of medical mannequins. Such work is valuable for domain-specific applications and offers significant spin-off potential to education and training in general.

Publications are generally increasing, and there is a continuing effort to expand beyond technical reports. While additional emphasis is needed to encourage publication in top-tier journals, as opposed to conference proceedings, documentation of the group’s productivity is consistent with that of similar organizations.

In many cases, affordable training technologies can only be achieved through the use of commercially available technologies or products. ARL staff are aware of such dependencies and are directly addressing them in projects involving virtual worlds, wearable displays, and lasers. Projects such as Head Mounted Displays for Augmented Reality Training and Lithographic Vertical-Cavity Surface-Emitting Lasers (VCSELs) are actively feeding technology enhancements, developed to meet Army-specific needs, back into the commercial arena.

Improving training methods and technologies requires multidisciplinary teams that include behavioral scientists, engineers, computer scientists, and training designers. In general, the research teams involved in the programs presented to the panel showed an appropriate mix of the needed disciplines.

There appears to be a proactive effort to recruit and develop a competent workforce. Early career engineers and scientists were well represented in the presented programs. The Training program seems to involve an active effort to recruit postdoctoral researchers from the local community and is also employing interns to grow the future workforce.

Training Area Organization

ARL Training R&D is organized around two broad research thrusts, each of which is divided into two subareas. These research thrusts and subareas are shown in Table 7.1. Table 7.1 also shows the seven research projects presented during the 2016 review and their location within the ARL training taxonomy.

The four subareas and their associated research projects highlight the importance of both methods and technology for training. The Adaptive Training and Education subarea seeks to improve the effective-

TABLE 7.1 Seven 2016 Training Research Projects, as Aligned within the Army Research Laboratory’s Human Systems Campaign’s Training Taxonomy

SOURCE: Supplement to Army Research Laboratory S&T Campaign Plans 2015-2035.

ness and efficiency of training by developing advanced intelligent tutoring and computer-based training. Key technologies being addressed include automated authoring, learner modeling, performance assessment, and instructional management. The Training Effectiveness subarea focuses on how components and characteristics of training systems—for example, media, devices, fidelity, instructors, and method of instruction—combine to determine a training outcome and how to measure those outcomes. The goal is to develop techniques to measure training outcomes and to create models to predict those outcomes for both individual and team training. The Enabling Technologies subarea focuses on simulation and training technologies that are foundational components of a variety of training systems. These technologies include virtual, mixed, and augmented reality; computer-generated or virtual humans; and human–computer interaction with the training environment. The Technology Applications subarea focuses on applying simulation and training technologies to meet domain-specific training needs.

Generalized Instructional Framework for Tutoring

The Generalized Instructional Framework for Tutoring (GIFT) project is developing the components necessary to assess a learner’s state, recommend an instructional strategy, and modify the learning environment to support adaptive intelligent tutoring using a modular, open-source architecture. GIFT represents the opportunity to significantly advance adaptive intelligent tutoring and illustrates the need for organizations to make long-term investments in key research areas. The GIFT presentation to the panel provided a clear overview of its architecture and technical status. It did not, however, provide insights into how user experience and cost savings are being captured and fed back into the development effort. ARL reported that discussions on a transition and sustainment strategy are ongoing with the Army’s Training and Doctrine Command (TRADOC) and the Advanced Distributed Learning (ADL) collaboration.

The GIFT project has demonstrated significant technical achievements, and ARL staff have documented those accomplishments for the training community. It has also established a broad and diverse set of international users. The expansion into psychomotor skills training represents a significant advancement of intelligent tutoring research. The annual expert meeting is an exemplar of excellent program management. In addition, the GIFT project has taken the initial steps toward technology transition by initiating discussions with TRADOC and the ADL collaboration.

Developing Models of Expert Performance for Support in an Adaptive Marksmanship Trainer

The adaptive marksmanship trainer project is attempting to develop models of expert performance and use those models to tutor individual marksmanship skills. The project has both customer and subject-matter expert support. Successful expansion of adaptive intelligent tutoring to include psychomotor skills training offers significant payoffs across a wide variety of areas, including medicine to sports.

Virtual World Research—Soldier Training Effectiveness

In an effort to reduce live training demands for dismounted soldiers, ARL is investigating the use of virtual world technology that combines aspects of both virtual and constructive simulations. Virtual worlds are readily available game-like environments (e.g., Minecraft, Second Life, World of Warcraft, and VBS3) that run on commodity computer platforms. A unique avatar that directly interacts with the simulated environment represents each trainee. Each avatar carries out appropriate actions based on the inputs provided by a trainee using various human–computer interaction technologies (e.g., mouse, joy stick, voice). If the training research community can correctly identify the cognitive and team train-

ing requirements, it may be possible to effectively train those cognitive and team skills using a virtual world and integrate that training with psychomotor skills during live training. The project began in April 2016 and has made substantial progress. It is addressing a training and technology area that has little training effectiveness data to guide potential implementations. However, it requires cognitive science and instructional design support to help identify the cognitive components of dismounted soldier training and design appropriate instruction. Because virtual worlds and serious games continue to attract a great deal of research interest, this project needs to actively explore collaborations and capitalize on related work in industry and government, such as VBS-Pointman, as well as the avatar research and development efforts at the Institute for Creative Technology (ICT).

Head-Mounted Displays for Augmented Reality Training

Augmented reality integrates digital information into the user’s environment in real time. It offers the potential to enhance live training by blending virtual objects into the live training environment. A key component technology is a wearable display to present these virtual objects to the soldier. However, training applications involving augmented reality are hindered by displays that have narrow field of view and poor resolution and that are overwhelmed by ambient light. This project is specifically addressing those limitations for live training. It has a well-defined technical approach and path and is a Small Business Innovative Research success story.

Simulated Human Tissue Performance

This project seeks to develop simulated tissues that more closely approximate the mechanical and material properties of actual human tissue. Current work is quantifying the characteristics of human tissue and identifying the differences between those human tissues and the simulated tissue used in a variety of medical mannequins. The assumption is that use of more realistic tissues will improve the effectiveness of simulation-based medical training. While this assumption may be correct, no training-need analyses indicating that the fidelity of tissue simulation leads to negative training or limits the capability to adequately train specific procedures were presented to the panel. The project has established strong collaborations with several medical schools, and it might be worth exploring opportunities for collaborative research on tissue properties and manufacturing with ARL’s Weapons and Materials Research Directorate.

Lithographic Oxide-free Vertical-Cavity Surface-Emitting Lasers

The Instrumented-Multiple Integrated Laser Engagement System (I-MILES) enables realistic engagement training in a sophisticated laser tag environment. This project is attempting to develop new lasers that will reduce cost, increase both accuracy and reliability, and improve eye safety. It has a clearly defined technology readiness level as a goal and a path forward that includes planned collaborations with the National Institute of Standards and Technology, the Defense Advanced Research Projects Agency (DARPA), and industry.

Opportunities and Challenges

This was the panel’s second visit to review ARL’s training and simulation R&D in Orlando, Florida. In 2012, the Simulation and Training Technology Center (STTC) had merged into the ARL Human

Research and Engineering Directorate. At the first onsite review of this area in June 2013, there was broad consensus by the panel that the human sciences were not having sufficient influence in areas of R&D where they could be drivers. With the notable exception of the intelligent tutoring area (GIFT), the extent to which this may still be the case was somewhat difficult to determine during the current review, because the programs sampled were generally focused on training technology rather than training science. For example, the adaptive marksmanship project was concerned with automated performance measurement. No real attempt was made to tie the work to a behaviorally based view of psychomotor skills training. The virtual worlds project is currently focused on technology and interfaces. Behavioral support to help identify cognitive components of dismounted soldier skills is supposed to be provided to the program in fall 2016; the synthetic tissues project is a technology project without any upfront training analysis. The assumption is that higher fidelity is better and that current data shows that synthetic tissues do not match the properties of human tissues. Augmented reality is a technology program, but it is addressing human performance limitations involving wearable, see-through displays. The behavioral components (e.g., psychophysical and instructional design) remain unclear. The laser project is a technology project focused in improving cost and performance for I-MILES. The absence of an overall roadmap and an explicitly defined sampling strategy, combined with presentations that were not particularly well crafted for the purposes of the review (lacking clear description of project goals, methods, data analyses, theoretical underpinnings, and connection with relevant extramural research), made it impossible to fully understand the behavioral, cognitive, or instructional components of the overall training program.

The GIFT project has the opportunity to become a de facto or de jure standard for adaptive, intelligent tutors. The continuing growth in users and the integration of psychomotor skills training represent expanding opportunities. However, to achieve its full potential, the GIFT project could continue to evolve broadly applicable models based on empirical research and formally document strengths and weaknesses as identified by its community. The GIFT project needs to define what constitutes scientific and technological success and establish a clear transition and sustainability path that includes maintenance of standardized interfaces and models independent of the core ARL S&T program. ARL also needs to develop a plan to maintain a research version of GIFT that will enable the expansion of intelligent tutoring into new domains and needs to support other training research efforts.

Integrating virtual simulation into live training for dismounted soldiers involves significant technical challenges. Virtual simulation (i.e., real people operating simulated systems) is a well-established method of training individuals and teams across a variety of skills, including psychomotor, decisional, and communication skills. In virtual simulation the individual’s interaction with the simulated environment is typically accomplished through the use of simulated equipment or systems (for example, gunnery and aircraft simulators). When an individual interacts directly with the environment without mediating equipment or systems, it becomes very difficult to create an effective human-in-the-loop simulation, because virtual simulation technologies do not allow users to directly grasp virtual objects or move unconstrained within large virtual spaces. Development of both augmented reality and virtual world technologies, combined with the development and evaluation of training strategies, offers potentially effective and affordable simulation media for dismounted soldiers.

The virtual human technology project conducted at the ICT at the University of Southern California has developed several avatars with realistic nonverbal cues to support interactive training across a variety of areas (e.g., junior leadership, interpersonal skills, and sexual harassment and assault prevention) and has developed a Virtual Human Toolkit. The Orlando training research portfolio has projects that include the use of human avatars, but no specific links were identified between the ICT technology and ongoing or proposed research projects in Orlando.

Department of Defense laboratories such as ARL have a unique opportunity to foster the expansion of fundamental knowledge in the sciences of learning and training. The application of fundamental knowledge to real-world training problems provides a means of identifying the strengths and weaknesses in our understanding of how adults learn, retain, and transfer knowledge and skills to address complex problems. In order to advance the sciences of learning and training, it is critical that both the successes and failures in applying that knowledge be communicated across the tech-base community. This requires continuing interchanges between the service laboratories themselves and the academic research community through attendance at scientific conferences and increased publication in referred journals in disciplines such as computer science, education, human factors, and psychology. It appears ARL could benefit from improved collaboration with universities for enhancement of their training programs. The implementation of on-line educational programs throughout the country has changed the learning paradigm for many different groups of students and employees.

The scope and complexity of training S&T is massive. As a result, it is difficult to present a unified picture of the area and of ARL’s ongoing S&T programs. Although the read-ahead material for the panel provided a great deal of useful information, it did not provide a unified roadmap of the portfolio, show the relationship between various research projects, or identify criteria for success and potential exit ramps. In addition, individual projects did not consistently identify key scientific and technical milestones or estimated project duration.

INTEGRATION TECHNOLOGIES

Accomplishments and Advancements

The integration technologies technical program within the Human Sciences Campaign focuses on supporting improved and effective bidirectional integration and mutual adaptation of soldier-technology systems for robust and optimal functioning in diverse operational environments and conditions. The goal of this campaign is to integrate operational systems and technological devices to optimize outcomes. The objective for this initiative is to close the loop between multiple systems and between systems and humans to enable the creation of hybrid human–system teams suited to demanding and unpredictable environments. The topics for the present assessment focused on R&D on closed-loop behavior that included efforts in cybernetics and brain–computer integration.

Optimized performance of hybrid teams of humans, systems, and technology requires a characterization of human capabilities, system functions, and human–system interactions, together with a deep understanding of the environments within which they could operate. The Human Sciences Campaign at ARL is actively pursuing research to more fully understand human capabilities and ways that systems can capitalize on or augment them. The long-term goal includes the development of systems with environmental awareness and the ability to analyze and understand the state of the human operator to allow the development of adaptive systems for individuals. An intermediate goal is to understand the range of variability and state-dependence of human performance and to use this to design systems that could adapt to the expected variations in performance.

This initiative includes research activities focused on measuring brain states in different tasks and environments, coupled with computational approaches for analyzing this information and integrating it with other machine-based computations. This may include specifying how multimodal interfaces may assist in achieving these aims. The integration program research team identified two topic areas for integration technologies and the emphasis on closed-loop behavior—cybernetics and brain–computer integration.

The topic of cybernetics brings a newly identified structure to the processing of closed-loop systems in which states and outcomes of the human system feedback to alter the behavior of technological systems. Cybernetics is a classic area of systems and control theory that focuses on communication and systems control; there is a broad and long-standing literature in systems engineering that may prove useful in understanding how hybrid human–technology teams operate. The topic of brain–computer integration relies more directly on the primary research on cognitive, affective, and physical brain states and their role in the development and coding of multimodal interfaces.

ARL’s integration program has achieved several unique and high-quality technical accomplishments. The team has defined a program and approach focused on characterizing and understanding human state (cognitive, affective, and physical) from neural and behavioral measurements. This effort can enable improved human–machine integration and interaction. They have organized and participate in a high-quality technical research community, including able junior research scientists, with strong publication records. They have been using state-of-the-art electroencephalography (EEG) methods to analyze brain state; augmenting the analyses by models of neural substrates and augmented other modalities. Staff successfully participated in the 6.1 refresh effort that has led to the creation and execution of a cohesive research program.

The integration technologies program has employed quality research methods in physiological and behavioral signal measurement and analyses, where they are using advanced machine learning techniques. The use of EEG for human state estimation was noteworthy. This broad and unique effort to assess and identify variations in underlying state includes basic research on models of neural substrate and the identification of pathways to real-world applications.

One project is a compelling exemplar of these activities. This is the project on the design, development, and demonstration of a multimodal human–system image analysis system. This demonstration project combined computer-image analysis as one expert system with EEG responses of humans searching for targets in rapid sequence to achieve optimized target recognition from high-throughput image sequences. Using deep learning neural network models (hierarchical convolution networks) of human brain responses and computer-image analysis, the goal is to combine both sources to achieve maximized target detection, a task with clear military-relevant applications. This project combines basic research on single-trial EEG, network learning models, and information fusion. The natural combination of implicit measures of human perceptual and cognitive processes alongside automated machine processing of images was an example of innovative progress in this area.

The scientists and engineers contributing to the integration technologies program include a strong neuroscience group with special expertise in EEG measures of brain activity coupled with access to other relevant measures such as fMRI at affiliated universities. The research team includes new and junior scientists with expertise in related statistical and neural network modeling with significant potential to model brain behavior in these measures. This includes important strengths in statistical modeling of EEG with generalized linear models and new developments in analysis of single-trials (relevant to current brain state) using deep learning neural network techniques.

The research team is highly qualified and focused. The research stakes are substantial, and the gains made by this group show a strong positive pace of change. The team has demonstrated research productivity with a strong publication record.

Opportunities and Challenges

The integration technologies program has the potential to significantly advance basic research and provide key information to inform the integration of the human into hybrid human–technology teams.

The program overview projecting the cybernetics approach was only recently initiated and has the potential to further inform this initiative, especially in relation to the technological systems components of integration technologies.

The integration technologies team seems to have assembled an active and expert group in EEG, statistical and neural network modeling, and for modeling of neural substrates of behavior. The opportunity exists for development of sub-goals that will move the research agenda in the direction of closed-loop systems for individuals, or of intermediate systems that could assess the likely operational capabilities of humans in certain environments, and adapt the systems to the range of performance in those environments. Another opportunity might be to integrate the more classical and modern developments in cybernetics and control theory to augment models of humans and systems. Presently, much of the modeling work on identifying brain states and markers in EEG is utilizing strong statistical methods, general approaches to regularization and cross-validation, and relatively new methods in deep learning.

In general, there were some disconnects between the overall presented program goals and the projects presented. For example, the introductory slides discuss integration of new technologies for soldiers, but new technology was only represented in a single project.

A number of opportunities for further development and associated challenges were identified. EEG as a measure offers a number of advantages (such as ambulatory measurements, a domain of particular expertise within the team), but EEG-based measurement may not be the best tool for all of the projects on which it is proposed to be used. The creation of generalized EEG models that work with the majority of individuals may not be possible. This provides an opportunity to develop novel techniques and algorithms to either generate several different models that work in different brain states or to develop truly individually adaptive measures and systems. The opportunity for collection of other state measures needs to be considered; those may be field-ready and are critical to understanding EEG-based signals. These constructs include, for example, pain, depression, multiple constructs of stress, and cognitive load. EEG data collection is important for performing state inference, but a multimodal view needs to be adopted on the signals that are gathered for this purpose. It is critical to know what other signals (e.g., psycho-physiology, movement, and temperature) need to be involved to augment or provide proxy measures as the work moves forward. The program would benefit from considering incorporating transcranial magnetic stimulation in the investigation of the central nervous system adaptation to random noise and vibration.

There was evidence in the material presented for some closed-loop behavior work (i.e., the tantalizing use case possibilities to address problems related to post-traumatic stress disorder and suicide); there is significant work that still needs to be accomplished to address these challenges. ARL noted that the cybernetics project within the Human Sciences Campaign is in its early stages. The project would benefit from a strong theoretical framework and specific use cases that will take the proposed project through the entire research cycle. There is significant literature in systems science, including in estimation and control (and associated fundamental notions of stability, observability, and controllability), which are topics central to the proposed studies in closed-loop behavior.

While there are interactions occurring between the researchers and soldiers (the primary users of any systems developed), the research needs to consider additional contacts to inform and improve the ecological validity of the projects. A systematic mechanism for feeding back information from customer experience into refining scientific studies would offer many benefits.

The cybernetics research program would also benefit by developing productive partnerships with system science experts, including the mathematical modeling of coupled human–machine systems, and the definition of compelling use cases to test and validate scientific methods (e.g., use cases could be drawn from the augmentation program).

AUGMENTATION

Accomplishments and Advancements

The augmentation program team is developing and enabling technological approaches to enhance soldier performance across wide-ranging Army-relevant scenarios. Augmenting the human with technology in order to improve performance has been in existence for centuries (e.g., eyeglasses). Warfare-specific augmentation, likewise, has been available for years (e.g., rifle scope). However, the explosion of technology in recent years has afforded the opportunity to provide the warfighter with viable augmentation options.

The benefits to be derived from developments in augmentation hinge on the balance between augmentation gains and augmentation burdens. The scientists and engineers working in this area demonstrated keen awareness of the trade-offs between the burden of additional gear, machinery, weight, and maintenance against the expected benefits from augmentation devices and algorithms. For example, soldiers are already burdened by enhancements in body armor, weapons, radios, and batteries that collectively weigh in excess of 75 pounds, and there are cases in excess of 100 pounds; the additional gains that any proposed enhancement could provide needs to be anticipated and quantified to ensure net benefits.

There is appropriate focus on issues such as long-term impact on soldiers who use augmentation devices and methods and the need to monitor withdrawal effects when augmentation is discontinued after prolonged use. A potentially important issue not fully considered deals with how the ethics involved with the implementation of augmentation are presented, monitored, and managed at ARL, beyond the standard internal review board process. Of the technologies presented, it would seem that there are significant ethical issues that need to be addressed with electrode-based stimulation of the brain and deep brain stimulation.

For the most part, ARL has focused on the physiological and physical ergonomic implications of augmentation, as opposed to cognitive and behavioral human factors. Wearing a suit may not only affect metabolism but also cognitive constructs such as situation awareness. The exclusion of cognitive constructs at this early stage of the work is understandable, but to be relevant and credible, ARL will need to address cognitive and other human–systems integration factors early in the development process.

Efforts to generate standardized methodologies and metrics are important. Conceptual graphs were presented that depicted the transition from a state of degraded capabilities to that of enhanced capabilities. As the research in this area progresses, there will be a need to develop and use a rigorous approach that quantifies this transition. A set of objective metrics will be needed to assess the effectiveness of augmentation approaches, devices, and algorithms. These metrics would, in all likelihood, be specific to each set of techniques. There are two compelling exemplars of this in the area of exoskeletons. Mooney et al.¹ used metabolic cost during human walking as a metric to assess an exoskeleton; Gams et al.² measured the effect of a knee exoskeleton on the metabolic cost of periodic squatting. The ARL augmentation team will likely need to develop and work with these kinds of frameworks.

The effort to develop metrics for the effects of augmentation needs to be complemented by development of metrics for the overhead imposed by augmentation. Often this overhead is directly and easily

___________________

¹ L.M. Mooney, E.J. Rouse, and M.H. Hugh, Autonomous exoskeleton reduces metabolic cost of human walking during load carriage, Journal of NeuroEngineering and Rehabilitation 11:80, 2014, https://jneuroengrehab.biomedcentral.com/articles/10.1186/1743-0003-11-80.

² A. Gams, T. Debevec, T. Petric, and J. Babic, Metabolic cost of squatting using robotic knee exoskeleton, pp. 184-190 in RAAD 2012: 21st International Workshop on Robotics in Alpe-Adria-Danube Region: Proceedings, Edizoini Scientifiche e Artistche, Torre del Greco, Italy.

quantified (e.g., the weight of the augmentation device). The trade-off between augmentation benefit and augmentation overhead needs to be expressed in terms of these metrics, and where applicable, using ratios of physical quantities. For example, if one studies a device that increases the blood oxygen saturation level of its user, one would use a ratio such as (SpO₂)/(device weight).³

Distribution of data sets is a valuable research contribution. Shared databases that allow researchers from around the world to test their algorithms under common conditions have been instrumental in moving the state of the art in many fields, such as speech recognition (e.g., the TIMIT database); visual object recognition (e.g., PASCAL and MNIST databases); and EEG analysis (e.g., PhysioBank, PysioToolKit and PhysioNet databases). Similarly, the database developed by ARL can, in principle, be as significant as these databases in providing a unique set of data. In addition to aiding other research efforts, this database could optimally be collected by the ARL team and provided to various branches of DOD to test the efficacy of their practices and techniques. This could contribute to establishing ARL as a technology leader in the field of augmentation.

The distribution of these data sets would be an important professional service as a reasonable short-term trade for the augmentation team’s relatively low productivity in journal publications. As the augmentation team and its research mature, they will need to increase journal publication while maintaining the distribution of these data sets. Establishing a recognized global presence generally requires substantial contributions to the open literature.

The development of physical augmentation is a major noteworthy accomplishment. The ARL augmentation team is well positioned to become a leading global force in the research and development of augmentation for healthy individuals. Research has led to important advances for the disabled, but research to help able-bodied professionals do their jobs is not effectively leveraging the advances in wearable robotics. While the military benefits of technologies such as those intended for reducing fatigue and stabilizing the aim of combat weapons are unquestionable, so are the significant high-value options these technologies offer other domains (e.g., robotic-assisted microsurgery).

Opportunities and Challenges

ARL’s R&D in augmentation is highly relevant to Army needs and has potential to elevate ARL to a leadership role in specific areas of this work. The augmentation team has published several very useful studies that will likely be influential in the literature on human enhancement. Of particular interest and possible impact is the work on controllability of brain networks. This work has been performed using highly simplified, noise-free, linear, discrete-time, and time-invariant network models. This work could be beneficially expanded to show that the simplified models are valid linearizations of the corresponding models that are more complete (which are nonlinear and only short-term stationary) and could push further with work on more realistic nonlinear mathematical models to explore aspects of nonlinear controllability.⁴

In general, the augmentation research efforts could be advanced by consideration of the following challenges and opportunities discussed below.

Increased interdisciplinary collaboration across military, academia, and industry would better inform the exoskeleton work. The augmentation program is richly connected to other human sciences expertise (e.g., biomechanics and cognitive science), but the extent and the level at which such interaction occurs across other relevant ARL communities was simply not evident. For example, collaboration with the

___________________

³ SpO₂ stands for peripheral capillary oxygen saturation, an estimate of the amount of oxygen in the blood.

⁴ H. Nijmeijer and A. van der Schaft, Nonlinear Dynamical Control Systems, Springer, New York, 1990.

system theoretic communities in robotics, control, and signal processing at ARL and other external research groups could productively enhance the work in the augmentation program

Cognitive augmentation (e.g., neurostimulation) is underdeveloped in comparison to the physical augmentation work being done by ARL and in contrast to cognitive augmentation research reported in the literature. The military (e.g., DARPA) and elements of the commercial sector (e.g., automotive) have been actively supporting augmented cognition R&D for many years, producing much that can be leveraged. Physical and cognitive augmentation will ultimately have to work together for the warfighter of the future. Embracing this notion during the early stages of development would yield more seamless and effective human-technical systems.

A plan is needed to map and facilitate the growth of research into the perceptual, cognitive, and physical components of augmentation. The augmentation research team needs to determine its niche and research focus with respect to perception, cognition, and physical performance and the level at which contributions can be made. The team needs to identify a specific research niche that goes beyond evaluation. Evaluation alone does not constitute science or research. This would be the foundation of a research plan that establishes priorities, resource allocations, and the roles of individual researchers.

Modeling, using both predictive models and simulation tools, needs to be more effectively and iteratively integrated into the scientific program. Augmentation research team members expressed skepticism about the value of extant modeling and simulation (M&S) tools and their limited relevance to real-world sources of variance. Nonetheless, there would be substantive benefits for more extensive use of M&S tools and related software. These may include the JACK software (for human ergonomic modeling and simulation); the RAMSIS software (a 3D computer-aided design tool for the ergonomic development of vehicles and cockpits); LifeMOD; and Make Human (for the prototyping of photo-realistic humanoids). In the area of cognitive modeling, tools can be helpful that are related to ACT-R (e.g., Java ACT-R) and SOAR. Other relevant tools are CHREST, CLARION, and OpenCog. The panel does not specifically advocate any of these specific tools but identifies them as examples of M&S tools that could be considered to assist the development work.

Working without M&S tools could be costly and limiting, because the alternative—extensive experimentation with humans—is logistically complex and expensive. Also, researcher intuition and field tests alone appear insufficient for the efforts needed by the augmentation research team. Among the potential advantages of adopting and using such models and M&S software are improved realization of goals during design and implementation; avoidance of rework costs by uncovering human-performance and feasibility issues earlier; ability to capture visually (and store) best practices for future programs; opportunities for sensitivity analysis and parameter tuning; and comparison of approaches and estimates of performance before experimentation with humans commences.

It is generally good practice to develop systems by first articulating a theoretical framework that predicts certain behaviors, followed by simulation of the framework to substantiate theoretical intuition, and then followed by validation of the simulation and theory with physical implementation and measurements of the system. Often, there are multiple feedback loops among these three parts of the design cycle that lead to the refinement of the theory, generation of more detailed predictions, and the achievement of more accurate validation. This design flow is particularly relevant to the augmentation program because of the added complexity of having a human in the loop of the systems and interaction with the real world. To enable a better definition of ARL’s targeted contributions to the field, accurate and precise modeling of the augmentation system is needed, coupled with a more theoretical, top-down view of the empirical work and clearer leverage of and reference to the extant literature.

ARL researchers need to extend the trade-off between gains and burdens by defining the procedures for using performance indices to illuminate the tipping point for various human augmentation systems.

ARL needs to consider developing numerical metrics for the gains achieved by augmentation as well as for the overhead that augmentation methods impose. ARL also needs to measure ratios between certain gains and the main overhead variables—for example, for the heart rate reducing gear device (the ratio between change in heart rate during certain physical activities as a result of this augmentation versus extra weight required by the heart-rate reducing gear). While such metrics would be specific to a certain kind of device or method, they may be usable in time to define a space of all key variables for a family of devices. This space can then be divided into regions that are labeled from highly efficient (where the ratios between improvement impacts and overhead is high) and regions that are inefficient (where the same ratios are low). Devising such partitioned performance spaces would allow a judicious decision on a tipping point (where a certain technology starts justifying its benefit by overcoming overhead). This approach would allow for systematic comparison of augmentation methods by measuring the distance between performance of each method and the boundary between the efficient and inefficient subspaces.

ASSESSING MISSION CAPABILITY OF SYSTEMS

Accomplishments

The work in this area is comprised of human-centered engineering and decision support methods, models, and tools supported under ARL’s Analysis and Assessment Campaign. Most of the projects that were presented in this area are responding to important and specific Army customer needs, such as the efforts dealing with human modeling, field assistance in S&T, human behaviors negatively affecting engineering solutions, and fire suppressant effectiveness. Soldier surveys conducted by ARL to identify and characterize problems with equipment and systems used in the field yield valuable feedback that can, if effectively acted on, save lives, promote mission effectiveness, and potentially provide long-term cost savings to the Army.

Progress and advances are evident in the integration of human factors and systems engineering tools. A prime example of this is in the integration of SysML activity diagrams as input to Improved Performance Research Integration Tool (IMPRINT) models. The facilities, tools, and test equipment used in this area (e.g., the renovated obstacle course) appear fully up to date and are exceptional.

Commendable efforts are under way at ARL to advance assessment science by developing new models, tools, and metrics to support the acquisition and fielding of effective human systems responsive to emerging missions and threats. For example, improving the integration of human factors and systems engineering tools in the soldier decision framework model is a significant step forward. Advances in technology, system and mission environments, and changes in soldier roles and the nature of the work they do require attendant and anticipatory advances in assessment science, methods, and tools.

Challenges and Opportunities

The Army continues to be a DOD leader for human systems integration as an integral part of DOD systems acquisition and engineering. As such, it is an important challenge and responsibility for ARL to push the science of human systems evaluation, not just to be the evaluators of record. ARL needs to lead the envisioning and development of new methods and models that are militarily relevant to future technologies, missions, and threats and that will support the acquisition and fielding of effective human systems. As an example, ARL leverage of advances in cognitive engineering can support diagnosis and assessment of complex human systems, human technology interfaces, training programs, and work redesign with methods and tools (e.g., cognitive task analysis) that can identify the mental demands

(e.g., workload) and cognitive skills (e.g., situation awareness, decision making, and planning) needed to complete a task or accomplish a mission.

Another major challenge is to improve the timeliness of human systems analysis and assessment to ensure that it is not cast aside, to the detriment of soldier and mission effectiveness, in the rush for rapid fielding due to war. The process of evaluating new systems in wartime has to allow a rapid response while not ignoring the human science factors. The problem is how to best tailor human systems processes to support wartime (rapid fielding) versus peacetime system acquisition. Enhanced use of modeling of system and mission environments needs to be considered for its potential to accelerate these processes.

ARL needs to be in the feedback loop, both sending and receiving on human systems integration (HSI) issues from system conceptualization, acquisition, and design, through fielding to prevent future problems and failures with the process and to ensure long-term improvements. For example, problems identified prior to deployment that were rectified need to be tracked to determine their impact in the field. Similarly, problems in the field that were identified might warrant changes in process to assure they are captured in the future. Field data on adherence to usage recommendations or human systems effectiveness of equipment need to be systematically fed back to modifications or redesign. This would enable continual improvement of the ARL analysis and assessment processes and their impact on future Army systems. An additional benefit would be a documented audit trail that could support validation of the cost-benefits of the process for use in future baselines.

ARL could use customer-funded projects to motivate human sciences S&T research ideas and directions and to provide pilot data. Because research is fundamental to ARL, even customer-funded projects could push the research agenda. A continual focus on using contract work to provide pilot data would enhance research and identify meaningful data or problems. If the linkage between ARL human sciences research and HSI applications were more transparent, it could facilitate investment by external operational customers in the work. For example, ARL researchers indicated strong desires to connect research-based findings with field-based operational effectiveness and customer-responsive work.

Researchers making connections with military personnel in the field to gain soldier perspective has the potential to be transformative. It can inform ARL research, motivate scientists and engineers, and lead to rapid and/or dramatic changes that can have significant impact on soldiers. Programs that enable ARL investigators to collect field data have been cut, thereby reducing and/or eliminating experiences and data that are extremely valuable to the HSI effort. Assuring these opportunities for research and assessment in the field continue and expand is a key priority. In a similar vein, the Army provides a course whereby new employees are introduced to Army systems and procedures. There would be value in making this real-world sensitization standard procedure available for all ARL civilian scientists and engineers.

OVERALL QUALITY OF THE WORK

Considering the information provided by ARL, it appears that the overall technical quality of the R&D program is good and continually improving. The strategic transformation to campaigns has gone well, and the establishment of ARL-wide initiatives, such as its open campus, is showing positive effects on many aspects of the technical work.

ARL continues on a trajectory of hiring highly skilled postdoctoral researchers, many of whom are being groomed to become full-time ARL employees. Publication in peer-reviewed journals and participation at professional conferences has continued to grow, coupled with increasing participation in professional activities (e.g., journal editing). Collaborations with peer communities appears healthy and provides ARL personnel with invaluable networking opportunities and the options to leverage quality

research elsewhere. ARL’s investment in quality R&D in the human sciences has increased its potential for impact on the present and future Army.

While there has been good progress at growing ARL basic science, more needs to be done. Unique contributions to science in the portfolio presented are not always evident. The key exception is the initiative on individual variability, which has effectively demonstrated the potential to make important contributions to the science of brain state measurement and individual differences. Overall, the ARL neuroscience laboratory, with its emphasis on high-quality R&D and peer-reviewed publications on par with distinguished university peers, is an exemplar on which to grow and sustain basic science across the human sciences KCIs.

The new emphasis on advancing assessment science in the human factors area of assessing mission capability of systems is commendable; it needs to be brought into balance with the assessment-for-hire field assistance in this KCI, to position ARL to effectively support the Army after next. The relationships built through good customer-driven work can be leveraged to gain support for the science and technology needed to advance this capability.

ARL may already possess the in-house expertise to effectively deal with many of the complex challenges anticipated to confront the Army after next; it is important to assure that beneficial synergies that exist across the KCIs are systematically leveraged. Three areas in human sciences might benefit from broader ARL exposure, engagement, and collaborations—big data analysis, autonomous or semiautonomous systems, and EEG-based studies in real-time (or near-real-time) systems.

Linking onsite research to field-based applications work and developing a unique set of capabilities relevant to Army needs, military operations specialties, and contexts potentially represents a unique strength and feature of the ARL human sciences. More work needs to be done here. The natural tension between the comfortable technology pull of customer-driven work and the disruptive potential of innovating through technology push needs to be better balanced to best impact the Army after next. The KCIs are promising in this regard; it may take some time to transform the current program of work and its representation to consistently align with this new core competency structure that links expertise across ARL directorates.

Taken individually, each area of the Human Sciences Campaign (humans in multiagent systems and real-world behavior and human variability) can be cast as a grand challenge, because it is complex, multidisciplinary, and involves many unknowns, requiring multilevel focuses on theory, data, modeling, and engineering to meet stated and implicit goals and objectives. For example, research on individual variability is one of the most difficult problems in the field of psychology; the planned new initiative on unobtrusive immersive measurement of behavior is potentially an enormous problem domain. Measuring and understanding behavior in the real world is an audacious objective given the limits to current understanding of nonlinear causal propagation and emergence in complex, real-world systems and environments. These grand challenges are further complicated by the interdependence and overlap of these areas of the campaign. A coherent strategy and vision, not yet evident, are critically needed for how ongoing and planned research holds together, capitalizes on useful existing theoretical and empirical baselines, and cumulatively builds to achieve goals and objectives over the near, medium, and long terms.

The following projects stand out as particularly noteworthy in the areas of training, integration technologies, and augmentation.

The training program is addressing important scientific and technical challenges that are relevant to both the Army and the larger education and training community. The GIFT project stands out for its significant technical leadership and achievements that are advancing the state of the art and of knowledge for the military training communities. It is a model of excellence in program management and has the opportunity to become a de facto or de jure standard for adaptive, intelligent tutors. Additionally,

the GIFT team has taken the commendable initial steps toward technology transition by engaging in discussions with TRADOC and the ADL collaboration.

The integration technologies program team has employed quality research methods in physiological and behavioral signal measurement and analyses, where they are using advanced machine-learning techniques. One project is a compelling exemplar of some of these activities: the design, development, and demonstration of a multimodal human-system image analysis system. This demonstration project combines computer image analysis, as one expert system, with EEG responses of humans searching for targets in rapid sequence to achieve optimized target recognition from high-throughput image sequences. Using deep learning neural network models (hierarchical convolution networks) of human brain responses and computer image analysis, the goal of this project is to combine both sources to achieve maximized target detection, a task with clear military-relevant applications. This project combines basic research on single-trial EEG, network learning models, and information fusion. The natural combination of implicit measures of human perceptual and cognitive processes alongside automated machine processing of images was an example of innovative progress in this area.

The development of physical augmentation is also a noteworthy accomplishment. The ARL augmentation team is well positioned to become a leading global force in the research and development of augmentation for healthy individuals. There are potential benefits to the Army of these enhanced performance capabilities, and there are also significant opportunities for valuable spin-offs to the civilian sector.

CONCLUSIONS AND RECOMMENDATIONS

Developing a detailed organizing strategy that includes measures of performance and that conveys the breadth and depth of ARL’s training S&T program is a major challenge. However, an evaluation of the scientific and technical program requires that ARL provide the following for the ARLTAB review: a top-level view of major training gaps and a listing of key S&T challenges, including a description of how those challenges are being addressed through near-, mid-, or far-term programs of research by ARL or external organizations. Without more detailed information, the ARLTAB cannot fully address the science, knowledge gaps, and risks associated with the Human Sciences Campaign.

There are common problems and solutions that can be transferred across the three human sciences thrusts presented for the 2016 assessment. Examples of crosscutting themes include cybersecurity for technology under development and multidisciplinary cross-training on techniques and technology. However, the mechanisms for bringing together researchers across the Human Sciences Campaign programs and projects to identify potential areas of collaboration were not apparent. Likewise, the mechanism for communicating potential crosscutting research across ARL campaigns and synergistic disciplines (e.g., human sciences and system theorists) was not made clear.