2
Computing Sciences Division
The Information Sciences Directorate (ISD) describes its programs in computing science as follows: Programs in the Computing Sciences Division are “focused on understanding the fundamental principles and techniques governing computational models and architectures for intelligent, trusted, and resilient computing.” These programs provide “the foundation for revolutionary capabilities for future warfighters in signal and data processing, data fusion, and social informatics.” The programs are Information Processing and Fusion, Computational Architecture and Visualization, Information and Software Assurance, Intelligent Systems, and Advanced Computing (an international program).1 The Computing Sciences Division seeks to conceive of and develop transformational research programs in the computing sciences for the U.S. Army to exploit new computing paradigms and novel information processing techniques.
To accomplish this vision, the division has organized around four programs: Information Assurance, Information Processing and Fusion, Computational Architecture and Visualization, and Intelligent Systems. Currently only three, Information Assurance, Information Processing and Fusion, and Computational Architecture and Visualization, have program managers and active programs, and only those three were reviewed. The fourth, Intelligent Systems, is awaiting the appointment of a program manager. The division’s budget of $29 million, including $6.6 million in core funds and $22.4 million in leveraged funds, supports 188 projects.
INFORMATION ASSURANCE PROGRAM
Overall Scientific Quality and Degree of Innovation
The scientific strategy and selection of projects were of very high quality. The principal investigators (PIs) engaged for the selected projects were highly qualified, and the resulting science was of the highest caliber. The Information Assurance Program had a clearly defined, cogent, and compelling strategy. The focus on the investigation of integrity, trustworthiness, and availability of cybersystems for future military installations and warfighters is substantive yet broad enough to address future environments. The strategic investments in cyberawareness and cyberdefenses are well placed at making progress in key scientific areas.
The Information Assurance Program was well executed, and the resulting science and innovation was exceptional. The investments in strategic projects with high-quality investigators has led to major results and consistent progress in key scientific problems of essential importance to the Army. Army Research Office (ARO)-sponsored investigators at the University of California, Santa Barbara, have created an automated evasive malware detection method based on real hardware systems that can enhance warfighter capability in cyberdefense. A common shortcoming of current virtual machine-based malware analysis
___________________
1 Army Research Laboratory, “Army Research Office: Information Sciences,” http://www.arl.army.mil/www/default.cfm?page=3227, accessed October 10, 2018.
and detection systems is that adversaries can detect the presence of virtual machines and avoid exhibiting malicious behavior to evade detection. The new approach, which executes and analyzes malware samples on native hardware instead of on a virtual machine, enables the capturing of true behavior profiles of evasive malwares for detection. The malware analysis system was recently transitioned to the Cyber Systems Division of the Air Force Life Cycle Management Center at Joint Base San Antonio-Lackland in San Antonio, Texas, for field test and usage. Algorithms of feature extraction for explosive hazard detection for countering improvised explosive devices have been developed at the University of Missouri with ARO funding. The technology has been transitioned to the U.S. Army Communications-Electronics Research, Development, and Engineering Center (CERDEC) Night Vision and Electronic Sensors Directorate for evaluation and field test.
The program has had substantially more impact on the science of security than might be expected given the research expenditures. This appeared to be largely owing to the continuity of the program and the effectiveness of the program manager.
There does not appear to be any need to make major adjustments to the program. The overall direction and the process used to create and execute the program portfolio are appropriate. An increase in the size and scope of the research program might allow better alignment with future Army needs.
Scientific Opportunity
The program could have more impact on the Army, without impacting existing program substance or strategic investments, by adding investments in higher risk and higher potential research projects. These investments would benefit from more direct integration with other ARO programs.
Significant Accomplishments
The impact and accomplishments of the program were outstanding. The number and quality of the publications resulting from this program were exceptional for a program of this size. For a program of this size, the number of students and postdoctoral researchers supported was outstanding.
The connection between the program accomplishments and the strategic plan was not always clear. It is important that the program manager identify the most important accomplishments as they relate to the key strategic goals for the Army and clearly define and use a set of metrics to measure the progress of the program. Appendix A of this report lists a broad set of metrics that ARO could consider for assessment of its programs.
Relevance/Transitions
There were many examples of significant transitions to start-ups and other companies, and to other Army organizations, including the Army Research Laboratory (ARL). The number and quality of transitions were exceptional for a program of this size (52 current projects). The portfolio was highly relevant and responsive to future Army needs.
ARO-funded research on anti-phishing techniques at Carnegie Mellon University (CMU) was transitioned to Wombat Security, a CMU start-up company. ARO also provided critical support during the early stage of the company to mature the technology for commercialization. Wombat has grown to become the major player in the anti-phishing market and was acquired in 2018 for $225 million, showing a huge impact that ARO-funded research and commercialization has brought about. Under ARO support, Intelligent Automation, Inc., developed the DeepRadio technology to provide reconfigurable embedded implementation of deep neural networks as a stand-alone radio platform for characterizing the radio frequency (RF) spectrum environment in real time and adapting to spectrum dynamics. This technology
has been transitioned to the CERDEC. At the Fort Dix field test, DeepRadio demonstrated effectiveness in detecting RF-interfering sources and mitigating their effects on wireless communications. In the field test, DeepRadio successfully learned the behavior of a dynamic jammer that does not transmit continuously and is hard to detect, using a deep neural network model and providing a more than 85 percent success rate of detecting potential jammers.
Additional Considerations
The projects, PIs, focus, relevance, and results of the Information Assurance Program were exceptional. This strength of the program was largely owing to the effectiveness and continuity of the program manager, who is widely viewed in the cybersecurity science and technology community as a thought leader. Documentation and formalizing of the mechanics and strategy of the Information Assurance Program could be used to educate other program managers and replicate the success of the program.
INFORMATION PROCESSING AND FUSION PROGRAM
Overall Scientific Quality and Degree of Innovation
The Information Processing and Fusion Program exhibited high-quality, innovative research with an excellent and diverse group of PIs. There is a very good mixture of projects. The program also extensively leverages funding sources such as Multidisciplinary University Research Initiatives (MURIs) and Small Business Innovation Research (SBIR) and Small Business Technology Transfer (STTR) programs, and it makes close connections with related Army efforts such as the Collaborative Technology Alliances (CTAs) at the Army Research Laboratory. The program performed very well in terms of funding leverage, collaboration, relevance to Army needs, publications, students supported, and transitions.
ARO-sponsored research in compressive sensing at Rice University has led to magnetic resonance imaging (MRI) technologies that are likely to revolutionize soldier healthcare by significantly reducing MRI acquisition time, enabling dramatically shortened scans. The compressive sensing framework provides methods and algorithms to computationally reconstruct images from minimum, compressed measurements, which reduces image acquisition time from 4 minutes for traditional MRI technologies to just 16 seconds. The major MRI vendor, Siemens, Inc., has licensed the technology from Rice University to create new systems based on compressive sensing MRI, which recently received approval from the Food and Drug Administration. This compressive sensing framework can also be applied to other imaging applications that use infrared or radar modalities for efficient image acquisition.
The presentation of the program vision did not make clear its connection with the research topics chosen for funding. The presentation of highlighted research projects did not articulate the significance of the research very well, making it challenging to assess the quality of the research. The value of some of the research topics, such as the MURI on Information Noncommutativity, was not made clear. There are challenges of communicating research content and new advances in this program area, but it should be possible to craft compelling material that shows the value of the research and the advances that are being made. For example, the presentations had a number of equations and graphics that did not effectively communicate any new information.
Scientific Opportunity
The objectives of the program cover a broad range of topics with growing scientific opportunities, such as multimodal data, information frameworks, sparse representations, causality inference, and value-driven information processing. The scientific objectives were focused on nearer term opportunities; longer term opportunities could be considered, and higher risk, potentially higher payoff topics could be included in the portfolio. There could be good future opportunities for this research area related to manned/unmanned teaming, robotic and autonomous systems, and robotic swarms.
Significant Accomplishments
New techniques using compressive sensing algorithms for data reconstruction have been transitioned from Rice University to Conoco-Phillips, a major energy company in the application of seismic sensing for oil exploration. Implementation of the new technique on the company’s system has saved several hundred million dollars in data acquisition and analysis costs.
The large amount of MURI funding, alignment with CTAs at the ARL, and SBIR/STTR funding indicates a high degree of success in building a diverse, dynamic research portfolio. The program could endeavor to feature major programmatic accomplishments (such as compressive sensing) more prominently. The publication count and quality are strong. Considering the size of the budget ($9 million supporting 75 projects, of which $2.5 million is ARO core funds and $6.5 million is leveraged funds), a commendable number of students and postdoctoral researchers are supported. However, it was sometimes hard to determine how the accomplishments mapped to the goals of the program.
Relevance/Transitions
The research areas covered by the Information Processing and Fusion Program overall have an extremely high relevance to the future Army and align with several of the army modernization priorities. The quantity and quality of the transitions, which include transitions within the Army and to the commercial sector, are commendable. The following are examples of transitions that have been accomplished in Army-relevant areas: video surveillance for human activity detection, multimodal biometrics, human-robot communication, and battlefield decision-making systems. The program manager makes a concerted effort to reach out to investigators with other organizations in the Army to create leveraging and transitions. However, the relevance of the programs to future Army needs was not articulated well during the review.
Additional Considerations
The Information Processing and Fusion Program includes projects that could transition to the new Intelligent Systems Program to accelerate the start of that program.
COMPUTATIONAL ARCHITECTURE AND VISUALIZATION PROGRAM
Overall Scientific Quality and Degree of Innovation
The Computer Architecture and Visualization Program articulated three scientific objectives: create interactive yet accurate visualizations coupled with simulations; create new energy-efficient architectures for multicore, hybrid, and exascale systems to operate in a resource-constrained environment; and devise
scalable and communication-efficient algorithms to effectively handle the complex data arising from Army applications. Within this, the work on visualization is largely independent of that in computer architecture. The strategy relating scientific objectives to the fundamental, long-term science was not as explicit and well-articulated as in the other programs within the division.
Particular strengths of the program are that the PIs have impressive research credentials, or promise thereof for the younger PIs, and that for the most part the areas under investigation are relevant to Army needs.
The visualization work is aimed at capabilities beyond classical visualization while remaining in line with the generation of simulation results in a way that avoids the display of visual artifacts that distort the interpretation of the results. This is an area that may have value both to classical computational science and in newer multi-agent applications. The work described addressed both areas.
The bulk of the recent computer architecture work seemed to be focused on a shorter time frame than that of many of the division’s other programs and was not focused on architecture as classically defined. The work described on message-passing libraries for big data and on scheduling for graphics processing units was reasonable but very short term; it is likely that similar work is being performed at a much larger scale by major commercial concerns. However, considering the titles of some of the other projects funded under the program but not reviewed, such as the work on approximate computing, and the backgrounds of their PIs, it is likely that some of the architecture work funded in the recent past did have a longer time horizon with a deeper microarchitecture focus, and that work may have held out the potential for developing more fundamental results.
The current portfolio of work in computer architecture within the Computer Architecture and Visualization Program may be more appropriately aligned with the yet-to-be initiated Intelligent Systems Program. The portfolio of work in computer architecture needs to include some higher risk, potentially higher payoff topics. In particular, the portfolio of work in computer architecture needs to include more projects that are focused on computer architecture, that have a longer time horizon, and that have the potential to suggest fundamental changes to computing systems. The above-mentioned approximate computing work is a good example of this. Working with some of the ARL projects, such as at Aberdeen on neuromorphic (e.g., IBM’s TrueNorth processor) and tiled architectures, may help identify issues with today’s emerging architectures that may be fodder for alternative approaches. Alignment with the proposed Intelligent Systems Program may also provide identification of application-level issues that may engender the need for new architecture features (e.g., special-purpose coprocessors to accelerate machine learning, such as Google’s tensor processing unit).
Scientific Opportunity
The computer architecture portion of the program is not forward-looking enough. There is significant scientific opportunity in computer architecture—in particular, in the design of special-purpose processors that can accelerate the solution of applications important to the Army. This also offers the opportunity to significantly reduce power consumption, which does not seem to be a focus of the current computer architecture projects, although it was one of the three stated scientific objectives of the Computer Architecture and Visualization Program. There appear to be only two projects in the portfolio of 34 projects that had energy efficiency as one of the design goals.
It is important that new projects in the Computer Architecture and Visualization Program have higher potential and higher risk. This is particularly the case for projects in the computer architecture space, which currently are too short-term focused and not focused enough on energy-efficient design.
Significant Accomplishments
Considering the size of the program ($1.6 million, including $1.2 million in core funding, supporting 34 current projects), the publication count and quality are strong, and a commendable number of students and postdoctoral researchers are supported. It was sometimes hard to determine how the accomplishments of the projects mapped to the scientific objectives of the program. It is necessary that the project PIs know about and understand the scientific objectives of the program and align their work with at least one of those objectives.
Relevance/Transitions
There were many examples of significant transitions to start-ups, companies, and other Army organizations. Examples are a personalized gait simulation, transitioned to the Institute for Creative Technology; new data structures and uncertainty qualification techniques, transitioned to Scalable Algorithmics, Inc.; mixed-criticality scheduling software, transitioned to General Motors Corporation; and load balancing algorithms, transitioned to the U.S. Army Tank Automotive Research, Development and Engineering Center (TARDEC). However, none of these involved architecture-related artifacts. On the whole, the portfolio was relevant to future Army needs.
Additional Considerations
A very large percentage of the recent grants seemed to be for institutions in the state of North Carolina. The pool of project proposers was not presented; ARO may want to examine that pool to determine whether greater diversity in institutions supported by the program’s funding is warranted.
INTELLIGENT SYSTEMS PROGRAM
The Intelligent Systems Program has not yet been initiated, but an overview of the plan for the program was presented. The topics to be covered by the Intelligent Systems Program represent a major and relevant opportunity for addressing future Army needs.
The program needs to be started as soon as possible, employing, if necessary, an interim program manager to get it started. Because this is a very “hot” topic and competition is fierce for experts, it may be worthwhile to consider a program manager who can grow into the position or an adjustment to the program’s focus to make the position more attractive. Appropriately performed, program adjustment, if any, would remain within the bounds of the stated program thrusts: advanced learning theory, methodology, and techniques; and adaptive, robust, and pervasive intelligent systems.
OVERALL ASSESSMENT
Overall Scientific Quality and Degree of Innovation
Overall, the scientific strategy and selection of projects were of high quality. The PIs engaged for the selected projects were highly qualified, and the resulting science was of high caliber.
Scientific Opportunity
The scientific objectives were generally focused on nearer term opportunities; longer term opportunities could be considered, and higher risk, potentially higher payoff topics could be included in the portfolio.
Significant Accomplishments
The programs generally performed very well in terms of funding leverage, relevance to Army needs, number and quality of publications, students supported, and transitions. The mapping of project accomplishments to programs’ strategic plans was not always clear, and consistent, meaningful metrics for assessing progress were generally lacking. Appendix A of this report lists a broad set of metrics that ARO could consider for assessment of its programs.
Relevance/Transitions
The division has selected its areas of focus to complement work supported by other agencies and does coordinate extensively but informally with them. The intent is to conduct longer term research in areas of Army-specific need that are not addressed by commercial and other government entities. This is a challenge in computing because the rate of change is so rapid, particularly since the end of Dennard scaling and the rise of data-centric computing.
The division’s programs showed impressive examples of transitions to other Army and wider DoD research and development elements and in some cases to commercial organizations.
