1
Introduction
This introductory chapter describes the biennial assessment process conducted by the National Academies of Sciences, Engineering, and Medicine’s Army Research Laboratory Technical Assessment Board (ARLTAB). It then describes the preparation and organization of the report, the assessment criteria, and the approach taken during the report preparation.
THE BIENNIAL ASSESSMENT PROCESS
The ARLTAB is guided by the following statement of task:
The National Academies of Sciences, Engineering, and Medicine will appoint a committee to be named the Army Research Laboratory Technical Assessment Board (ARLTAB), to be overseen by the Laboratory Assessments Board (LAB), to continue the function of providing assessments of the scientific and technical quality of the research at the Army Research Laboratory (ARL). The committee will develop a series of reports that summarize the ARLTAB’s findings, conclusions, and recommendations related to the quality of ARL’s research programs.
Every two years, the committee will assess six portions of the ARL technical program (three in year 1, three in year 2). To conduct its assessments the ARLTAB will be assisted by up to six separately appointed panels. The assessment completed in Year 1 will be released as an interim assessment report, which will be combined with the second year’s assessment as a final biennial report.
The committee will also review the extramural basic research programs at the Army Research Office (ARO), a subdivision of the ARL. Each year the ARLTAB will be supported by an additional panel that will focus on selected ARO programs, and the ARLTAB will annually provide a separate assessment report.
While the primary role of the ARLTAB is to provide peer assessment, it may also offer advice on related matters when requested by the ARL Director.
The charge of the ARLTAB is to provide biennial assessments of the scientific and technical quality of the Army Research Laboratory (ARL). These assessments include the development of findings and recommendations related to the quality of ARL’s research, development, and analysis programs. The ARLTAB is charged to review the work in ARL’s research core competencies: network and information sciences, computational sciences, human sciences, materials and manufacturing sciences, propulsion sciences, ballistic sciences, and protection sciences,1 but not the work of the Army Research Office (ARO), a key element of the ARL organization that manages and supports basic research. However, all ARLTAB panels receive reports of how the research and development (R&D) activities of ARO and ARL
___________________
1 The ARLTAB will assess the ballistic sciences and protection sciences research core competencies in 2021 because the planned in-person 2020 meeting of the panel could not take place owing to COVID-19. Its meeting could not be conducted virtually because the research in ballistic sciences and protection sciences areas involved controlled unclassified information and classified information.
are coordinated. The ARO programs are assessed by different panels that are also overseen by the ARLTAB.
In addition, at the discretion of the ARL director, the ARLTAB reviews selected portions of the work conducted by the Collaborative Technology Alliances (CTAs) and Cooperative Research Alliances (CRAs). Although the ARLTAB’s primary role is to provide peer assessment, it may also offer advice on related matters when requested to do so by the ARL director; such advice focuses on technical rather than programmatic considerations. To conduct its assessments, the ARLTAB is assisted by five National Academies panels, each of which focuses on one or more of ARL’s research core competencies. The ARLTAB’s assessments are commissioned by ARL itself rather than by one of its parent organizations.
For this assessment, the ARLTAB consisted of 10 leading scientists and engineers whose collective experience spans the main topics within ARL’s scope. Five panels, each of which focuses on one or more of ARL’s research core competencies, report to the ARLTAB. Nine of the ARLTAB members serve as chairs or co-chairs of these panels or the panels assessing the ARO programs. The panels range in size from 13 to 19 members, whose expertise is carefully matched to the technical fields covered by the areas that they review. Selected members of each panel attend each annual review. In total, 68 members participated in the reviews that led to this report. All panel and ARLTAB members participate without compensation.
The National Academies appointed the ARLTAB and panel members with an eye to assembling a slate of experts without conflicts of interest and with balanced perspectives. The experts include current and former executives and research staff from industrial R&D laboratories, leading academic researchers, and staff from the Department of Energy (DOE) national laboratories and federally funded R&D centers. Seventeen are members of the National Academy of Engineering, and four are members of the National Academy of Sciences. A number have been leaders in relevant professional societies. The ARLTAB and its panels are supported by National Academies staff, who interact with ARL on a continuing basis to ensure that the ARLTAB and the panels receive the information they need to carry out their assessments. The ARLTAB and panel members serve for finite terms, generally 4 to 6 years, so that viewpoints are regularly refreshed and the expertise of the ARLTAB and panel members continues to match ARL’s activities. Biographical information on ARLTAB members appears in Appendix B.
In 2019 and 2020, four panels reviewed the following research core competencies of ARL:
- Human sciences,
- Network and information sciences,
- Computational sciences,
- Materials and manufacturing sciences, and
- Propulsion sciences.
This biennial report summarizes the findings of the ARLTAB from the reviews conducted by the panels in 2019 and 2020 and subsumes the 2019-2020 interim report.2
PREPARATION AND ORGANIZATION OF THIS REPORT
The current report is the eleventh biennial report of the ARLTAB. Its first biennial report was issued in 2000; annual reviews were issued in 1996, 1997, 1998, 2013, 2015, 2017, and 2019. As with the earlier reviews, this report contains the ARLTAB’s judgments about the quality of ARL’s work. (Chapters 2
___________________
2 National Academies of Sciences, Engineering, and Medicine, 2020, 2019-2020 Assessment of the Army Research Laboratory: Interim Report, The National Academies Press, Washington, D.C.
through 6 focus on the research core competencies areas, and Chapter 7 provides a discussion of crosscutting issues across all of ARL.) The rest of this chapter explains the rich set of interactions that supports those judgments.
The amount of information that is funneled to the ARLTAB, including the evaluations by the recognized experts who make up the ARLTAB’s panels, provides a solid foundation for a thorough peer review. This review is based on a large amount of information received from ARL and on interactions between ARL staff and the ARLTAB and its panels.
To further improve the overall process of preparing for review, the ARLTAB requests that information listing the full spectrum of projects and programs in each ARL campaign be provided and an interrelated mapping of projects across all campaigns be provided as a read-ahead before the panel meeting. The process of preparing such a snapshot in time would be of utility in focusing future reviews as well as to ARL in developing cohesiveness and priorities.
Most of the information exchange occurs during the annual meetings convened by the respective panels at the appropriate ARL sites. Both in formal interactions during technical presentations and in less formal interactions during poster sessions and during joint lunch or dinner sessions,3 ARL evinces a very healthy level of information exchange and acceptance of external comments. The assessment panels and the ARLTAB engaged in many constructive interactions with ARL staff during their annual site visits in 2019 and the virtual meetings in 2020. In addition, useful collegial exchanges took place between panel members and individual ARL investigators during the meetings as ARL staff members sought clarification about panel comments or questions and drew on panel members’ contacts and sources of information.
Each panel’s review meeting lasted about 2.5 days, during which time the panel members received a combination of overview briefings by ARL management and technical briefings by ARL staff. Prior to the meetings, panels received extensive materials for review, including selected staff publications.
The overview briefings brought the panels up to date on the broad scope of ARL’s scientific and technical work. This context-building step was needed because the panels are purposely composed of people who, while experts in the technical fields covered by ARL’s research core competencies that they reviewed, were not engaged in collaborative work with ARL. Technical briefings for the panels focused on R&D goals, strategies, methodologies, and results of selected projects at the laboratory. Briefings were targeted at coverage of a representative sample of each of ARL’s research core competencies. Briefings included poster sessions that allowed direct interaction among the panelists and staff of projects that were not covered in the briefings.4
Ample time during both the overview and the technical briefings was devoted to discussion, which enabled panel members to pose questions and ARL staff to provide additional technical and contextual information to clarify panel members’ understanding. The panels also devoted sufficient time to closed-session deliberations, during which they developed findings and identified important questions or gaps in panel understanding. Those questions or gaps were discussed during follow-up sessions with ARL staff so that the panel was confident of the accuracy and completeness of its assessments. Panel members continued to refine their findings, conclusions, and recommendations during written exchanges and teleconferences among themselves after the meetings.
In addition to the insights that they gained from the panel meetings, ARLTAB members received exposure to ARL and its staff at ARLTAB meetings each winter. The 2019 and 2020 ARLTAB meetings refined elements of the assessment process focused on ARL’s research core competencies, including read-ahead materials, review agendas, and expertise required within the panels.
___________________
3 As noted earlier, the 2020 panel meetings on materials and manufacturing sciences and propulsion sciences were held virtually and did not benefit from such joint lunch or dinner sessions.
4 Agendas of the panel meetings can be found at the National Academies of Sciences, Engineering, and Medicine website at http://www8.nationalacademies.org/cp/ for each respective panel.
ASSESSMENT CRITERIA
During the assessment, the ARLTAB and its panels considered the following questions posed by the ARL director:
- Is the scientific quality of the research of comparable technical quality to that executed in leading federal, university, and industrial laboratories both nationally and internationally?
- Does the research program reflect a broad understanding of the underlying science and research conducted elsewhere?
- Does the research employ the appropriate laboratory equipment and numerical models?
- Are the qualifications of the research team compatible with the research challenge?
- Are the facilities and laboratory equipment state of the art?
- Are programs crafted to employ the appropriate mix of theory, computation, and experimentation?
To assist ARL in addressing promising technical approaches, the ARLTAB also considered the following questions:
- Are there especially promising projects that, with improved direction or resources, could produce outstanding results that can be transitioned ultimately to the field?
- Are there promising outside-the-box concepts that could be pursued but are not currently in the ARL portfolio?
Within the general framework described earlier, the ARLTAB also developed, and the panels selectively applied, detailed assessment criteria organized in the following four categories (Appendix C presents the complete set of assessment criteria):
- Project goals and plans. Criteria in this category relate to the extent to which projects address ARL strategic technical goals and are planned to effectively achieve the stated objectives.
- Methodology and approach. Criteria in this category address the appropriateness of the hypotheses that drive the research, of the tools and methods applied to the collection and analysis of data, and of the judgments about future directions of the research.
- Capabilities and resources. Criteria in this category relate to whether current and projected equipment, facilities, and human resources are appropriate to achieve success of the projects.
- Scientific community. Criteria in this category relate to cognizance of and contributions to the scientific and technical community whose activities are relevant to the work performed at ARL.
APPROACH TAKEN DURING REPORT PREPARATION
This report represents the ARLTAB’s consensus findings and recommendations, developed through deliberations, which included consideration of the notes prepared by the panel members summarizing their assessments. The ARLTAB’s aim with this report is to provide guidance to the ARL director that will help ARL sustain its process of continuous improvement. To that end, the ARLTAB examined its extensive and detailed notes from the many ARLTAB panel and individual interactions with ARL during 2019-2020. From those notes, it distilled a shorter list of the main trends, opportunities, and challenges that merit attention at the level of the ARL director and the management team. The ARLTAB used that list as the basis for this report. Specific ARL projects are used to illustrate these points in the following
chapters when it is helpful to do so, but the ARLTAB did not aim to present the director with a detailed account of interactions with bench scientists. The draft of this report was subsequently honed and reviewed according to the National Academies’ procedures before being released.
The ARLTAB applied a largely qualitative rather than quantitative approach to the assessment. The approach of the ARLTAB and its panels relied on the experience, technical knowledge, and expertise of its members, whose backgrounds were carefully matched to the core technical competency areas in which ARL activities are conducted. The ARLTAB and its panels reviewed selected examples of the scientific and technological research performed by ARL; it was not possible to review all ARL programs and projects exhaustively. Given the necessarily non-exhaustive nature of the review process, the omission of mention of any particular program or project should not be interpreted as a negative reflection on the omitted program or project.
The ARLTAB’s goal was to identify and report salient examples of accomplishments and opportunities for further improvement with respect to the technical merit of ARL work and specific elements of ARL’s resource infrastructure that are intended to support the technical work. Collectively, these highlighted examples for each ARL research core competency are intended to portray an overall impression of the laboratory while preserving useful mention of suggestions specific to projects and programs that the ARLTAB considered to be of special note within the set of those examined.
Along these lines, in applying the first assessment criteria (“Is the scientific quality of the research of comparable technical quality to that executed in leading federal, university, and industrial laboratories both nationally and internationally?”), the ARLTAB uniformly characterize ARL’s work as high, moderate, or low quality compared to that executed in leading federal, university, and industrial laboratories both nationally and internationally.
REPORT CONTENT
This chapter has addressed the biennial assessment process used by the ARLTAB and its five panels. Chapters 2 through 6 provide detailed assessments of the research core competencies reviewed during 2019-2020. Chapter 7 presents findings common across multiple research core competencies. The appendixes provide ARL’s research core competencies and their mapping to the technical areas reviewed in 2019 and 2020, biographical information on the ARLTAB members, the assessment criteria used by ARLTAB and its panels, and a list of acronyms found in this report.
