National Academies Press: OpenBook
« Previous: Summary
Suggested Citation:"1 Introduction." National Research Council. 2013. 2011-2012 Assessment of the Army Research Laboratory. Washington, DC: The National Academies Press. doi: 10.17226/18269.
×

1

Introduction

This introductory chapter describes the biennial assessment process conducted by the National Research Council’s (NRC’s) Army Research Laboratory Technical Assessment Board (ARLTAB). It then describes the preparation and organization of the report, the assessment criteria, and the approach taken during the report preparation.

THE BIENNIAL ASSESSMENT PROCESS

The charge of ARLTAB is to provide biennial assessments of the scientific and technical quality of the Army Research Laboratory (ARL). These assessments include the development of findings and recommendations related to the quality of ARL’s research, development, and analysis programs. The ARLTAB is charged to review the work of ARL’s six directorates but not the work of the Army Research Office (ARO), a key element of the ARL organization that manages and supports basic research; however, all ARLTAB panels receive reports of how the research and development activities of ARO and ARL are coordinated. At the discretion of the ARL Director, the ARLTAB reviews selected portions of the work conducted by the Collaborative Technology Alliances (CTAs). Although the ARLTAB’s primary role is to provide peer assessment, it also may offer advice on related matters when requested to do so by the ARL Director; such advice focuses on technical rather than programmatic considerations. The ARLTAB is assisted by six NRC panels that focus on particular portions of the ARL program. The ARLTAB’s assessments are commissioned by ARL itself rather than by one of its parent organizations.

For this assessment, the ARLTAB consisted of seven leading scientists and engineers whose collective experience spans the major topics within ARL’s scope. Six panels, one for each of ARL’s

Suggested Citation:"1 Introduction." National Research Council. 2013. 2011-2012 Assessment of the Army Research Laboratory. Washington, DC: The National Academies Press. doi: 10.17226/18269.
×

directorates,1 report to the ARLTAB. Six of the ARLTAB members serve as panel chairs. The panels range in size from 14 to 22 members, whose expertise is carefully matched to the technical fields covered by the directorate(s) that they review. In total, 110 experts participated, without compensation, in the process that led to this report.

The NRC appoints the ARLTAB and panel members with an eye to assembling slates of experts without conflicts of interest and with balanced perspectives. The 110 experts include current and former executives and research staff from industrial research and development (R&D) laboratories, leading academic researchers, and staff from Department of Energy national laboratories and federally funded R&D centers. Twenty-seven of them are members of the National Academy of Engineering, and 4 are members of the National Academy of Sciences. A number have been leaders in relevant professional societies, and several are past members of organizations such as the Army Science Board and the Defense Science Board. The ARLTAB and its panels are supported by NRC staff, who interact with ARL on a continuing basis to ensure that the ARLTAB and panels receive the information that they need to carry out their assessments. ARLTAB and panel members serve for finite terms, generally 4 to 6 years, so that viewpoints are regularly refreshed and the expertise of the ARLTAB and panel members continues to match the ARL’s activities.

Biographical information on the ARLTAB members appears in Appendix B, along with a list of each panel’s members.

PREPARATION AND ORGANIZATION OF THIS REPORT

The current report is the seventh biennial report of ARLTAB. Its first biennial report was issued in 2000; annual reviews were issued in 1996, 1997, and 1998. As with the earlier reviews, this report contains the ARLTAB’s judgments about the quality of ARL’s work (Chapters 2 through 7 focus on the individual directorates, and Chapter 8 provides a discussion of crosscutting issues across all of ARL). The rest of this chapter explains the rich set of interactions that support those judgments.

The amount of information that is funneled to the ARLTAB, including the evaluations of the recognized experts who make up the ARLTAB’s panels, provides a solid foundation for a thorough peer review. This review is based on a large amount of information received from ARL and on interactions between ARL staff and the ARLTAB and its panels. Most of the information exchange occurs during the annual meetings convened by the respective panels at the appropriate ARL sites. Both at scheduled meetings and in less formal interactions, ARL evinces a very healthy level of information exchange and acceptance of external comments. The assessment panels engaged in many constructive interactions with ARL staff during their annual site visits in 2011 and 2012. The introductory sections of Chapters 2 through 7 provide the dates of the panel site visits for each directorate. In addition, useful collegial exchanges took place between panel members and individual ARL investigators outside of scheduled meetings as ARL staff members sought additional clarification about panel comments or questions and drew on panel members’ contacts and sources of information.

Each panel meeting lasted 2.5 days, during which time the panel members received a combination of overview briefings by ARL management and technical briefings by ARL staff. Prior to the meetings, panels received extensive materials for review, including selected staff publications.

___________________________

1The six ARL directorates are the Computational and Information Sciences Directorate (CISD), Human Research and Engineering Directorate (HRED), Sensors and Electron Devices Directorate (SEDD), Survivability/Lethality Analysis Directorate (SLAD), Vehicle Technology Directorate (VTD), and Weapons and Materials Research Directorate (WMRD). Appendix A provides information summarizing the organization and resources of ARL and its directorates.

Suggested Citation:"1 Introduction." National Research Council. 2013. 2011-2012 Assessment of the Army Research Laboratory. Washington, DC: The National Academies Press. doi: 10.17226/18269.
×

The overview briefings brought the panels up to date on ARL’s long-range planning. This context-building step is needed because the panels are purposely composed mostly of people who—while experts in the technical fields covered by the directorates(s) that they review—are not engaged in work focused on ARL matters. Technical briefings for the panels focused on the R&D goals, strategies, methodologies, and results of selected projects at the laboratory. Briefings were targeted toward coverage of a representative sample of each directorate’s work over the 2-year assessment cycle. Briefings included poster sessions that allowed direct interaction among the panelists and staff of projects that either were not covered in the briefings or had been covered in prior years.

Ample time during both overview and technical briefings was devoted to discussion, which enabled panel members to pose questions and ARL staff to provide additional technical and contextual information to clarify panel members’ understanding. The panels also devoted sufficient time to closed-session deliberations, during which they developed findings and identified important questions or gaps in panel understanding. Those questions or gaps were discussed during follow-up sessions with ARL staff so that the panel was confident of the accuracy and completeness of its assessments. Panel members continued to refine their findings, conclusions, and recommendations during written exchanges and teleconferences among themselves after the meetings.

In addition to the insights that they gained from the panel meetings, ARLTAB members received exposure to ARL and its staff at ARLTAB meetings each winter. The 2011 ARLTAB meeting focused on the ARL crosscutting research areas, and the 2012 ARLTAB meeting focused on refining elements of the assessment process, including read-ahead materials, review agendas, and expertise required within the panels.

ASSESSMENT CRITERIA

During the assessment, the ARLTAB and its panels considered the following questions posed by the ARL Director:

1. Is the scientific quality of the research of technical quality comparable to that executed in leading federal, university, and/or industrial laboratories both nationally and internationally?

2. Does the research program reflect a broad understanding of the underlying science and research conducted elsewhere?

3. Does the research employ the appropriate laboratory equipment and/or numerical models?

4. Are the qualifications of the research team compatible with the research challenge?

5. Are the facilities and laboratory equipment state of the art?

6. Does the research reflect an understanding of the Army’s requirement for the research or the analysis?

7. Are programs crafted to employ the appropriate mix of theory, computation, and experimentation?

8. Is the work appropriate to the ARL niche?

9. Are there especially promising projects that, with application of adequate resources, could produce outstanding results that could be transitioned ultimately to the field?

Within the general framework described above, the ARLTAB also developed and the panels selectively applied detailed assessment criteria organized in the following six categories (Appendix C presents the complete set of assessment criteria):

Suggested Citation:"1 Introduction." National Research Council. 2013. 2011-2012 Assessment of the Army Research Laboratory. Washington, DC: The National Academies Press. doi: 10.17226/18269.
×

1. Effectiveness of interaction with the scientific and technical community—criteria in this category relate to cognizance of and contributions to the scientific and technical community whose activities are relevant to the work performed at ARL;

2. Impact on customers—criteria in this category relate to cognizance of and contributions in response to the needs of the Army customers who fund and benefit from ARL R&D;

3. Formulation of projects’ goals and plans—criteria in this category relate to the extent to which projects address ARL strategic technical goals and are planned effectively to achieve stated objectives;

4. R&D methodology—criteria in this category address the appropriateness of the hypotheses that drive the research, of the tools and methods applied to the collection and analysis of data, and of the judgments about future directions of the research;

5. Capabilities and resources—criteria in this category relate to whether current and projected equipment, facilities, and human resources are appropriate to achieve success of the projects; and

6. Responsiveness to the ARLTAB’s recommendations—with respect to this criterion, the ARLTAB does not consider itself to be an oversight committee. The ARLTAB has consistently found ARL to be extremely responsive to its advice, so the criterion of responsiveness encourages discussion of the variables and contextual factors that affect ARL’s implementation of responses to recommendations rather than an accounting of responses to the ARLTAB’s recommendations.

APPROACH TAKEN DURING REPORT PREPARATION

This report represents the ARLTAB’s consensus findings and recommendations, developed through deliberations that included consideration of the notes prepared by the panel members summarizing their assessments. The ARLTAB’s aim with this report is to provide guidance to the ARL Director that will help ARL sustain its process of continuous improvement. To that end, the ARLTAB examined its extensive and detailed notes from the many ARLTAB, panel, and individual interactions with ARL during the 2011-2012 period. From those notes it distilled a shorter list of the main trends, opportunities, and challenges that merit attention at the level of the ARL Director and his management team. The ARLTAB used that list as the basis for this report. Specific ARL projects are used to illustrate these points in the following chapters when it is helpful to do so, but the ARLTAB did not aim to present the Director with a detailed account of 2 years’ worth of interactions with bench scientists. The draft of this report was subsequently honed and reviewed according to NRC procedures before being released.

The approach to the assessment by the ARLTAB and its panels relied on the experience, technical knowledge, and expertise of its members, whose backgrounds were carefully matched to the technical areas within which the ARL activities are conducted. The ARLTAB and its panels reviewed selected examples of the standards and measurements activities and the technological research presented by ARL; it was not possible to review all ARL programs and projects exhaustively. The ARLTAB’s goal was to identify and report salient examples of accomplishments and opportunities for further improvement with respect to the technical merit of the ARL work, its perceived relevance to ARL’s definition of its mission, and apparent specific elements of ARL’s resource infrastructure that are intended to support the technical work. Collectively, these highlighted examples for each ARL directorate are intended to portray an overall impression of the laboratory while preserving useful mention of suggestions specific to projects and programs that the ARLTAB considered to be of special note within the set of those examined. The ARLTAB applied a largely qualitative rather than quantitative approach to the assessment. The assess-

Suggested Citation:"1 Introduction." National Research Council. 2013. 2011-2012 Assessment of the Army Research Laboratory. Washington, DC: The National Academies Press. doi: 10.17226/18269.
×

ment panels’ site visits are currently scheduled to be repeated annually, and the assessment report to be issued biennially.

REPORT CONTENT

This chapter discusses the biennial assessment process used by ARLTAB and its six panels. Chapters 2 through 7 provide detailed assessments of each of the six ARL directorates. Chapter 8 presents an overview focused on crosscutting issues across all of ARL. The appendixes provide the ARL organizational chart, biographical information on the ARLTAB members, a list of the panel membership, the assessment criteria used by ARLTAB and its panels, and a list of acronyms found in the report.

Suggested Citation:"1 Introduction." National Research Council. 2013. 2011-2012 Assessment of the Army Research Laboratory. Washington, DC: The National Academies Press. doi: 10.17226/18269.
×
Page 13
Suggested Citation:"1 Introduction." National Research Council. 2013. 2011-2012 Assessment of the Army Research Laboratory. Washington, DC: The National Academies Press. doi: 10.17226/18269.
×
Page 14
Suggested Citation:"1 Introduction." National Research Council. 2013. 2011-2012 Assessment of the Army Research Laboratory. Washington, DC: The National Academies Press. doi: 10.17226/18269.
×
Page 15
Suggested Citation:"1 Introduction." National Research Council. 2013. 2011-2012 Assessment of the Army Research Laboratory. Washington, DC: The National Academies Press. doi: 10.17226/18269.
×
Page 16
Suggested Citation:"1 Introduction." National Research Council. 2013. 2011-2012 Assessment of the Army Research Laboratory. Washington, DC: The National Academies Press. doi: 10.17226/18269.
×
Page 17
Next: 2 Computational and Information Sciences Directorate and Network Science Enterprise »
2011-2012 Assessment of the Army Research Laboratory Get This Book
×
Buy Paperback | $45.00 Buy Ebook | $35.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

The charge of the Army Research Laboratory Technical Assessment Board (ARLTAB) is to provide biennial assessments of the scientific and technical quality of the research, development, and analysis programs at the Army Research Laboratory (ARL). The ARLTAB is assisted by six panels, each of which focuses on the portion of the ARL program conducted by one of ARL's six directorates1. When requested to do so by ARL, the ARLTAB also examines work that cuts across the directorates. For example, during 2011-2012, ARL requested that the ARLTAB examine crosscutting work in the areas of autonomous systems and network science.

The overall quality of ARL's technical staff and their work continues to be impressive. Staff continue to demonstrate clear, passionate mindfulness of the importance of transitioning technology to support immediate and longer-term Army needs. Their involvement with the wider scientific and engineering community continues to expand. Such continued involvement and collaboration are fundamentally important for ARL's scientific and technical activities and need to include the essential elements of peer review and interaction through publications and travel to attend professional meetings, including international professional meetings. In general, ARL is working very well within an appropriate research and development niche and has been demonstrating significant accomplishments, as exemplified in the following discussion, which also addresses opportunities and challenges.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!