National Academies Press: OpenBook
Suggested Citation:"Front Matter." National Research Council. 2002. Reliability Issues for DOD Systems: Report of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/10561.
×

Reliability Issues for DoD Systems

REPORT OF A WORKSHOP

Committee on National Statistics

Francisco Samaniego and Michael Cohen, Editors

Division of Behavioral and Social Sciences and Education

NATIONAL RESEARCH COUNCIL OF THE NATIONAL ACADEMIES

THE NATIONAL ACADEMIES PRESS
Washington, D.C. www.nap.edu

Suggested Citation:"Front Matter." National Research Council. 2002. Reliability Issues for DOD Systems: Report of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/10561.
×

THE NATIONAL ACADEMIES PRESS
500 Fifth Street, NW Washington, DC 20001

NOTICE: The project that is the subject of this report was approved by the Governing Board of the National Research Council, whose members are drawn from the councils of the National Academy of Sciences, the National Academy of Engineering, and the Institute of Medicine. The members of the committee responsible for the report were chosen for their special competences and with regard for appropriate balance.

The project that is the subject of this report was supported by contract DASW01-94-C-0119 between the National Academy of Sciences and the Director of Operational Test and Evaluation at the Department of Defense. Any opinions, findings, conclusions, or recommendations expressed in this publication are those of the author(s) and do not necessarily reflect the views of the organizations or agencies that provided support for the project.

International Standard Book Number 0-309-08606-X

Additional copies of this report are available from the
National Academies Press,
500 Fifth Street, NW, Lockbox 285, Washington, DC 20055 (800) 624-6242 or (202) 334-3313 (in the Washington metropolitan area); Internet, http://www.nap.edu

Copyright 2002 by the National Academy of Sciences. All rights reserved.

Printed in the United States of America

Suggested citation: National Research Council (2002) Reliability Issues for DoD Systems: Report of a Workshop. Committee on National Statistics. Francisco Samaniego and Michael Cohen, editors. Washington, DC: The National Academies Press.

Suggested Citation:"Front Matter." National Research Council. 2002. Reliability Issues for DOD Systems: Report of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/10561.
×

THE NATIONAL ACADEMIES

Advisers to the Nation on Science, Engineering, and Medicine

The National Academy of Sciences is a private, nonprofit, self-perpetuating society of distinguished scholars engaged in scientific and engineering research, dedicated to the furtherance of science and technology and to their use for the general welfare. Upon the authority of the charter granted to it by the Congress in 1863, the Academy has a mandate that requires it to advise the federal government on scientific and technical matters. Dr. Bruce M. Alberts is president of the National Academy of Sciences.

The National Academy of Engineering was established in 1964, under the charter of the National Academy of Sciences, as a parallel organization of outstanding engineers. It is autonomous in its administration and in the selection of its members, sharing with the National Academy of Sciences the responsibility for advising the federal government. The National Academy of Engineering also sponsors engineering programs aimed at meeting national needs, encourages education and research, and recognizes the superior achievements of engineers. Dr. Wm. A. Wulf is president of the National Academy of Engineering.

The Institute of Medicine was established in 1970 by the National Academy of Sciences to secure the services of eminent members of appropriate professions in the examination of policy matters pertaining to the health of the public. The Institute acts under the responsibility given to the National Academy of Sciences by its congressional charter to be an adviser to the federal government and, upon its own initiative, to identify issues of medical care, research, and education. Dr. Harvey V. Fineberg is president of the Institute of Medicine.

The National Research Council was organized by the National Academy of Sciences in 1916 to associate the broad community of science and technology with the Academy’s purposes of furthering knowledge and advising the federal government. Functioning in accordance with general policies determined by the Academy, the Council has become the principal operating agency of both the National Academy of Sciences and the National Academy of Engineering in providing services to the government, the public, and the scientific and engineering communities. The Council is administered jointly by both Academies and the Institute of Medicine. Dr. Bruce M. Alberts and Dr. Wm. A. Wulf are chair and vice chair, respectively, of the National Research Council.

www.national-academies.org

Suggested Citation:"Front Matter." National Research Council. 2002. Reliability Issues for DOD Systems: Report of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/10561.
×

COMMITTEE ON NATIONAL STATISTICS 2002

JOHN E. ROLPH (Chair),

Marshall School of Business, University of Southern California

JOSEPH G. ALTONJI,

Department of Economics, Northwestern University

ROBERT BELL,

AT&T Laboratories, Florham Park, New Jersey

LAWRENCE D. BROWN,

Department of Statistics, University of Pennsylvania

ROBERT M. GROVES,

Survey Research Center, The University of Michigan

HERMANN HABERMANN,

Statistics Division, United Nations, New York

JOEL HOROWITZ,

Department of Economics, Northwestern University

WILLIAM D. KALSBEEK,

Survey Research Unit, Department of Biostatistics, University of North Carolina

ARLEEN LEIBOWITZ,

Department of Policy Studies, School of Public Policy, University of California at Los Angeles

RODERICK J.A. LITTLE,

School of Public Health, University of Michigan

THOMAS A. LOUIS,

Division of Biostatistics, University of Minnesota

DARYL PREGIBON,

AT&T Laboratories, Florham Park, New Jersey

NORA CATE SCHAEFFER,

Department of Sociology, University of Wisconsin-Madison

MATTHEW D. SHAPIRO,

Department of Economics, University of Michigan

ANDREW A. WHITE, Director

Suggested Citation:"Front Matter." National Research Council. 2002. Reliability Issues for DOD Systems: Report of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/10561.
×

Preface and Acknowledgments

The workshop with which this volume is concerned has a number of important historical antecedents. The Department of Defense (DoD) has long been keenly interested in quantitative analyses assessing the performance of the hardware and software used in its various activities. Its interest in the possibility of revisiting the statistical techniques emphasized in its handbooks and manuals since the early 1960s began in earnest with a DoD-sponsored workshop hosted by the National Academies in September 1992. That workshop served as the stimulus for a study by a panel of the Committee on National Statistics, National Research Council. The panel’s report, Statistics, Testing and Defense Acquisition: New Approaches and Methodological Improvements (1998) strongly advocates various forms of modernization in the statistical practices used in the various stages of the DoD acquisition process.

Two individuals within DoD played key roles in progressing to the essential next stage: active engagement between the statistical research community and DoD statisticians, engineers, and managers, with a view toward developing a common understanding of the agency’s current needs and the statistical research community’s most promising methods for addressing those needs. I wish to thank Nancy Spruill, Office of the Under Secretary of Defense for Acquisition, Technology, and Logistics, and Ernest Seglie, Science Advisor to the Director of Operational Test and Evaluation, for recognizing the urgency of the problem facing DoD, for raising the level of consciousness about that urgency within the department and among

Suggested Citation:"Front Matter." National Research Council. 2002. Reliability Issues for DOD Systems: Report of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/10561.
×

its own statistical contacts, and for their energetic work in developing the financial support for a new series of workshops on statistical topics of importance to DoD. In the workshop that is the subject of this report—the first in the series—they took a leading role in the identification of potential expert discussants from the defense community. Finally, Drs. Spruill and Seglie put the workshop organizers in touch with other scientists and administrators within DoD who served us as indispensable advisors on the organization and planning of the workshop. Among these, Allen Beckett, Dolores Etter, Hollis Hunter, Robert Nemetz, and Philip Rodgers merit special mention.

I would also like to acknowledge the very special contributions made by a distinguished group of statistical researchers who kindly accepted our invitation to make presentations at the workshop. Each of them took the time to engage in early discussions with DoD personnel working in the areas on which they would speak; this preliminary contact served to enhance both the relevance and the impact of their comments and contributions at the workshop. We gratefully acknowledge the contributions of each of the civilian presenters and discussants at the workshop: Wallace Blischke, University of Southern California; Jane Booker, Los Alamos National Laboratory; Frank Camm, the RAND Corporation; Larry Crow, General Dynamics; Siddhartha Dalal, Telcordia Technologies; Robert Easterling, Sandia National Laboratories; Donald Gaver, Naval Postgraduate School; William Meeker, Iowa State University; William Padgett, University of South Carolina; Stephen Pollock, University of Michigan; Jesse Poore, University of Tennessee; Ananda Sen, Oakland University; Samuel Saunders, Washington State University; Fritz Scholz, the Boeing Company; Duane Steffey, San Diego State University; and Michael Tortorella, Bell Laboratories. I also thank John Rolph and Bill Eddy of the Committee on National Statistics, who served with me as the organizing subcommittee for the workshop.

The workshop also benefitted greatly from extremely informative remarks by its defense community discussants: Allen Beckett, Office of the Under Secretary of Defense for Acquisition, Technology, and Logistics; James Crouch, Wright-Patterson Air Force Base; Paul Ellner, Army Materiel System Analysis Activity; Jack Ferguson, Software Intensive Systems; Arthur Fries, Institute for Defense Analyses; Walter Hollis, Office of the Deputy Under Secretary of the Army; Fred Myers, Office of the Under Secretary of Defense for Acquisition, Technology, and Logistics; Margaret Myers, Office of the Deputy Assistant Secretary of Defense; Ernest Seglie, Office of the Director of Operational Test and Evaluation; Nancy Spruill,

Suggested Citation:"Front Matter." National Research Council. 2002. Reliability Issues for DOD Systems: Report of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/10561.
×

Office of the Under Secretary of Defense for Acquisitions, Resources and Analysis; and Marion Williams, Air Force Operational Test and Evaluation Command.

Several invited guests came to our rescue by serving as session chairs: Asit Basu, University of Missouri-Columbia; Henry Block, University of Pittsburgh; Philip Boland, University College, Dublin; Ronald Glaser, Lawrence Livermore National Laboratory; Patricia Jacobs, Naval Postgraduate School; and Simon Wilson, Trinity College, Dublin.

This workshop report has been reviewed in draft form by individuals chosen for their diverse perspectives and technical expertise, in accordance with procedures approved by the Report Review Committee of the National Research Council. The purpose of this independent review is to provide candid and critical comments that will assist the institution in making its published report as sound as possible and to ensure that the report meets institutional standards for objectivity, evidence, and responsiveness to the charge. The review comments and draft manuscript remain confidential to protect the integrity of the process.

We thank the following individuals for their review of this report: Luis Escobar, Department of Statistics, Louisiana State University; Hans Mark, Department of Aerospace Engineering, University of Texas; Vijay Nair, Department of Statistics, University of Michigan; and Nozer Singpurwalla, Department of Operations Research, George Washington University.

Although the reviewers listed above provided many constructive comments and suggestions, they were not asked to endorse the content of the report nor did they see the final draft of the report before its release. The review of this report was overseen by William F. Eddy, Department of Statistics, Carnegie Mellon University. Appointed by the National Research Council, he was responsible for making certain that an independent examination of this report was carried out in accordance with institutional procedures and that all review comments were carefully considered. Responsibility for the final content of this report rests entirely with the authors and the institution.

I would also like to thank Arthur Fries, Institute for Defense Analyses, who contributed greatly to the workshop by helping to identify speakers and to orient their presentations so they were as compatible as possible with the workshop goals. In addition, Art carefully read several drafts of the present report.

Finally, I thank Michael Cohen, Agnes Gaskin, and Julia Kisa, staff of the Committee on National Statistics. Michael Cohen did a yeoman’s job

Page viii Cite
Suggested Citation:"Front Matter." National Research Council. 2002. Reliability Issues for DOD Systems: Report of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/10561.
×

as chief liaison between DoD and the Committee on National Statistics, and more than anyone else provided the impetus for the steady advance of both the scientific and practical issues associated with a successful workshop report. Michael developed the first draft of this report and has been a full partner with me since the beginning of this ambitious project. He is to be commended for his strong dedication to serving simultaneously the highest standards of the discipline and the most pressing needs of DoD. Agnes Gaskin, greatly assisted by Julia Kisa, handled all administrative details with great care and good humor. Finally, Eugenia Grohman managed the production of this report, including the enlistment of Rona Brier as technical editor, who served superbly in that role.

Francisco Samaniego, Chair

Workshop on Reliability Issues for Defense Systems

Suggested Citation:"Front Matter." National Research Council. 2002. Reliability Issues for DOD Systems: Report of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/10561.
×
Suggested Citation:"Front Matter." National Research Council. 2002. Reliability Issues for DOD Systems: Report of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/10561.
×
This page in the original is blank.
Suggested Citation:"Front Matter." National Research Council. 2002. Reliability Issues for DOD Systems: Report of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/10561.
×
Page R1
Suggested Citation:"Front Matter." National Research Council. 2002. Reliability Issues for DOD Systems: Report of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/10561.
×
Page R2
Suggested Citation:"Front Matter." National Research Council. 2002. Reliability Issues for DOD Systems: Report of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/10561.
×
Page R3
Suggested Citation:"Front Matter." National Research Council. 2002. Reliability Issues for DOD Systems: Report of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/10561.
×
Page R4
Suggested Citation:"Front Matter." National Research Council. 2002. Reliability Issues for DOD Systems: Report of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/10561.
×
Page R5
Suggested Citation:"Front Matter." National Research Council. 2002. Reliability Issues for DOD Systems: Report of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/10561.
×
Page R6
Suggested Citation:"Front Matter." National Research Council. 2002. Reliability Issues for DOD Systems: Report of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/10561.
×
Page R7
Page viii Cite
Suggested Citation:"Front Matter." National Research Council. 2002. Reliability Issues for DOD Systems: Report of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/10561.
×
Page R8
Suggested Citation:"Front Matter." National Research Council. 2002. Reliability Issues for DOD Systems: Report of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/10561.
×
Page R9
Suggested Citation:"Front Matter." National Research Council. 2002. Reliability Issues for DOD Systems: Report of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/10561.
×
Page R10
Next: 1. Introduction and Overview »
Reliability Issues for DOD Systems: Report of a Workshop Get This Book
×
Buy Paperback | $39.00 Buy Ebook | $31.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

The final report of the National Research Council's (NRC) Panel on Statistical Methods for Testing and Evaluating Defense Systems (National Research Council, 1998) was intended to provide broad advice to the U.S. Department of Defense (DoD) on current statistical methods and principles that could be applied to the developmental and operational testing and evaluation of defense systems. To that end, the report contained chapters on the use of testing as a tool of system development; current methods of experimental design; evaluation methods; methods for testing and assessing reliability, availability, and maintainability; software development and testing; and validation of modeling and simulation for use in operational test and evaluation. While the examination of such a wide variety of topics was useful in helping DoD understand the breadth of problems for which statistical methods could be applied and providing direction as to how the methods currently used could be improved, there was, quite naturally, a lack of detail in each area.

To address the need for further detail, two DoD agencies-the Office of the Director of Operational Test and Evaluation and the Office of the Under Secretary of Defense for Acquisition, Technology, and Logistics-asked the NRC's Committee on National Statistics to initiate a series of workshops on statistical issues relevant to defense acquisition. The aim of each workshop is to inform DoD about the methods that represent the statistical state of the art and, through interactions of the statistical and defense communities, explore their relevance for DoD application.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!