Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
- S Oversight Committee for the Workshop on Statistical Methods in Sohware Engineering for Defense Systems Siddhartha R. Dalal, Jesse H. Poore, and Michael L. Cohen, editors Committee on National Statistics Division of Behavioral and Social Sciences and Education and Committee on Applied and Theoretical Statistics Division on Engineering and Physical Sciences NATIONAL RESEARCH COUNCIL OF THE NATIONAL ACADEMIES THE NATIONAL ACADEMIES PRESS Washington, D.C. www.nap.edu
THE NATIONAL ACADEMIES PRESS · 500 Fifth Street, NW · Washington, DC 20001 NOTICE: The project that is the subject of this report was approved by the Governing Board of the National Research Council, whose members are drawn from the councils of the National Academy of Sciences, the National Academy of Engineering, and the Institute of Medicine. The members of the committee responsible for the report were chosen for their special competences and with regard for appropriate balance. This material is based upon work supported by the National Science Foundation under Grant No. SBR-9709489. Any opinions, findings, conclusions, or recommendations expressed in this publication are those of the authorLs) and do not necessarily reflect the views of the National Science Foundation. International Standard Book Number 0-309-08983-2 (Book) International Standard Book Number 0-309-52631-0 (PDF) Library of Congress Control Number 2003110750 Additional copies of this report are available from National Academies Press, 500 Fifth Street, N.W., Lockbox 285, Washington, DC 20055; (800) 624-6242 or (202) 334- 3313 (in the Washington metropolitan area); Internet, http://www.nap.edu. Printed in the United States of America. Copyright 2003 by the National Academy of Sciences. All rights reserved. Suggested citation: National Research Council. (2003~. Innovations in Software Engi- neeringforDefense Systems. Oversight Committee for the workshop on Statistical Meth- ods in Software Engineering for Defense Systems. S.R. Dalal, ).H. Poore, and M.L. Cohen, eds. Committee on National Statistics, Division of Behavioral and Social Sci- ences and Education. Washington, DC: The National Academies Press.
THE NATIONAL ACADEMIES Advisers to the Nation on Stienre, Engineering, and Medicine The National Academy of Sciences is a private, nonprofit, self-perpetuating society of distinguished scholars engaged in scientific and engineering research, dedicated to the furtherance of science and technology and to their use for the general welfare. Upon the authority of the charter granted to it by the Congress in 1863, the Academy has a mandate that requires it to advise the federal government on scientific and technical matters. Dr. Bruce M. Alberts is president of the National Academy of Sciences. The National Academy of Engineering was established in 1964, under the charter of the National Academy of Sciences, as a parallel organization of outstanding engineers. It is autonomous in its administration and in the selection of its members, sharing with the National Academy of Sciences the responsibility for advising the federal govern- ment. The National Academy of Engineering also sponsors engineering programs aimed at meeting national needs, encourages education and research, and recognizes the superior achievements of engineers. Dr. Wm. A. Wulf is president of the National Academy of Engineering. The Institute of Medicine was established in 1970 by the National Academy of Sciences to secure the services of eminent members of appropriate professions in the examination of policy matters pertaining to the health of the public. The Institute acts under the responsibility given tO the National Academy of Sciences by its congressional charter to be an adviser to the federal government and, upon its own initiative, to identify issues of medical care, research, and education. Dr. Harvey V. Fineberg is president of the Institute of Medicine. The National Research Council was organized by the National Academy of Sciences in 1916 to associate the broad community of science and technology with the Academy's purposes of furthering knowledge and advising the federal government. Functioning in accordance with general policies determined by the Academy, the Council has become the principal operating agency of both the National Academy of Sciences and the National Academy of Engineering in providing services to the govern- ment, the public, and the scientific and engineering communities. The Council is administered jointly by both Academies and the Institute of Medicine. Dr. Bruce M. Alberts and Dr. Wm. A. Wulf are chair and vice chair, respectively, of the National Research Council. www. nationa l-academies.org
OVERSIGHT COMMITTEE FOR THE WORKSHOP ON STATISTICAL METHODS IN SOFTWARE ENGINEERING FOR DEFENSE SYSTEMS DARYL PREGIBON (Chair9, AT&T Laboratories, Florham Park, New Jersey BARRY BOEHM, Computer Science Department, University of Southern California SIDDHARTHA R. DALAL, Xerox Corporation, Webster, New York WILLIAM F. EDDY, Department of Statistics, Carnegie Mellon University JESSE H. POORE, Department of Computer Science, University of Tennessee, Knoxville JOHN E. ROLPH, Marshall School of Business, University of Southern California FRANCISCO l. SAMANIEGO, Division of Statistics, University of California, Davis ELAINE WEYUKER, AT&T Laboratories, Florham Park, New Jersey MICHAEL L. COHEN, Staidly Director MICHAEL SIRI, Project Assistant V
COMMITTEE ON NATIONAL STATISTICS 2002-2003 JOHN E. ROLPH (Chair9, Marshall School of Business, University of Southern California JOSEPH G. ALTONII, Department of Economics, Yale University ROBERT M. BELL, AT&T Laboratories, Florham Park, New Jersey LAWRENCE D. BROWN, Department of Statistics, University of Pennsylvania ROBERT M. GROVES, Survey Research Center, University of Michigan JOEL L. HOROWITZ, Department of Economics, Northwestern University WILLIAM D. KALSBEEK, Department of Biostatistics, University of North Carolina ARLEEN LEIBOWITZ, School of Public Policy Research, University of California, Los Angeles THOMAS A. LOUIS, Department of Biostatistics, Johns Hopkins University VIIAYAN NAIR, Department of Statistics, University of Michigan DARYL PREGIBON, AT&T Laboratories, Florham Park, New Jersey KENNETH PREWITT, School of International and Public Affairs, Columbia University NORA CATE SCHAEFFER, Department of Sociology, University of Wisconsin, Madison MATTHEW D. SHAPIRO, Department of Economics, University of Michigan ANDREW A. WHITE, Director v
COMMITTEE ON APPLIED AND THEORETICAL STATISTICS 2002-2003 SALLIE KELLER-McNULTY (Chair9, Los Alamos National Laboratory SIDDHARTHA DALAL, Xerox Corporation, Webster, New York CATHRYN DIPPO, Consultant, Washington, DC THOMAS KEPLER, Duke University Medical Center, Durham, North Carolina DANIEL L. McFADDEN, University of California, Berkeley RICHARD OLSHEN, Stanford University DAVID SCOTT, Rice University, Houston, Texas DANIEL L. SOLOMON, North Carolina State University EDWARD C. WAYMIRE, Oregon State University, Corvallis, Oregon LELAND WILKINSON, SPSS, Incorporated, Chicago SCOTT T. WEIDMAN, Director v'
Acknowledgments The Workshop on Statistical Methods in Software Engineering for Defense Systems, jointly sponsored by the Committee on National Statistics and the Committee on Applied and Theoretical Statistics of the National Research Council (NRC), grew out of a need to more fully examine issues raised in the NRC reports Statistics, Testing, and Defense Acquisition: New Approaches and Methodological Improvements (NRC, 1998) and Reliability Iss?~esfor Defense Systems: Report of a Workshop (NRC, 20021. The former stressed, among other things, the importance of software re- quirements specification, and the latter demonstrated the benefits gained from model-based testing. Concurrent with these activities, there was an increasing recognition in the Office of the Secretary of Defense that soft- ware problems were a major source of defects, delays, and cost overruns in defense systems in development, and that approaches to addressing these problems were very welcome. The Workshop on Statistical Methods in Software Engineering for Defense Systems, which took place July 19-20, 2001, was funded jointly by the offices of the Director of Operational Test and Evaluation (DOT&E) and the Undersecretary of Defense for Acquisition, Technology, and Logis- tics (OUSD(AT&L)~. DOT&E, headed by Tom Christie, was ably repre- sented at various planning meetings by Ernest Seglie, the science advisor for DOT&E, and OUSD (AT&L) was represented by Nancy Spruill. Both Ernest Seglie and Nancy Spruill were enormously helpful in suggesting topics for study, individuals to serve as expert defense discussants, and in- . . v''
- - - vIll ACKNOWLEDGMENTS vised guests to supplement floor discussion. Their contribution was essen- tial to the success of this workshop. We would also like to thank the following Office of the Secretary of Defense employees for their assistance: Walter Benesch, Michael Falat, lack Ferguson, Ken Hong Fong, Liz Rodriguez Johnson, Steven King, Bob Nemetz, Rich Turner, and Bob Williams. We are very grateful to Norton Compton for his help on a number of administrative issues and his can-do spirit. Scott Weidman, staff director of the NRC's Committee on Applied and Theoretical Statistics, provided very helpful advice in structuring the workshop. We would also like to thank Art Fries of the Institute for De- fense Analysis, Dave Zubrow of the Software Engineering Institute, and David Nicholls of IIT Research Institute for their assistance. The Oversight Committee for the Workshop on Statistical Methods in Software Engineering for Defense Systems met and interacted only infor- mally, often via telephone and e-mail, and in person in small groups of two and three. It relied to a great extent on the content of the presentations given at the workshop. Therefore, the Committee would like to express its gratitude to the speakers and the discussants: Frank Apicella, Army Evalua- tion Center, Vic Basili, University of Maryland, Patrick Carnes, Air Force Operational Test and Evaluation Center, Ram Chillarege, consultant, Tom Christian, WR-ALC, Delores Etter, Office of the Secretary of Defense, William Farr, Naval Surface Warfare Center, lack Ferguson, Office of the Undersecretary of Defense for Science and Technology, lanes Gill, PAX River Naval Air Systems, Software Safety, Constance Heitmeyer, Naval Re- search Lab, Brian Hicks, loins Advanced Strike Technology, Mike Houghtaling, IBM, Jerry Huller, Raytheon, Ashish Jain, Telcordia Tech- nologies, Stuart Knoll, loins Staff, l-8, Scott Lucero, Army SW Metrics, Army Evaluation Center, Luqi, Naval Postgraduate School, Ron Manning, Raytheon, David McClung, University of Texas, Margaret Myers, Office of the Assistant Secretary of Defense for Command, Control, Communica- tions and Intelligence, km O'Bryon, Office of the Director of Operational Test and Evaluation, Lloyd Pickering, Army Evaluation Center, Stacy l. Prowell, University of Tennessee, Manish Rathi, Telcordia Technologies, Harry Robinson, Microsoft, Mike Saboe, Tank Automotive and Armaments Command, Ernest Seglie, Office of the Director of Operational Test and Evaluation, Linda Sheckler, Penn State University, Applied Research Lab, KishorTrivedi, Duke University, Amjad Umar, Telcordia Technologies, and Steven Whitehead, Operational Test and Evaluation Force Technical Advi-
ACKNOWLEDGMENTS AX sor. Further, we would like to give special thanks to Constance Heitmeyer and Amjad Umar for their help in writing sections of the report. We would also like to thank Michael Cohen for help in organizing the workshop and in drafting sections of the report, and Michael Siri for his excellent attention to all administrative details. We would also like to thank Cameron Fletcher for meticulous technical editing of the draft report, catching numerous errors. This report has been reviewed in draft form by individuals chosen for their diverse perspectives and technical expertise, in accordance with proce- dures approved by the Report Review Committee of the NRC. The pur- pose of this independent review is to provide candid and critical comments that will assist the institution in making the published report as sound as possible and to ensure that the report meets institutional standards for ob- jectivity, evidence, and responsiveness to the study charge. The review comments and draft manuscript remain confidential to protect the integ- rity of the deliberative process. We thank the following individuals for their participation in the re- view of this report: Robert M. Bell, AT&T Research Laboratories, Florham Park, NJ, Philip E. Coyle, III, Consultant, Los Angeles, CA, Michael A. Houghtaling, IBM/ Systems Group/ Storage Systems Division, Tucson, AZ, and Lawrence Markosian, QSS Group, Inc., NASA Ames Research Center, Moffett Field, CA. Although the reviewers listed above have provided many constructive comments and suggestions, they were not asked to endorse the conclusions or recommendations nor did they see the final draft of the report before its release. The review of this report was overseen by William G. Howard, fir., Independent Consultant, Scottsdale, AZ. Appointed by the NRC, he was responsible for making certain that an independent ex- amination of this report was carried out in accordance with institutional procedures and that all review comments were carefully considered. Re- sponsibility for the final content of this report rests entirely with the authoring committee and the institution. Daryl Pregibon, Chair Workshop on Statistical Methods in Software Engineering for Defense Systems
Contents EXECUTIVE SUMMARY 1 MOTIVATION FOR AND STRUCTURE OF THE WORKSHOP Structure of the Workshop, 6 Workshop Limitations, 10 2 REQUIREMENTS AND SOFTWARE ARCHITECTURAL ANALYSIS Software Cost Reduction, 13 Sequence-Based Software Specification, 16 TESTING METHODS AND RELATED ISSUES Introduction to Model-Based Testing, 20 Markov Chain Usage Models, 24 AETG Testing, 29 Integration of AETG and Markov Chain Usage Models, 32 Test Automation, 33 Methods for Testing Interoperability, 34 x' 1 5 12 20
xI' 4 DATA ANALYSIS TO ASSESS PERFORMANCE AND TO SUPPORT SOFTWARE IMPROVEMENT Measuring Software Risk, 40 Fault-Tolerant Software: Measuring Software Aging and Rejuvenation, 44 Defect Classification and Analysis, 45 Bayesian Integration of Project Data and Expert Judgment in Parametric Software Cost Estimation Models, 48 NEXT STEPS REFERENCES APPENDICES A WORKSHOP AGENDA AND SPEAKERS B GLOSSARY AND ACRONYM LIST C BIOGRAPHICAL SKETCHES CONTENTS 40 51 56 61 66 70