Optimizing U.S. Air Force and Department of Defense Review of Air Force Acquisition Programs (2009)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Optimizing U.S. Air Force and Department of Defense Review of Air Force Acquisition Programs Committee on Optimizing U.S. Air Force and Department of Defense Review of Air Force Acquisition Programs Air Force Studies Board Division on Engineering and Physical Sciences 

THE NATIONAL ACADEMIES PRESS   500 Fifth Street, N.W.   Washington, DC 20001 NOTICE: The project that is the subject of this report was approved by the Governing Board of the National Research Council, whose members are drawn from the councils of the National Academy of Sciences, the National Academy of Engineering, and the Institute of Medicine. The members of the committee responsible for the report were chosen for their special competences and with regard for appropriate balance. This is a report of work supported by Grant FA9550-08-1-0029 between the U.S. Air Force and the National Academy of Sciences. Any opinions, findings, conclusions, or recommen- dations expressed in this publication are those of the author(s) and do not necessarily reflect the view of the organizations or agencies that provided support for the project. International Standard Book Number-13:  978-0-309-13918-2 International Standard Book Number-10:  0-309-13918-X Limited copies of this report are Additional copies are available from: available from: Air Force Studies Board The National Academies Press National Research Council 500 Fifth Street, N.W. 500 Fifth Street, N.W. Lockbox 285 Washington, DC 20001 Washington, DC 20055 (202) 334-3111 (800) 624-6242 or (202) 334-3313 (in the Washington metropolitan area) Internet, http://www.nap.edu Copyright 2009 by the National Academy of Sciences. All rights reserved. Printed in the United States of America 

The National Academy of Sciences is a private, nonprofit, self-perpetuating society of distinguished scholars engaged in scientific and engineering research, dedicated to the furtherance of science and technology and to their use for the general welfare. Upon the authority of the charter granted to it by the Congress in 1863, the ­Academy has a ­mandate that requires it to advise the federal government on scientific and technical matters. Dr. Ralph J. Cicerone is president of the National Academy of Sciences. The National Academy of Engineering was established in 1964, under the charter of the National Academy of Sciences, as a parallel organization of outstanding ­ engineers. It is autonomous in its administration and in the selection of its members, sharing with the National Academy of Sciences the responsibility for advising the federal government. The National Academy of Engineering also sponsors engineering programs aimed at meeting national needs, encourages education and research, and recognizes the superior achievements of engineers. Dr. Charles M. Vest is president of the National Academy of Engineering. The Institute of Medicine was established in 1970 by the National Academy of Sciences to secure the services of eminent members of appropriate professions in the examination of policy matters pertaining to the health of the public. The Institute acts under the responsibil- ity given to the National Academy of Sciences by its congressional charter to be an adviser to the federal government and, upon its own initiative, to identify issues of medical care, research, and education. Dr. Harvey V. Fineberg is president of the Institute of Medicine. The National Research Council was organized by the National Academy of Sciences in 1916 to associate the broad community of science and technology with the Academy’s purposes of furthering knowledge and advising the federal government. Functioning in accordance with general policies determined by the Academy, the Council has become the principal operating agency of both the National Academy of Sciences and the National Academy of Engineering in providing services to the government, the public, and the scien- tific and engineering communities. The Council is administered jointly by both Academies and the Institute of Medicine. Dr. Ralph J. Cicerone and Dr. Charles M. Vest are chair and vice chair, respectively, of the National Research Council. www.national-academies.org 

COMMITTEE ON Optimizing U.S. Air Force and Department of Defense Review of Air Force Acquisition Programs Rand H. Fisher, Chair, The Aerospace Corporation J. Daniel Stewart, Vice Chair, University of Tennessee John A. Betti, Department of Defense (retired) Christopher L. Blake, Lockheed Martin Aeronautics claude m. bolton, jr., Defense Acquisition University Allan V. Burman, Jefferson Solutions John T. Dillard, U.S. Naval Postgraduate School charles e. franklin, Raytheon Company (retired) Charles L. Johnson II, Boeing Company LESLIE F. KENNE, LK Associates Andrew P. Sage, George Mason University Mark D. Schaeffer, ManTech SRS Technologies George R. Schneiter, Consultant Robert J. Skalamera, RJS Consulting Richard Szafranski, Toffler Associates randall s. weidenheimer, Northrop Grumman Mission Systems Rebecca A.Winston, Winston Strategic Management Consultants Staff JAMES C. GARCIA, Study Director KAMARA E. BROWN, Research Associate Zeida Patmon, Program Associate MARGUERITE SCHNEIDER, Administrative Coordinator ENITA A. WILLIAMS, Research Associate (Until October 2008) URRIKKA B. WOODS, Program Associate iv 

AIR FORCE STUDIES BOARD GREGORY S. MARTIN, Chair, GS Martin Consulting TAYLOR W. LAWRENCE, Vice Chair, Raytheon Company FRANK J. CAPPUCCIO, Lockheed Martin Corporation THOMAS DARCY, EADS North America Defense Company PAMELA A. DREW, Northrop Grumman Corporation KENNETH E. EICKMANN, U.S. Air Force (retired) JOHN V. FARR, Stevens Institute of Technology RAND H. FISHER, The Aerospace Corporation Michael J. Gianelli, Boeing Company (retired) JACQUELINE GISH, Northrop Grumman Corporation KENNETH C. HALL, Duke University WESLEY L. HARRIS, Massachusetts Institute of Technology LESLIE KENNE, LK Associates DONALD J. KUTYNA, U.S. Air Force (retired) Lester L. Lyles, The Lyles Group DEBASIS MITRA, Bell Laboratories Matt L. Mleziva, Wildwood Strategic Concepts CHANDRA KUMAR N. PATEL, Pranalytica, Inc. Gerald F. Perryman, Jr., Raytheon Company GENE W. RAY, GMT Ventures MARVIN R. SAMBUR, Headquarters, U.S. Air Force (retired) J. Daniel Stewart, University of Tennessee Staff MICHAEL A. CLARKE, Director JESSICA BROKENBURR, Financial Assistant KAMARA E. BROWN, Research Associate WILLIAM E. CAMPBELL, Senior Program Associate SARAH CAPOTE, Research Associate LISA COCKRELL, Senior Program Associate RESHAWN ELLIS, Senior Project Assistant (Until October 2008) GREGORY EYRING, Senior Program Officer ERIN C. FITZGERALD, Associate Program Officer CARTER W. FORD, Program Officer JAMES C. GARCIA, Senior Program Officer CHRIS JONES, Financial Manager  

Zeida Patmon, Program Associate MARGUERITE SCHNEIDER, Administrative Coordinator DANIEL E.J. TALMAGE, JR., Program Officer Shannon C. Thomas, Program Associate ENITA A. WILLIAMS, Research Associate (until October 2008) URRIKKA B. WOODS, Program Associate vi 

Preface The Deputy Assistant Secretary of the Air Force for Science, Technology, and Engineering asked the National Research Council (NRC) to review and assess the increasing number of prescribed program reviews and assessments that U.S. Air Force space and nonspace acquisition programs in all Department of Defense (DOD) acquisition categories (ACATs) are required to undergo and to recom- mend ways to improve the effectiveness and efficiency of those program reviews in terms of their goals, objectives, content, and requirements. The Committee on Optimizing U.S. Air Force and DOD Review of Air Force Acquisition Programs was formed in May 2008 to conduct this review. In developing its approach to the study, the committee concluded that the most important objective of the program reviews referred to above is to contrib- ute to the successful execution of acquisition programs. To this end, the com- mittee examined the substantial library of past studies; conducted a variety of interviews; surveyed managers of acquisition programs and program executive officers (PEOs); and constructed a matrix to summarize program reviews in terms of their purpose, target audience, and product output. Additionally, the committee is aware that substantial changes in acquisition policies and their implementation are being considered. The committee’s findings, conclusions, and recommendations address prin- cipally the subject of program reviews and also make some observations on other critical elements of acquisition success. The committee acknowledges and appreciates the contribution of the mem- bers of the Air Force Studies Board (AFSB) of the NRC for developing the study statement of task in concert with the Air Force sponsor. The AFSB was established in 1996 as a unit of the NRC at the request of the U.S. Air Force. The AFSB vii 

viii PREFACE brings to bear broad military, industrial, and academic scientific, engineering, and management expertise on Air Force technical challenges and other issues of importance to senior Air Force leaders. It discusses potential studies of interest, develops and frames study tasks, ensures proper project planning, suggests poten- tial committee members and reviewers for reports produced by fully independent ad hoc study committees, and convenes meetings to examine strategic issues. The board members, listed on page v, were not asked to endorse the committee’s conclusions or recommendations, nor did they review the final draft of this report before its release, although board members with appropriate expertise may be nominated to serve as formal members of study committees (as were Rand Fisher, Dan Stewart, and Leslie Kenne in this case) or as report reviewers. The committee thanks the many persons who helped provide information to the committee, including all the guest speakers shown in Appendix B, their organizations, and supporting staff members; all the Air Force program man­ agers and program executive officers who completed the committee’s survey; the Air Force sponsor of this study, Terry Jaggers, and his staff members, including Lt Col Ed Masterson and Lt Col Don Hill; and Kristen Baldwin in the Office of the Secretary of Defense. The committee is very grateful to Connie Citro, director of the NRC Committee on National Statistics, for her counsel in developing the committee’s survey of Air Force program managers and in evaluating its results. The committee is also grateful to the NRC staff members who provided support throughout the study. Finally, as chair and vice chair of the study committee, we extend special thanks to the committee members for their commitment and diligence, which enabled us to complete the task successfully. Rand H. Fisher, Chair J. Daniel Stewart, Vice Chair Committee on Optimizing U.S. Air Force and Department of Defense Review of Air Force Acquisition Programs 

Acknowledgment of Reviewers This report has been reviewed in draft form by individuals chosen for their diverse perspectives and technical expertise, in accordance with procedures approved by the National Research Council’s Report Review Committee. The purpose of this independent review is to provide candid and critical comments that will assist the institution in making its published report as sound as possible and to ensure that the report meets institutional standards for objectivity, evidence, and responsiveness to the study charge. The review comments and draft manuscript remain confidential to protect the integrity of the deliberative process. We wish to thank the following individuals for their review of this report: Edward C. Aldridge, Jr., NAE, U.S. Department of Defense (retired), Meyer J. Benzakein, NAE, Ohio State University, Andrew Brown, Jr., NAE, Delphi Corporation, David E. Crow, NAE, University of Connecticut, Susan E. Dong, BAE Systems, James H. Frey, Frey Associates, Peter Hantos, Aerospace Corporation, Donald A. Lamontagne, Star Mountain Consulting, Inc., Robert H. Latiff, Science Applications International Corporation, and Robert R. Shannon, NAE, University of Arizona. Although the reviewers listed above provided many constructive comments and suggestions, they were not asked to endorse the conclusions or recommenda- tions, nor did they see the final draft of the report before its release. The review ix 

 ACKNOWLEDGMENT OF REVIEWERS of this report was overseen by Robert A. Frosch, NAE, Harvard University, and G. Brian Estes, U.S. Navy (retired). Appointed by the National Research Council, they were responsible for making certain that an independent examination of this report was carried out in accordance with institutional procedures and that all review comments were carefully considered. Responsibility for the final content of this report rests entirely with the authoring committee and the institution. 

Contents SUMMARY 1 1 introduction 5 Statement of Task, 5 Background, 5 Methodology, 13 Presentations, 13 Literature Review, 14 Survey and Other Data, 14 Comparative Matrix, 14 Integration and Synthesis of Data, 15 2 findings and Conclusions 16 Reviews Have Both Benefits and Costs, 23 Execution of Reviews (Sequencing, Timing, Participation), 24 Program Review Planning, 27 3 RECOMMENDATIONS 30 Implementing the Recommendations, 35 xi 

xii CONTENTS APPENDIXES A Biographical Sketches of Committee Members 41 B Meetings and Speakers 51 C Related Studies 55 D Survey 58 E Types of Reviews 88 

Acronyms A5 Headquarters Air Force Deputy Chief of Staff for Air, Space, Information Operations, Plans and Requirements (A3/5) A5R Headquarters Air Force Deputy Chief of Staff for Air, Space, Information Operations, Plans and Requirements (A3/5), Director of Operational Capability Requirements AAC Air Armament Center ACAT acquisition category ACE acquisition center of excellence ADM acquisition decision memorandum AF Air Force AFAA Air Force Audit Agency AFCSE Air Force Center for Systems Engineering AFMC Air Force Materiel Command AFRB Air Force Review Board AFROCC Air Force Requirements for Operational Capability Council AFSB Air Force Studies Board AFSO21 Air Force Smart Operations for the 21st Century AOTR assessment of operational test readiness AP acquisition plan APB acquisition program baseline ASC Aeronautical Systems Center ASD (NII) Assistant Secretary of Defense for Networks and Information Integration ASP acquisition strategy panel ASR alternative system review xiii 

xiv ACRONYMS CD concept decision CDR critical design review CPI continuous process improvement CSAF Chief of Staff of the Air Force CSB configuration steering board CSP cost, schedule, performance DAB Defense Acquisition Board DAE Defense Acquisition Executive DAES Defense Acquisition Executive Summary DAPS Defense Acquisition Program Support DASD (NII) Deputy Assistant Secretary of Defense for Networks and Information Integration DAU Defense Acquisition University DDR&E Director of Defense Research and Engineering DOD Department of Defense DOD IG Department of Defense Inspector General DOT&E Director of Operational Test and Evaluation DRR design readiness review DSAB Defense Space Acquisition Board D&SWS develop and sustain warfighting systems DT&E development, test, and evaluation DUSD (A&T) Deputy Under Secretary of Defense for Acquisition and Technology EDT Eastern daylight time ESC Electronic Systems Center FCA functional configuration audit FOA field operating agency FRP full rate production GAO Government Accountability Office HQ headquarters IBR integrated baseline review IIPT integrating integrated product team IPA independent program assessment IPT integrated product team J-8 Joint Chiefs of Staff Force Structure, Resources, and Assessment Directorate 

ACRONYMS xv JAT joint assessment team JCIDS Joint Capabilities Integration and Development System JCS Joint Chiefs of Staff JROC Joint Requirements Oversight Council JROCM JROC memorandum KDP key decision point KP knowledge point KTR contractor LHA logistics health assessment MAJCOM major command MDA milestone decision authority MILSATCOM military satellite communications MRA manufacturing readiness assessment MRL manufacturing readiness level MS milestone NAS National Academy of Sciences NII/AT&L Networks and Information Integration/Acquisition, Technology, and Logistics NRC National Research Council OC2 oversight, command, and control OIPT overarching integrated product team OPR office of primary responsibility OSD Office of the Secretary of Defense OT operational test OT&E operational test and evaluation OTRR operational test readiness review PCA physical configuration audit PCDRA post-CDR assessment PDR preliminary design review PEO program executive officer PEO/SR program executive officer sufficiency review Perf performance PM program manager PO Program Office PRR production readiness review PSA Office of Portfolio Systems Acquisition PSR program support review 

xvi ACRONYMS RADM rear admiral RFP request for proposals SAE Service Acquisition Executive SDD system development and demonstration SDO Services Designated Official SDR system design review SE system engineering SEAM systems engineering assessment model SECAF Secretary of the Air Force SecDef Secretary of Defense SES Senior Executive Service SFR system functional review SMC Space and Missile Systems Center SME subject matter expert SRR system requirements review SSE Software and Systems Engineering S&T Science and Technology ST&E Science, Technology, and Engineering SVR system verification review TD technology development TRA technology readiness assessment TRR test readiness review TSAT Transformational Satellite Communications USAF United States Air Force USD (AT&L) Under Secretary of Defense for Acquisition, Technology, and Logistics VCJCS Vice Chairman, Joint Chiefs of Staff