Pre-Milestone A and Early-Phase Systems Engineering: A Retrospective Review and Benefits for Future Air Force Systems Acquisition (2008)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

1 Introduction and Overview The time required to execute large, government-sponsored systems devel- opment programs has more than doubled over the past 30 years, and the cost growth has been at least as great. Many causes for this trend have been sug- gested, including the increased complexity of the systems involved; instability of external ­ funding; loss of “mission urgency” after the end of the Cold War; diminished depth of talent in the government and contractor community; require- ments creep; the need to satisfy the demands of an increasingly diverse user community; ­inadequate up-front project planning; lack of management oversight, accountability, and clear metrics on both the government and contractor sides; and the exponential growth in, and reliance on, software. Nevertheless, this trend is particularly puzzling given the enormous productivity advantages conferred by the advent of the Internet and e-mail, the revolution in electronics and computer technology, and advances in knowledge-management and collaboration tools such as computer-aided design (CAD), computer-aided manufacturing (CAM), computer-aided software engineering (CASE), and modeling and simulation. During World War II and the early Cold War, programs such as the ­Manhattan Project, the Defense Support Program (DSP), the intercontinental ballistic mis- sile, and the U-2 surveillance aircraft all delivered very quickly, generally in fewer than 6 years, first products that today would be described as major systems. Currently, such major programs would likely require 10 to 20 years to complete. System complexity has grown dramatically, and products are not delivered under the same technological, human, and organizational guidelines as before. There has been about a threefold increase in delivery time for most major systems. Table 1-1 summarizes some well-known examples comparing historical program accomplishments with those of more recent programs. Table 1-2 shows some 14 

INTRODUCTION AND OVERVIEW 15 TABLE 1-1  Representative Development Times of Major Historical and Recent Programs Years to First Use from Program and Year of First Use Contractor Selection Historical programs Manhattan Project (1945) 2½ Defense Support Program (1970) 5½ Intercontinental Ballistic Missile (1958) 3½ Apollo (1967) 8 F-104 (1958) 5 SR-71 (1962) 3 Recent programs Future Imagery Architecture-Optical 13 (projected when canceled) Space Based Infrared Systems/Boost Surveillance and >20 Tracking System (to be determined) B-2 bomber (1993) 11 Joint Strike Fighter (to be determined) ≈13 F-22 (2005) 14 TABLE 1-2  Cost and Schedule Outcomes Sorted by Percentage of Product Development Remaining Development Cost Growth Schedule Growth Remaining Program (Percent)a (Months) (Percent) Aerial Common Sensor 45 24 85 Future Combat System 48 48 78 Joint Strike Fighter 30 23 60 Expeditionary Fighting Vehicle 61 48 49 C-130 Avionics Modernization 122 Delays anticipated Undetermined Global Hawk (RQ-4B) 166 Delays anticipated Undetermined aCost growth is expressed as the percentage change in program development cost estimates in 2005 base-year dollars. SOURCE: Reprinted from Government Accountability Office, 2006, Major Weapon Systems Continue to Experience Cost and Schedule Problems Under DOD’s Revised Policy, GAO-06-368, April. Avail- able at http://www.gao.gov/highlights/d06368high.pdf. Last accessed April 2, 2007. modern programs to further illustrate the current trend toward increasing cost and time to deployment. In an effort to develop consistent policies and methodologies to address cost and schedule overruns, the Department of Defense (DOD) has published numer- ous policies, undertaken studies, and developed several guidebooks such as the   See Defense Science Board, 2007, 21st Century Strategic Technology Vectors, Vol. I: Accelerating the Transition of Technologies into US Capabilities, April, Washington, D.C.: OUSD (AT&L). 

16 PRE-MILESTONE A AND EARLY-PHASE SYSTEMS ENGINEERING 5000 series, a Systems of Systems Guidebook, and the Systems Engineering Plan Guidebook. The individual services and intelligence agencies have also published policies and guides to supplement the DOD policies and to develop service- and agency- specific processes. For example, the National Reconnaissance Office (NRO) has made attempts over the past few years to address some of the acquisition dif- ficulties that it has experienced. It has emphasized a more rigorous adherence to milestone decision gates and has made the extensive use of independent reviews of program readiness a necessary step before proceeding to the next phase of a program. It has also modified its acquisition schedules to align major decisions more closely with the results of major design reviews, and mandated more fre- quent post-Milestone C reviews by the decision authority. On a more technical level, the NRO, in cooperation with its industry team members, has reinstituted a minimum essential set of specifications and standards on such diverse topics as systems engineering (SE) and the qualification of key components. Yet despite the myriad of new and revised processes throughout government acquisition organizations, there is little sign that performance is returning to the development productivity that was achieved decades ago. Indeed, one is tempted to conclude that performance diminishes as procurement organizations mature and their processes become more complex. This is counter to the trend in the private sector, where automobiles, commercial aircraft, commercial spacecraft, and consumer electronics have experienced 50 to 70 percent reductions in cycle times. Recent studies done by the Government Accountability Office (GAO) have expressed continuing concern about program cost and schedule growth problems, even under the revised policies being promulgated by the DOD. As the GAO stated in 2006: Changes made in DoD’s acquisition policy over the past 5 years have not elimi- nated cost and schedule problems for major weapons development programs. Of the 23 major programs we assessed, 10 are already expecting development cost overruns greater than 30 percent or have delayed the delivery of initial opera- tional capability to the warfighter by at least 1 year. The overall impact of these costly conditions is a reduction in the value of DoD’s defense dollars and a lower return on investment. Poor execution of the revised acquisition policy is a major cause of DoD’s continued problems. The DoD frequently bypasses key steps of the knowledge-based process outlined in the policy, falls short of attaining key knowledge, and continues to pursue revolutionary—rather than evolutionary or incremental—advances in capability. Nearly 80 percent of the programs GAO reviewed did not fully follow the knowledge-based process to develop a sound   Defense Acquisition University, 2007, Systems and Software Engineering Publications and Docu- ments. Available from http://www.acq.osd.mil/se/publications.htm. Last accessed on May 2, 2007.   See Defense Science Board, 2007, 21st Century Strategic Technology Vectors, Vol. I: Accelerating the Transition of Technologies into US Capabilities, April, Washington, D.C.: OUSD (AT&L). 

INTRODUCTION AND OVERVIEW 17 business case before committing to systems development. Most of the programs we reviewed started system development with immature technologies, and half of the programs that have held design reviews did so before achieving a high level of design maturity. These practices increase the likelihood that problems will be discovered late in development when they are more costly to address. Furthermore, DoD’s continued pursuit of revolutionary leaps in capability also runs counter to the policy’s guidance. The DoD has not closed all of the gaps in the policy that GAO identified nearly 3 years ago, particularly with regard to adding controls and criteria. Effective controls require decision makers to measure progress against specific criteria and ensure that managers capture key knowledge before moving to the next acquisition phase. However, DoD’s policy continues to allow managers to approach major investment decisions with many unknowns. Without effective controls that require program officials to satisfy specific criteria, it is difficult to hold decision makers or program managers accountable to cost and schedule targets. In this environment, decision-making transparency is crucial, but DoD is lacking in this area as well.  The Air Force and DOD are concerned about the impact that this trend is having in terms of the cost of fielding new systems, the erosion of spending power, and perhaps more importantly, the loss of agility to respond to rapidly changing threats. As suggested above, programs may fail or exhibit cost and schedule overruns for many reasons. Some of these are external to the program, such as funding instability; others are internal to the program and thus under the control of DOD managers. Two critical factors in the success or failure of programs that fall in the latter category are the need for high-quality systems engineering and the related issue of the need for a high-quality systems engineering workforce. These success factors are the focus of this report and are described further below. SYSTEMS ENGINEERING The complexity of systems that humans choose to build, manage, and control continues to grow, with no sign of letting up—outpacing the ability of engineers to develop processes and tools to manage their development. The committee considered the degree to which growth in complexity is responsible for the before-mentioned increases in the cost and time required to deploy new systems. It found a number of legacy programs that appear to be as complex as or more complex than follow-on programs developed more recently at much greater cost and time. The early imaging satellites and the DSP are examples of such early successful programs, while the Space Based Infrared Systems (SBIRS) program (DSP follow-on) has experienced numerous cost and schedule overruns (see   Government Accountability Office, 2006, Major Weapon Systems Continue to Experience Cost and Schedule Problems Under DOD’s Revised Policy, GAO-06-368, April. Available at http://www. gao.gov/highlights/d06368high.pdf. Last accessed on April 2, 2007. 

18 PRE-MILESTONE A AND EARLY-PHASE SYSTEMS ENGINEERING Chapter 2). The Apollo program in the 1960s, arguably one of the most complex space programs ever, took fewer than 8 years to complete. But in one respect the complexity of most large systems today seems to be much greater, and that is in the complexity of the missions that the systems are asked to serve and in the number and diversity of users, supporters, and admin- istrators of the systems. Indeed, it is often the increased complexity of external interfaces, more than internal system design complexity, that is the cause of extended development times and costs. Software-intensive systems represent a special challenge because of the myriad of possible logic paths that can be woven through their codes. As Moore’s law continues to drive down the size of computers and drive up their speed and capability, functionality that was once deeply embedded in the physical configu- ration of components has begun to emerge as software, enabling synergies among components that would have been unimaginable only a few years ago. The successful design, manufacture, and operation of these complex systems demands an engineering discipline capable of comprehending and managing all of their components and their interactions, and that discipline is called systems engineering. Simply stated, SE is the translation of a user’s needs into a definition of the system and its architecture through an iterative process that results in an effective system design. Systems engineering was born in the telecommunications industry of the 1940s and nurtured by the challenges of World War II, when project managers and chief engineers, with the assistance of key subsystem leads, oversaw the development of aircraft, ships, and other weapons systems. The post-World War II creation of more complex systems—for example, ballistic missiles and communi- cation systems—led to the formalization of SE as an engineering discipline. The development teams, especially for large weapons systems, employed thousands of engineers and required the use of formal methods to integrate subsystems into useful and reliable systems. Today the profession of SE is fairly well evolved. SE has experienced tre- mendous growth and recognition within the academic world and has a strong professional society advocate (the International Council on Systems Engineer- ing, or INCOSE). Most importantly, SE has been recognized within industry as a profession that is critical to the development of complex systems. As noted by Blanchard and Fabrycky, systems engineering is good engineer- ing with the following areas emphasized: • A top-down approach is required, viewing the system as a whole. Although engineering activities in the past have very adequately covered the design of various system components, the necessary overview and an understand-   A more rigorous and richer discussion of SE is found in Appendix C of this report.   Benjamin Blanchard and Wolter Fabrycky, 2005, Systems Engineering and Analysis (4th Edition), Englewood Cliffs, N.J.: Prentice Hall. 

INTRODUCTION AND OVERVIEW 19 ing of how these components effectively fit together has not always been present. • A life cycle orientation is required, addressing all phases, including sys- tem design and development, production and/or construction, distribu- tion operation, sustaining maintenance and support, and retirement and m ­ aterial phaseout. Emphasis in the past has been placed primarily on system design activities, with little consideration given to their impact on production, operations, support, and disposal. • A better and more complete effort is required relative to the initial iden- tification of system requirements, relating these requirements to specific design goals, the development of appropriate design criteria, and the f ­ ollow-on analysis to ensure the effectiveness of early decision making in the design process. A common illustration of the SE framework is the “Vee” model shown in Figure 1-1. The SE process begins at the upper left with the definition of user Systems Engineering Feedback Gather & Define Operational Requirements Acceptance System Integration System Design & Verification De com te gra De & e ify po fin Ver& Component Test e Detailed Design Int se Component Engineering Fabricate & Code FIGURE 1-1  The “Vee” model of systems engineering. This model is generally attrib- uted to the National Aeronautics and Space Administration, which in 1988 saw a benefit in bending the waterfall model into the “V” shape for software development. SOURCE: Modified from K. Forsberg and H. Mooz, 1992, “The Relationship of Systems Engineering to the Project Life Cycle,” Engineering Management Journal 4(3):36-43. Copyright 1992 by IEEE. Reprinted by permission from IEEE. fig 1-1 

20 PRE-MILESTONE A AND EARLY-PHASE SYSTEMS ENGINEERING requirements and of system concepts that meet those requirements. It continues down through system design and fabrication, then up through testing, integration, verification, and delivery of a product. Since SE encompasses the entire system life cycle, many SE diagrams continue to the right with segments representing system upgrades, maintenance, repair, and finally, disposal. DEPARTMENT OF DEFENSE ACQUISITION PROCESS Figure 1-2 provides an illustration of system development similar to that represented in Figure 1-1, but this time in the language of the DOD acquisition process. Across the top of Figure 1-2 are the points at which important management decisions are made—Milestones A, B, and C. Concept development and refine- ment occur before Milestone A, and further technology development to flesh out the concept occurs before Milestone B. Only after Milestone B does a program become an enterprise with dedicated funding behind it. The nature of systems engineering changes significantly after Milestone B. Pre-Milestone A, systems A B C Concept Technology System Development Production & Operations Refinement Development & Demonstration Deployment & Support System Life Cycle Acquisition Process Combat Developer Materiel Developer PM —Total Life Cycle System Manager Air Force Materiel Command Acquisition Framework High Less Ability to Ability to Influence LCC Influence (85% of Cost LCC Decisions Little Ability to (70-75% Made) Influence LCC (90-95% of Cost of Cost Decisions Minimum Ability to Influence LCC (95% of Cost Decisions (10%-15%) Made) Decisions Made) Made) (5%-10%) 28% Life Cycle Cost 72% Life Cycle Cost FIGURE 1-2  DOD life cycle acquisition process. Points A, B, and C at the top of the figure represent Milestones A, B, and C. LCC, life cycle cost. SOURCE: ­ Richard An- drews, 2003, An Overview of Acquisition Logistics, Fort Belvoir, Va.: Defense Acquisi- tion University. Available at http://www.afcea.org/events/pastevents/documents/Track4 1-2 Session4AMCEmphasisonCustomerFocusedITInitiatives.ppt#364,12,Slide 12. Last ac- cessed on November 20, 2007. 

INTRODUCTION AND OVERVIEW 21 engineering involves translating the needs of the user into clearly stated key per- formance parameters (KPPs) and evolving the system concept and preliminary concept of operations (CONOPS) that satisfy these needs. After Milestone B, the emphasis shifts to the flowdown of requirements, interface management, per- formance prediction, system verification, and change control. The use of formal systems engineering practices throughout the life cycle of an acquisition program is critical to fielding the required system on time and within budget. Importantly, Figure 1-2 shows that about three-quarters of the total system life cycle costs are influenced by decisions made before the end of the concept refinement phase at Milestone A, while about three-quarters of life cycle funds are not actually spent until after Milestone C. This means that while high-­quality SE is necessary during the entire acquisition cycle, the application of SE to decisions made in the pre-Milestone A period is critical to avoiding (or at least minimizing) cost and schedule overruns later in the program. HISTORY OF AIR FORCE DEVELOPMENT PLANNING Prior to 1990, there were various organizations among the military service acquisition and development communities that focused on critical aspects of what is currently referred to as systems engineering. For example, the Air Force had within the Air Force Systems Command (AFSC) a structured organization whose mission was the front-end part of the total systems engineering process described above. “Strategy-to-task,” a term invented by Lt. Gen. Glenn Kent that describes the process encompassed by the left-hand portions of Figures 1-1 and 1-2, was addressed by such organizations. The name given to these organizations was “Development Planning,” or just “Planning.” Their role was to employ various tools and techniques to define defense strategies, identify gaps in accomplishing those strategies, define con- cepts to address the gaps, use modeling and simulations or prototyping as ways to refine and test concepts, and provide early systems requirements to the systems developers for specific programs. Inherent in this role was the ability to under- stand the state of the art of the technical possibilities available from technology centers (laboratories, universities, industry, and so on), as well as to understand the needs of the user community (warfighters). These are all key attributes of a good pre-Milestone A systems engineering process. Successful programs dis- cussed in Chapter 2 as “best practices” (e.g., C-5 and B-2) were originated during the “development planning” era. Unfortunately, these planning organizations within the Air Force began to erode in the early 1990s, and at the same time the resources to support their   Edward L. Warner III and Glenn A. Kent, 1984, A Framework for Planning the Employment of Air Power in Theater War, N-2038-AF, Santa Monica, Calif.: RAND; see also David E. Thaler, 1993, Strategies to Tasks: A Framework for Linking Means and Ends, MR-300-AF, Santa Monica, Calif.: RAND. 

22 PRE-MILESTONE A AND EARLY-PHASE SYSTEMS ENGINEERING missions eroded. It is not clear whether the changes in the overall acquisition structure for the Air Force precipitated this erosion. However, the decreased role of the Air Force’s acquisition command (originally AFSC, now the Air Force Materiel Command [AFMC]) in program execution contributed to this situation because there ceased to be a strong command-led focus on the function of devel- opment planning. Previously, the acquisition command ensured the availability of funding, manpower, and processes to support this front-end planning at every systems development center (aeronautics, electronics, weapons, and space). This role of the acquisition command headquarters also provided a sort of standards and evaluation of the processes and tools used by the various development plan- ning organizations. Today, there is renewed interest within the Air Force in strengthening the involvement of the acquisition command in the total acquisition process and program execution. As a part of this initiative, there is an opportunity to task the command to once again be the functional lead for development planning. Statement of Task and Committee approach The Committee on Pre-Milestone A Systems Engineering was tasked to look at the role that SE can play during the defense acquisition life cycle in addressing the root causes of program failure. The original statement of task that had been developed with the sponsor before the study began addressed the role of systems engineering in the full defense acquisition life cycle. During the committee’s first meeting with the sponsor, it became apparent that, while the full acquisition life cycle was of interest, the sponsor was especially interested in the role that systems engineering could play during the pre-Milestone A and early phases of a program. The Air Force has concluded that many potential problems can be addressed by sound SE throughout the acquisition life cycle. However, currently there are few formal SE processes applied to Air Force development programs prior to the Milestone A review. The committee agreed to devote most of its attention during the study to the pre-Milestone A and early phases, and the sponsor and the com- mittee agreed to revise the statement of task accordingly. See Box 1-1. However, a limited broader discussion is necessary, because (1) systems engineering best practices and lessons learned apply through multiple program phases; (2) pre-Milestone A and early-phase systems engineering does not g ­ uarantee successful acquisition if poor decisions are made in subsequent pro-   Terry Jaggers, the Deputy Assistant Secretary of the Air Force for Science, Technology, and Engi- neering, sponsored the study. In consultation with the National Research Council’s Air Force Studies Board, he initiated the study and framed its terms, arranged for the Air Force to award and fund the study contract, and helped author the statement of task. He also collaborated with Mark Schaeffer, Director of Systems and Software Engineering in the Office of the Deputy Under Secretary of Defense for Acquisition and Technology. Mr. Schaeffer actively supported the study, essentially acting as an “unofficial” cosponsor. 

INTRODUCTION AND OVERVIEW 23 BOX 1-1 Statement of Task The National Research Council (NRC) will A. Examine Air Force programs that were considered acceptable and unaccept- able by DOD and assess the contribution of pre-Milestone A and early-phase systems engineering to the positive or negative development outcome. Mile- stone A is defined as the end of the Concept Refinement Phase and the b ­ eginning of the Technology Development Phase of the acquisition life cycle. 1. From these examples, describe ways that pre-Milestone A and early-phase systems engineering were, or should have been, accomplished to produce successful results. 2. Assess, describe, and when possible quantify the benefits of pre-Milestone A and early-phase systems engineering in successful programs, and the results of poorly executed systems engineering in terms of cost, schedule, and performance. B. Determine the minimum level of pre-Milestone A and early-phase systems engineering required for program success, and current Air Force barriers to implementation, both on concepts leading to an Analysis of Alternatives (AoA) and for the post-AoA selected alternative. C. Develop a framework/methodology for requirements and development orga- nizations to use to ensure proper pre-Milestone A and early-phase system engineering is accomplished. D. Discuss the positive effects expected to accrue across the remainder of the life cycle by properly accomplished systems engineering during the pre-Milestone A and early phases. E. Discuss issues associated with adequacy and training of the entire workforce relevant to requirements, acquisition, and technology. F. Recommend, in terms of law, policy, processes, and resources (tools, man- power, and funding) changes to enable and ensure the Air Force conducts a ­ dequate pre-Milestone A and early-phase systems engineering and the means for seamless transition from concept development through the genesis of a program office. gram phases and, conversely, program failure post-Milestone A is not necessarily attributable to poor pre-Milestone A or early-phase systems engineering; and (3) the availability of data and proven models that could be used for analysis to causally link pre-Milestone A and early-phase systems engineering with program outcome is questionable. 

24 PRE-MILESTONE A AND EARLY-PHASE SYSTEMS ENGINEERING To address the statement of task, the committee reviewed the literature and heard from many speakers involved in defense system acquisition programs. The committee also drew on the expertise and extensive knowledge possessed by its members, who have had many years of personal experience both in defense acquisition programs and in the practice of SE. The study sponsor did not provide the committee with a list of programs that are considered “successful” or “unsuccessful” by DOD. In addressing statement of task item A, the committee used its own judgment and examined a number of existing case studies of DOD programs in the literature and developed several new ones that it felt illustrate both successful and unsuccessful application of SE, focusing particularly on the pre-Milestone A and early phases. The com­mittee notes, however, that many programs that are judged successful in retrospect were considered to be in trouble during their execution. Thus the perception of “success­ful” and “unsuccessful” programs can change with time and perspec- tive. For purposes of this report, however, the committee does not believe that a program that requires far more time or money to develop than it would have in the 1960s and 1970s can be judged successful even if it ultimately meets its objectives. In a world in which the threats and technology are both evolving ever more rapidly, one cannot be satisfied with excessive deployment times and costs. The lessons learned from these case studies are summarized in Chapter 2. The committee quickly determined the near impossibility of quantitatively isolating, testing, and proving direct causal links between pre-Milestone A and early-phase SE and later program cost, schedule, and performance outcomes. Many studies have searched for and proposed actions to address the root causes of the cost, schedule, and performance problems that seemingly have become the norm for current defense acquisition programs. Consistently, such studies have found that the causes and their effects are complex and interrelated. 10 The com- mittee believes that high-quality pre-Milestone A and early-phase SE certainly contributes to later positive outcomes; however, available data did not allow the contribution of that SE to be reliably isolated from that of other factors such as requirements maturity and stability, funding stability, and domain knowledge of the development team. In that context, the committee addressed statement of task item A(2) qualitatively. The committee addressed statement of task items B and C together by developing a checklist of items that constitute good SE practice during both the pre-Milestone A and pre-Milestone B periods. This checklist is presented in Chapter 4 (Box 4-1). As required in statement of task item B, the checklist items are divided into those that should be completed prior to the analysis of alterna-   See Appendix B for a list of speakers and the presentations made to the committee. 10  Ronald Kadish, Gerald Abbott, Frank Cappuccio, Richard Hawley, Paul Kern, and Donald K ­ ozlowski, 2006, Defense Acquisition Performance Assessment. Available at http://www.acq.osd.mil/ dapaproject/documents/DAPA-Report-web/DAPA-Report-web-feb21.pdf. Last accessed on April 2, 2007. 

INTRODUCTION AND OVERVIEW 25 tives (AoA), and those that may be completed afterward. Associated with most checklist items are brief statements of the benefits that are expected to accrue throughout the program life cycle as a result of properly executing that item, as required in statement of task item D. The discussion of issues associated with the systems engineering workforce and training, required under statement of task item E, is taken up in Chapter 3. That chapter provides a snapshot of the demographics of the current SE work- force as well as insights into current industry SE programs to help keep DOD programs on time and on budget. The committee’s policy recommendations on needed changes to law, pro- cesses, and resources, required under statement of task item F, are distributed throughout the report as they arise in the context of the discussion in various chapters. These recommendations are presented together in the Summary at the beginning of the report. In preparing this report, the committee assumed that readers are generally familiar with SE and defense acquisition. The committee included brief defini- tions and descriptions where appropriate; however, it did not attempt to provide extensive tutorials on these subjects. Doing so would have been well beyond its charter and resources, and extensive literature and online resources already serve that purpose.11 11  See, for example, the International Council on Systems Engineering at http://www.incose.org/ practice/whatissystemseng.aspx, and the Defense Acquisition Guide Book at https://akss.dau.mil/dag/ welcome.asp.