The role of the U.S. Department of Defense (DoD) in providing oversight and management of the acquisition process was emphasized by our sponsor as an issue that might benefit from the panel’s review. To help us understand those issues as broadly as possible, we undertook a brief review of system development for commercial products, in particular, of those companies recognized as producing products with high reliability. This topic was a feature of the panel’s workshop, and it informed our work.
Throughout this report we often make comparisons, both implicit and explicit, between system development as practiced by companies that are recognized as producing highly reliable systems and current practice in defense acquisition. Such comparisons are useful, but it is important to keep in mind that system development in defense acquisition has important differences that distinguish it from commercial system development.
The first section below discusses key differences between commercial and defense acquisition. The second section discusses the use of an incentive system for defense acquisition with regard to reliability. The final section of the chapter presents a view of best practices by Tom Wissink of Lockheed Martin and Lou Gullo of Raytheon, who have extensive experience in developing reliable defense systems.
The first difference between commercial and defense acquisition is the sheer size and complexity of defense systems. In developing, manufacturing, and fielding a complicated defense system (e.g., an aircraft or land vehicle),
one is dealing with many subsystems, each of substantial complexity: this characteristic, by itself, poses enormous management and technological challenges. A ship, for instance, might have a single program manager but more than 100 acquisition resource managers. Or a defense system may be a single element in an extensive architecture (e.g., command, control, and communications [C3] network), with separate sets of interface requirements evolving as the architecture components progress through their individual stages of acquisition and deployment. New systems may also have to interface with legacy systems. Furthermore, defense systems often strive to incorporate emerging technologies. Although nondefense systems can also be extremely complicated and use new technologies, they often are more incremental in their development, and therefore tend to be more closely related to their predecessors than are defense systems.
Second, there are significant differences in program management between commercial and defense system development. Commercial system development generally adheres to a single perspective, which is embodied in a project manager with a clear profit motive and direct control of the establishment of requirements, system design and development, and system testing. In contrast, defense system development has a number of relatively independent “agents,” including the system development contractor; a DoD program manager; DoD testers; and the Office of the Secretary of Defense, which has oversight responsibilities. There is also the military user, who has a different relationship with the contractor than a customer purchasing a commercial product from a manufacturer (see below).
These different groups have varying perspectives and somewhat different incentives. In particular, there is sometimes only limited information sharing between contractors and the various DoD agents. This lack of communication is a serious constraint when considering system reliability, because it prevents DoD from providing more comprehensive oversight of a system’s readiness for developmental and operational testing with respect to its reliability performance. It also limits DoD’s ability to target its testing program to aspects of system design that are contributing to deficient system reliability. Once prototypes have been provided to DoD for developmental testing, let alone for operational testing, the system design is relatively set, and cost-efficient opportunities for enhancing reliability may have been missed.
A third difference between defense and commercial system development concerns risk. In the commercial world, the manufacturer carries most of the risks that would result from developing a system with poor reliability performance. Such risks include low sales, increased warranty expenses, the loss of customer goodwill for products with poor reliability, and increased life-cycle costs for systems that are maintained by the manufacturer (e.g., locomotives and aircraft engines). Consequently, commercial manufacturers have a strong
motivation to use reliability engineering methods that help to ensure that the system will meet its reliability goals by the time it reaches the customer.
For defense systems, however, the government and the customers (i.e., the military services that will use the system) generally assume most of the risk created by poor system reliability. Therefore, the system developers do not have a strong incentive to address reliability goals early in the acquisition process, especially when it can be difficult to quantify the possibly substantial downstream benefits of (seemingly) costly up-front reliability improvement efforts.
This third issue is extremely important: if developers shared some of the risk, then they would be much more likely to focus on reliability early in development, even without significant DoD oversight. Risk sharing is one component of the larger question of establishing a system of rewards and penalties—either during development or after delivery—for achieving (or exceeding) target levels or final requirements for reliability or for failing to do so.
Although DoD has at times used incentives to reward contractors for exceeding requirements, the panel is unaware of any attempt to institute a system of warranties for a defense system that provides for contractor payments for failure to meet reliability requirements.1 And in considering incentives, we are unaware of any studies of whether offering such payments has succeeded in motivating developers to devote greater priority to meeting reliability requirements. Most relevant to this report is the idea that rewards could be applied at intermediate points during development, with rewards for systems that are assessed to be ahead of intermediate reliability goals, or penalties for systems that are assessed to be substantially behind such goals. (See discussion in Chapter 7.)
In considering an incentive system, we note some complications that need to be kept in mind in its development. First, such a system must be based on the recognition that some development problems are intrinsically harder than others by requiring a nontrivial degree of technology development. Having a penalty for not delivering a prototype on time or not meeting a reliability requirement may dissuade quality developers from bidding on such development contracts because of the perceived high risk. The best contractors may be the ones that know how difficult it will be to meet a requirement and might therefore not offer a proposal for a difficult acquisition program because of a concern about incurring penalties because
1 For a useful consideration of statistical issues involved with product warranties, see Blischke and Murthy (2000).
of something that could not be anticipated. Therefore, it would be useful, to the extent possible, to link any incentive payments, either positive or negative, to the intrinsic challenge of a given system.
Looking at a later stage in the acquisition process, providing incentives (or penalties) for performance after delivery of prototypes, rather than during intermediate stages of development, would have the advantage of being able to use more directly relevant information, in that the the reliability level achieved by a system could be directly assessed through DoD developmental and operational testing. But for earlier, intermediate reliability goals, the assessment is carried out by the contractor, so an incentive system may lead to various attempts to affect (“game”) the assessment. For instance, the test environments and the stress levels for those environments might be selected to avoid ones that are particularly problematic for a given system. Moreover, whether a test event should or should not be counted as a failure or whether a test event itself is considered “countable” can sometimes rely on judgment and interpretation. It is also important to note that testing for intermediate reliability goals is not likely to have large sample sizes, and such estimates are therefore likely to have substantial uncertainty. As a result and depending on the decision rule used, there are likely to be either substantial consumer or producer risks in such decisions concerning incentive payments.
Finally, offering incentives regarding delivery schedules raises another concern: potential tradeoffs between early delivery and reliability achievement. Because life-cycle costs and overall system performance can be sensitive to system reliability, compromising on reliability performance to meet an inflexible deadline is often a bad bargain.
For the panel’s September 2011 workshop, we asked Tom Wissink of Lockheed Martin and Lou Gullo of Raytheon to discuss best practices that are currently used to develop reliable systems and how DoD could promote and support such actions in its requests for proposals and contracts. We asked them to include in their comments the use of design-for-reliability methods, reliability testing, and use of reliability growth models. We also asked them to comment on the impact of recent changes in procedures brought about by the adoption of ANSI/GEIA-STD-0009 and DTM 11-003 and to provide suggestions on how best to move forward to help produce more reliable defense systems in the future.2 To answer our questions, Wissink and Gullo not only relied on their own experiences, but also talked with engineers at their companies. They presented their answers both in
2 Please see footnotes 4 and 5 in Chapter 1 on the role of these two documents.
writing and orally at the workshop. Overall, Wissink and Gullo said, design for reliability requires
- identification and mitigation of design weaknesses,
- detection and resolution of mission critical failures, and
- characterization of design margins and continuous design improvements to decrease failures in the field and reduce total cost of ownership over the system or product life cycle.
Regarding the communication of the need for design for reliability in proposals, they said that every acquisition contract should specify (1) the system reliability requirement as a key performance parameter, (2) what reliability engineering activities should be performed during system development to achieve the requirement and (3) the means to verify the reliability requirement. “Reliability growth management as part of design for reliability represented in proposals is a way to specifically require actions to improve reliability over time with reliability assessment of standard reliability metrics over the system/product life cycle, starting early in the development phase,” Wissink and Gullo wrote.
Winnick and Gullo suggested that the DoD acquisition requests for proposals should ask for a written reliability growth management plan and a reliability growth test plan as part of the integrated test planning. There should also be a “reliability growth incentive award scale and incentive fee scheduled during intervals in the development cycle so that the contractor is rewarded for favorable reliability growth that exceeds customer expectations.” Reliability growth management planning entails the development of reliability growth curves for the system, major subsystems, products, and assemblies, along with the plan for achieving specified reliability values.
Reliability growth management “includes reliability assessments and a test plan that contains various types of testing (e.g., accelerated life tests, highly accelerated life tests), adequate test time in the program schedule, and test samples to demonstrate increasing reliability with increasing confidence over time.” Furthermore, they wrote, reliability growth management provides “a means for tracking reliability growth from system level to assembly or configuration item level, and monitoring progress….” It also includes “intervals in the development phase to allow for implementation of design change corrective actions to positively affect reliability with each subsequent design modification.”
Winnick and Gullo expanded on what DoD should require in a request for proposal (RFP). They said that the RFP should require that the different support contractors for each major subsystem provide a reliability growth
profile3 to demonstrate the anticipated reliability growth over time. This curve should be a member of a set of models that are approved by DoD for such use. The software used to implement this should provide standard reporting outputs to program management to verify and validate the reliability growth curve for the particular program.
The outputs, they said, should include plotting the reliability growth curves over time (for each major subsystem) to provide the following information:
- the expected starting point (initial reliability), with the context in a separate document that supports the data sources and the rationale for its selection;
- the number of tests planned during the development program to be used to verify that starting point;
- the expected reliability growth profile with the context in a separate document that supports the data sources and the rationale for the selection of the points on the graph;
- the number of tests needed to produce that profile, the schedule for these tests, and the schedule for implementing design change corrective actions for the failures that are expected to occur, resulting in design reliability improvements; and
- a risk assessment for the starting point, reliability growth profile, and number of tests necessary to meet the required reliability levels on the growth curve.
They also noted that every DoD acquisition RFP and test and evaluation master plan or systems engineering plan “should require contractors to provide a reliability growth management plan and a reliability growth test plan as part of the integrated test plan and reliability growth profile.”
The presentation by Winnick and Gullo and the workshop discussion on this topic provided the panel with a better understanding of what contractors would be willing to provide in support of the production of reliable systems. We note, however, several reservations about their conclusions.
There is a great deal of variability in the complexity of DoD systems and the constitution of their respective “major subsystems.” We are not convinced that a formal mathematical reliability growth curve should be developed for every major subsystem of every DoD system—although some sound plan for growing and demonstrating reliability is appropriate for each major subsystem. The approval of the reliability growth tools that are implemented for a specific subject system and its major subsystems is tacit
3 A reliability growth profile is a representation of future reliability growth as a function over time: see Chapter 4.
in the DoD oversight of acquisition processes, which includes the review and approval of essential test and evaluation documents. Furthermore, we do not envision the construction of a master list of DoD-approved reliability growth tools. As discussed throughout the report, the viability and appropriateness of such tools will need to be case specific.