Reliability Growth Enhancing Defense System Reliability (2015) / Chapter Skim
Currently Skimming:
10 Conclusions and Recommendations
10 Conclusions and Recommendations
Pages 135-166
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 135... ...
cover the many steps and aspects of the acquisition process, presented in this chapter in roughly chronological order: analysis of alternatives; request for proposals; an outline reliability demonstration plan; raising the priority of reliability; design for reliability; reliability growth testing; design changes; information on operational environments; acquisition contracts; delivery of prototypes for developmental testing; developmental testing; and intermediate reliability goals.
|
From page 136... ...
After there is a decision to proceed, reliability requirements are first introduced and justified in the RAM-C (reliability, availability, maintainability, and cost) document, which lays out the reliability requirements for the intended system and contains the beginnings of a reliability model justifying that the reliability requirement is technically feasible.
|
From page 137... ...
RECOMMENDATION 1 The Under Secretary of Defense for Acquisi tion, Technology, and Logistics should ensure that all analyses of alterna tives include an assessment of the relationships between system reliability and mission success and between system reliability and life-cycle costs. The next stage in the acquisition process is the setting of reliability requirements.
|
From page 138... ...
To ensure that the required report about the reliability requirements reflects input from people with the necessary expertise, DoD should require that an external panel examines the arguments behind such assertions prior to the issuance of an RFP. That assessment of reliability requirements should be delivered to the Joint Requirements Oversight Council (JROC)
|
From page 139... ...
Reliability engineers of the services involved in each particular acquisition should have full access to the technical report and should be consulted prior to the finalization of the RFP. REQUESTS FOR PROPOSALS As argued above, requests for proposals should include reliability requirements and their justification -- highlighting the reliability goals for specific subsystems that are believed to be keys to system reliability -- by demonstrating that they are necessary either to have a high probability of successfully carrying out the intended missions or by showing that a reliability level is necessary to limit life-cycle costs.3 We acknowledge here, too, that prior to any system development, the assessment of feasibility and the linking of the level of system reliability to life-cycle costs is at best informed guesswork.
|
From page 140... ...
In situations in which new technology is involved, DoD may instead issue a request for information to engage knowledgeable people from industry in the process of preparing the report on requirements. If new or developing technology may be needed in the system, the process of evolutionary acquisition needs to be considered.4 In this case, necessary, achievable, measurable, and testable reliability requirements for the system during each acquisition spiral need to be specified and justified.
|
From page 141... ...
To address this, we suggest that DoD archive the history of the development of the initial reliability requirement in the RFP and how that initial requirement evolved throughout development and even in initial fielding and subsequent use. AN OUTLINE RELIABILITY DEMONSTRATION PLAN Knowing the design of the tests that will be used to evaluate a system in development is an enormous help to developers in understanding the missions and the stresses the system is expected to face.
|
From page 142... ...
As with the technical report on reliability requirements recommended above and because an outline reliability demonstration plan also has substantial technical content, it should be reviewed by an expert panel prior to its inclusion in an RFP. This expert panel should include reliability engineers and system users, members from the testing community, and members from outside of the service responsible for system acquisition.
|
From page 143... ...
The outline reliability demonstration plan should also provide the technical basis for how test and evaluation will track in a statistically defendable way the current reliability of a system in development given the likely number of government test events as part of developmental and operational testing. Prior to being included in the request for proposal for an acquisition program, the outline reliability demonstra tion plan should be reviewed by an expert external panel.
|
From page 144... ...
RECOMMENDATION 5 The Under Secretary of Defense for Acqui sition, Technology, and Logistics should ensure that reliability is a key performance parameter: that is, it should be a mandatory contractual requirement in defense acquisition programs. DESIGN FOR RELIABILITY AND RELIABILITY TESTING As discussed throughout this report, there are two primary ways, in combination, to achieve reliability requirements in a development program: reliability can be "grown" by using a test-analyze-fix-test process, and the initial design can be developed with system reliability as an objective.
|
From page 145... ...
However, a careful analysis can identify proposed systems and development plans that are or are not likely to meet the reliability requirements without substantial increase in development costs and/or extensive time delays. To develop a reasonable estimate of the initial reliability corresponding to a system design, one would start with the reliability of the components and subsystems that have been used in previous systems and engineering arguments and then combine this information using reliability block diagrams, fault-tree analyses, and physics-of-failure modeling.
|
From page 146... ...
Appendix D provides a critique of MILHDBK-217 as a method for predicting the reliability of newly developed electronic components. The basic problem with MIL-HDBK-217 is that it does not identify the root causes, failure modes, and failure mechanisms of electronic components.
|
From page 147... ...
In order to improve the current reliability assessment process, there needs to be a transition to a science-based approach for determining how component hazard rates vary as a function of time. For many applications, the notion of the constant failure rate should be replaced by a composite instantaneous hazard rate which is based on root-cause failure mechanisms.
|
From page 148... ...
The failure mechanisms are validated through experimentation and replication by multiple researchers. Industry is now recognizing that an understanding of potential failure mechanisms leads to eliminating them cost-effectively, and is consequently demanding an approach to reliability modeling and assessment that uses knowledge of failure mechanisms to encour age robust designs and manufacturing practices (Cushing et al., 1993)
|
From page 149... ...
Further, we realize that there is nothing particular in this regard with electronic components, and therefore, we are recommending that such techniques be used to help design for and assess the reliability of sub systems early in system development for all components in all subsystems. In particular, physics of failure should be utilized to identify potential wear-out failure modes and mitigations for enhancing long-term reliability performance.
|
From page 150... ...
The panel should report to JROC, which should use this information in awarding acquisition contracts. Software reliability growth through the test-analyze-and-fix approach can be assessed using various metrics, including build success rate, code dependency metrics, code complexity metrics, assessments of code churn and code stability, and code velocity.
|
From page 151... ...
They should also support the allocation of adequate budget reserves that may become necessary if the originally envisioned reliability growth plan turns out to be optimistic. RELIABILITY GROWTH TESTING One can certainly argue that one reason that many defense systems fail to achieve their reliability requirements is because there are too many defects that have not been discovered when the system enters operational testing.
|
From page 152... ...
The information should cover all types of hardware testing, including testing under operationally relevant conditions, and any use of accelerated or highly accelerated testing.7 The contractor should also provide DoD with information on all types of software testing, including the results of code reviews, automated testing, fault seeding, security testing, and unit test coverage. In order to ensure that this information is provided to DoD, acquisition contracts will need to be written so that this access is mandated and proposals will need to state that contractors agree to share this information.
|
From page 153... ...
The budget for acquisition contracts should include a line item to provide DoD with full access to such data and other analyses. MODELING IN CONJUNCTION WITH ACCELERATED TESTING Similar to the panel's concerns above about the use of reliability growth models for extrapolation, models used in conjunction with accelerated testing linking extreme use to normal use also use extrapolation and therefore need to be validated for this use.
|
From page 154... ...
To provide for some independent testing, the contractor should provide DoD with fully documented software that conducts automated software testing for all of its software-intensive subsystems and for the full system when the full system is a software system. This documentation will enable DoD to test the software for many times the order of magnitude of replications that would otherwise be possible in either developmental or operational testing.
|
From page 155... ...
RECOMMENDATION 17 The Under Secretary of Defense for Acqui sition, Technology, and Logistics should ensure that there is a line item in all acquisition budgets for oversight of subcontractors' compliance with reliability requirements and that such oversight plans are included in all proposals. ACQUISITION CONTRACTS The above recommendations would require contractors to lay out their intended design for reliability and reliability testing activities in acquisition proposals.
|
From page 156... ...
Furthermore, even though there will likely be appreciable reliability growth as a result of developmental and operational testing, not only is it limited by the lack of operational realism in developmental testing and the short time frame of operational testing, but there also will likely be reliability "decline" due to the developmental test/operational test gap (see Chapter 8.) Although the magnitudes of these increases and decreases cannot be determined a priori, one can increase overall reliability of all systems by requiring that prototypes achieve their reliability requirements on delivery to DoD.
|
From page 157... ...
Such assessments will, at times, be later demonstrated by DoD to be high or low, but this approach will support a learning process about how to better merge such information in the future. DEVELOPMENTAL TESTING The monitoring of system reliability prior to operational testing is important, because it is likely that relying on operational tests to expose reliability problems will result in too many defense systems exhibiting deficient system reliability when fielded (see Chapter 7)
|
From page 158... ...
Reliability deficiencies can continue to arise after deployment, partly because however realistic an operational test strives to be, it will always d iffer from field deployment. Field operations can stress systems in unforeseen ways and reveal failure modes that were not likely to have been unearthed in either developmental or operational testing.
|
From page 159... ...
Because of changes in technology trends, the evolution of complex supply-chain interactions, a cost-effective and efficient parts selection and management process is needed to perform this assessment. INTERMEDIATE RELIABILITY GOALS Setting target values for tracking system reliability during development is important for discriminating between systems that are likely to achieve their reliability requirements and those that will struggle to do so.
|
From page 160... ...
. After prototype delivery, the specified initial reliability level could be the reliability assessed by the contractor on delivery or during early full-system developmental testing; the final level would be the specified requirement, and its date would be the scheduled time for initiation of operational testing.
|
From page 161... ...
Analysis of these data may also determine the factors that play a role in achieving reliability requirements. For example, it would be of great importance to determine which design-for-reliability techniques or what budgets for design for reliability were predictive of higher or lower rates of initial full system developmental test reliability levels.
|
From page 162... ...
Finally, DoD should seek to identify leading examples of good practice of the development of reliable systems of various distinct types and collect them in a casebook for use by program managers. Once it is better understood how near the initial reliability needs to be to the required level to have a good chance of attaining the required level prior to entry into operational testing, acquisition contracts could indicate the reliability levels that need to be attained by the contractor before a system is promoted to various stages of development.
|
From page 163... ...
Partly as a result of the absence of these details, there was no guarantee that reliability-related activities would take place. In fact, proposals that did allocate a substantial budget and development time and detail in support of specific design-for-reliability procedures and comprehensive testing have been implicitly penalized because their costs of development were higher and their delivery schedules were longer in comparison with proposals that made less specific assertions as to how their reliability requirements would be met.
|
From page 164... ...
In its work to produce this report, the panel did identify a number of research areas that DoD might consider supporting in the future: reliability design support, comprehensive reliability assessment, and assessing aspects of reliability that are difficult to observe in development and operational testing. With regard to reliability design support, research on the relationship between physical failure mechanisms and new technologies seems warranted.
|
From page 165... ...
CONCLUSIONS AND RECOMMENDATIONS 165 There are also intriguing near- and long-term reliability issues that would be difficult to observe in development and operational testing. We note, particularly: • the identification, characterization, and modeling of the effects of defects that can lead to early failures (infant mortality)
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.