National Academies Press: OpenBook

Reliability Growth: Enhancing Defense System Reliability (2015)

Chapter:Appendix A: Recommendations of Previous Relevant Reports of the Committee on National Statistics

« Previous: References
Suggested Citation:"Appendix A: Recommendations of Previous Relevant Reports of the Committee on National Statistics." National Research Council. 2015. Reliability Growth: Enhancing Defense System Reliability. Washington, DC: The National Academies Press. doi: 10.17226/18987.
×

Appendix A

Recommendations of Previous Relevant Reports of the Committee on National Statistics

The Committee on National Statistics has carried out a number of studies sponsored by the Under Secretary of Defense for Acquisition, Technology, and Logistics and the Director of Operational Test and Evaluation of the U.S. Department of Defense that are relevant to this study. The previous studies covered the application of statistical, system engineering, and software engineering techniques to improve the development of defense systems. Many of the conclusions and recommendations of these studies are relevant to the development of reliable defense systems, the topic of this study.

The rest of this appendix reproduces those conclusions and recommendations, a large number of which have not been fully implemented. Their inclusion here serves both to highlight that some of the issues in this report have a long history and to emphasize the connections among the many parts of the system of defense development, acquisition, and testing.

The reports are listed in chronological order; the conclusions and recommendations are produced in full. All the reports were published by and are available from the National Academies Press.

Statistics, Testing, and Defense Acquisition: New Approaches and Methodological Improvements (1998)

This study was a general overview of the application of statistical methods to many components of defense acquisition.

Suggested Citation:"Appendix A: Recommendations of Previous Relevant Reports of the Committee on National Statistics." National Research Council. 2015. Reliability Growth: Enhancing Defense System Reliability. Washington, DC: The National Academies Press. doi: 10.17226/18987.
×

RECOMMENDATION 7.1 The Department of Defense and the military services should give increased attention to their reliability, availability, and maintainability data collection and analysis procedures because deficiencies continue to be responsible for many of the current field problems and concerns about military readiness. (p. 105)

RECOMMENDATION 7.2 The Test and Evaluation Master Plan and associated documents should include more explicit discussion of how reliability, availability, and maintainability issues will be addressed, particularly the extent to which operational testing, modeling, simulation, and expert judgment will be used. The military services should make greater use of statistically designed tests to assess reliability, availability, and maintainability and related measures of operational suitability. (p. 106)

RECOMMENDATION 7.3 As part of an increased emphasis on assessing the reliability, availability, and maintainability of prospective defense systems, reasonable criteria should be developed for each system. Such criteria should permit a balancing of a variety of considerations and be explicitly linked to estimates of system cost and performance. A discussion of the implications for performance, and cost of failing, if the system’s demonstrated reliability, availability, and maintainability characteristics fall below numerical goals should be included. (p. 107)

RECOMMENDATION 7.4 Operational test agencies should promote more critical attention to the specification of statistical models of equipment reliability, availability, and maintainability and to supporting the underlying assumptions. Evidence from plots, diagnostics, and formal statistical tests—developed from the best currently available methods and software—should be used to justify the choice of statistical models used in both the design and the analysis of operational suitability tests. (p. 113)

RECOMMENDATION 7.6 Service test agencies should carefully document, in advance of operational testing, the failure definitions and criteria to be used in scoring reliability, availability, and maintainability data. The objectivity of the scoring procedures that were actually implemented should be assessed and included in the reporting of results. The sensitivity of final reliability, availability, and maintainability estimates to plausible alternative interpretations of test data, as well as subsequent assumptions concerning operating tempo and logistics support, should be discussed in the reporting. (p. 116)

Suggested Citation:"Appendix A: Recommendations of Previous Relevant Reports of the Committee on National Statistics." National Research Council. 2015. Reliability Growth: Enhancing Defense System Reliability. Washington, DC: The National Academies Press. doi: 10.17226/18987.
×

RECOMMENDATION 7.7 Methods of combining reliability, availability, and maintainability data from disparate sources should be carefully studied and selectively adopted in the testing processes associated with the Department of Defense acquisition programs. In particular, authorization should be given to operational testers to combine reliability, availability, and maintainability data from developmental and operational testing as appropriate, with the proviso that analyses in which this is done be carefully justified and defended in detail. (p. 119)

RECOMMENDATION 7.8 All service-approved reliability, availability, and maintainability data, including vendor-generated data, from technical, developmental, and operational tests, should be properly archived and used in the final preproduction assessment of a prospective system. After procurement, field performance data and associated records should be retained for the system’s life, and used to provide continuing assessment of its reliability, availability, and maintainability characteristics. (p. 120)

RECOMMENDATION 7.9 Any use of model-based reliability predictions in the assessment of operational suitability should be validated a posteriori with test and field experience. Persistent failure to achieve validation should contraindicate the use of reliability growth models for such purposes. (p. 122)

RECOMMENDATION 7.10 Given the potential benefits of accelerated reliability testing methods, we support their further examination and use. To avoid misapplication, any model that is used as an explicit or implicit component of an accelerated reliability test must be subject to the same standards of validation, verification, and certification as models that are used for evaluation of system effectiveness. (p. 124)

RECOMMENDATION 7.11 The Department of Defense should move aggressively to adapt for all test agencies the Organization for International Standardization (ISO) standards relating to reliability, availability, and maintainability. Great attention should be given to having all test agencies ISO-certified in their respective areas of responsibility for assuring the suitability of prospective military systems. (p. 125)

RECOMMENDATION 7.12 Military reliability, availability, and maintainability testing should be informed and guided by a new battery of military handbooks containing a modern treatment of all pertinent topics in the fields of reliability and life testing, including, but not limited to, the design and analysis of standard and accelerated tests, the

Suggested Citation:"Appendix A: Recommendations of Previous Relevant Reports of the Committee on National Statistics." National Research Council. 2015. Reliability Growth: Enhancing Defense System Reliability. Washington, DC: The National Academies Press. doi: 10.17226/18987.
×

handling of censored data, stress testing, and the modeling of and testing for reliability growth. The modeling perspective of these handbooks should be broad and include practical advice on model selection and model validation. The treatment should include discussion of a broad array of parametric models and should also describe nonparametric approaches. (p. 126)

Innovations in Software Engineering for Defense Systems (2003)

This study covered statistically oriented software engineering methods that were useful for the development of defense systems.

RECOMMENDATION 1 Given the current lack of implementation of state-of-the-art methods in software engineering in the service test agencies, initial steps should be taken to develop access to—either in-house or in a closely affiliated relationship—state-of-the-art software engineering expertise in the operational or developmental service test service agencies. (p. 2)

RECOMMENDATION 2 Each service’s operational or developmental test agency should routinely collect and archive data on software performance, including test performance data and data on field performance. The data should include fault types, fault times and frequencies, turnaround rate, use scenarios, and root cause analysis. Also, software acquisition contracts should include requirements to collect such data. (p. 3)

RECOMMENDATION 3 Each service’s operational or developmental test agency should evaluate the advantages of the use of state-of-the-art procedures to check the specification of requirements for a relatively complex defense software-intensive system. (p. 3)

RECOMMENDATION 6 DoD needs to examine the advantages and disadvantages of the use of methods for obligating software developers under contract to DoD to use state-of-the-art methods for requirements analysis and software testing, in particular, and software engineering and development more generally. (p. 4)

Suggested Citation:"Appendix A: Recommendations of Previous Relevant Reports of the Committee on National Statistics." National Research Council. 2015. Reliability Growth: Enhancing Defense System Reliability. Washington, DC: The National Academies Press. doi: 10.17226/18987.
×

Testing of Defense Systems in an Evolutionary Acquisition Environment (2006)

This study considered the methods that would be applicable to defense systems development that is implemented in stages.

  1. … revise DoD testing procedures to explicitly require that developmental tests have an operational perspective …,
  2. require (DoD) contractors to share all relevant data on system performance and the results of modeling and simulation …

Industrial Methods for the Effective Development and Testing of Defense Systems (2012)

This study examined the potential of the use of system and software engineering methods for defense system development.

The performance of a defense system early in development is often not rigorously assessed, and in some cases the results of assessments are ignored; this is especially so for suitability assessments. This lack of rigorous assessment occurs in the generation of system requirements; in the timing of the delivery of prototype components, subsystems, and systems from the developer to the government for developmental testing; and in the delivery of production-representative system prototypes for operational testing. As a result, throughout early development, systems are allowed to advance to later stages of development when substantial design problems remain.

Suggested Citation:"Appendix A: Recommendations of Previous Relevant Reports of the Committee on National Statistics." National Research Council. 2015. Reliability Growth: Enhancing Defense System Reliability. Washington, DC: The National Academies Press. doi: 10.17226/18987.
×

This page intentionally left blank.

Suggested Citation:"Appendix A: Recommendations of Previous Relevant Reports of the Committee on National Statistics." National Research Council. 2015. Reliability Growth: Enhancing Defense System Reliability. Washington, DC: The National Academies Press. doi: 10.17226/18987.
×
Page175
Suggested Citation:"Appendix A: Recommendations of Previous Relevant Reports of the Committee on National Statistics." National Research Council. 2015. Reliability Growth: Enhancing Defense System Reliability. Washington, DC: The National Academies Press. doi: 10.17226/18987.
×
Page176
Suggested Citation:"Appendix A: Recommendations of Previous Relevant Reports of the Committee on National Statistics." National Research Council. 2015. Reliability Growth: Enhancing Defense System Reliability. Washington, DC: The National Academies Press. doi: 10.17226/18987.
×
Page177
Suggested Citation:"Appendix A: Recommendations of Previous Relevant Reports of the Committee on National Statistics." National Research Council. 2015. Reliability Growth: Enhancing Defense System Reliability. Washington, DC: The National Academies Press. doi: 10.17226/18987.
×
Page178
Suggested Citation:"Appendix A: Recommendations of Previous Relevant Reports of the Committee on National Statistics." National Research Council. 2015. Reliability Growth: Enhancing Defense System Reliability. Washington, DC: The National Academies Press. doi: 10.17226/18987.
×
Page179
Suggested Citation:"Appendix A: Recommendations of Previous Relevant Reports of the Committee on National Statistics." National Research Council. 2015. Reliability Growth: Enhancing Defense System Reliability. Washington, DC: The National Academies Press. doi: 10.17226/18987.
×
Page180
Next: Appendix B: Workshop Agenda »
Reliability Growth: Enhancing Defense System Reliability Get This Book
×
Buy Paperback | $60.00 Buy Ebook | $48.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

A high percentage of defense systems fail to meet their reliability requirements. This is a serious problem for the U.S. Department of Defense (DOD), as well as the nation. Those systems are not only less likely to successfully carry out their intended missions, but they also could endanger the lives of the operators. Furthermore, reliability failures discovered after deployment can result in costly and strategic delays and the need for expensive redesign, which often limits the tactical situations in which the system can be used. Finally, systems that fail to meet their reliability requirements are much more likely to need additional scheduled and unscheduled maintenance and to need more spare parts and possibly replacement systems, all of which can substantially increase the life-cycle costs of a system.

Beginning in 2008, DOD undertook a concerted effort to raise the priority of reliability through greater use of design for reliability techniques, reliability growth testing, and formal reliability growth modeling, by both the contractors and DOD units. To this end, handbooks, guidances, and formal memoranda were revised or newly issued to reduce the frequency of reliability deficiencies for defense systems in operational testing and the effects of those deficiencies. Reliability Growth evaluates these recent changes and, more generally, assesses how current DOD principles and practices could be modified to increase the likelihood that defense systems will satisfy their reliability requirements. This report examines changes to the reliability requirements for proposed systems; defines modern design and testing for reliability; discusses the contractor's role in reliability testing; and summarizes the current state of formal reliability growth modeling. The recommendations of Reliability Growth will improve the reliability of defense systems and protect the health of the valuable personnel who operate them.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!