The Committee on National Statistics has carried out a number of studies sponsored by the Under Secretary of Defense for Acquisition, Technology, and Logistics and the Director of Operational Test and Evaluation of the U.S. Department of Defense that are relevant to this study. The previous studies covered the application of statistical, system engineering, and software engineering techniques to improve the development of defense systems. Many of the conclusions and recommendations of these studies are relevant to the development of reliable defense systems, the topic of this study.
The rest of this appendix reproduces those conclusions and recommendations, a large number of which have not been fully implemented. Their inclusion here serves both to highlight that some of the issues in this report have a long history and to emphasize the connections among the many parts of the system of defense development, acquisition, and testing.
The reports are listed in chronological order; the conclusions and recommendations are produced in full. All the reports were published by and are available from the National Academies Press.
Statistics, Testing, and Defense Acquisition: New Approaches and Methodological Improvements (1998)
This study was a general overview of the application of statistical methods to many components of defense acquisition.
RECOMMENDATION 7.1 The Department of Defense and the military services should give increased attention to their reliability, availability, and maintainability data collection and analysis procedures because deficiencies continue to be responsible for many of the current field problems and concerns about military readiness. (p. 105)
RECOMMENDATION 7.2 The Test and Evaluation Master Plan and associated documents should include more explicit discussion of how reliability, availability, and maintainability issues will be addressed, particularly the extent to which operational testing, modeling, simulation, and expert judgment will be used. The military services should make greater use of statistically designed tests to assess reliability, availability, and maintainability and related measures of operational suitability. (p. 106)
RECOMMENDATION 7.3 As part of an increased emphasis on assessing the reliability, availability, and maintainability of prospective defense systems, reasonable criteria should be developed for each system. Such criteria should permit a balancing of a variety of considerations and be explicitly linked to estimates of system cost and performance. A discussion of the implications for performance, and cost of failing, if the system’s demonstrated reliability, availability, and maintainability characteristics fall below numerical goals should be included. (p. 107)
RECOMMENDATION 7.4 Operational test agencies should promote more critical attention to the specification of statistical models of equipment reliability, availability, and maintainability and to supporting the underlying assumptions. Evidence from plots, diagnostics, and formal statistical tests—developed from the best currently available methods and software—should be used to justify the choice of statistical models used in both the design and the analysis of operational suitability tests. (p. 113)
RECOMMENDATION 7.6 Service test agencies should carefully document, in advance of operational testing, the failure definitions and criteria to be used in scoring reliability, availability, and maintainability data. The objectivity of the scoring procedures that were actually implemented should be assessed and included in the reporting of results. The sensitivity of final reliability, availability, and maintainability estimates to plausible alternative interpretations of test data, as well as subsequent assumptions concerning operating tempo and logistics support, should be discussed in the reporting. (p. 116)
RECOMMENDATION 7.7 Methods of combining reliability, availability, and maintainability data from disparate sources should be carefully studied and selectively adopted in the testing processes associated with the Department of Defense acquisition programs. In particular, authorization should be given to operational testers to combine reliability, availability, and maintainability data from developmental and operational testing as appropriate, with the proviso that analyses in which this is done be carefully justified and defended in detail. (p. 119)
RECOMMENDATION 7.8 All service-approved reliability, availability, and maintainability data, including vendor-generated data, from technical, developmental, and operational tests, should be properly archived and used in the final preproduction assessment of a prospective system. After procurement, field performance data and associated records should be retained for the system’s life, and used to provide continuing assessment of its reliability, availability, and maintainability characteristics. (p. 120)
RECOMMENDATION 7.9 Any use of model-based reliability predictions in the assessment of operational suitability should be validated a posteriori with test and field experience. Persistent failure to achieve validation should contraindicate the use of reliability growth models for such purposes. (p. 122)
RECOMMENDATION 7.10 Given the potential benefits of accelerated reliability testing methods, we support their further examination and use. To avoid misapplication, any model that is used as an explicit or implicit component of an accelerated reliability test must be subject to the same standards of validation, verification, and certification as models that are used for evaluation of system effectiveness. (p. 124)
RECOMMENDATION 7.11 The Department of Defense should move aggressively to adapt for all test agencies the Organization for International Standardization (ISO) standards relating to reliability, availability, and maintainability. Great attention should be given to having all test agencies ISO-certified in their respective areas of responsibility for assuring the suitability of prospective military systems. (p. 125)
RECOMMENDATION 7.12 Military reliability, availability, and maintainability testing should be informed and guided by a new battery of military handbooks containing a modern treatment of all pertinent topics in the fields of reliability and life testing, including, but not limited to, the design and analysis of standard and accelerated tests, the
handling of censored data, stress testing, and the modeling of and testing for reliability growth. The modeling perspective of these handbooks should be broad and include practical advice on model selection and model validation. The treatment should include discussion of a broad array of parametric models and should also describe nonparametric approaches. (p. 126)
Innovations in Software Engineering for Defense Systems (2003)
This study covered statistically oriented software engineering methods that were useful for the development of defense systems.
RECOMMENDATION 1 Given the current lack of implementation of state-of-the-art methods in software engineering in the service test agencies, initial steps should be taken to develop access to—either in-house or in a closely affiliated relationship—state-of-the-art software engineering expertise in the operational or developmental service test service agencies. (p. 2)
RECOMMENDATION 2 Each service’s operational or developmental test agency should routinely collect and archive data on software performance, including test performance data and data on field performance. The data should include fault types, fault times and frequencies, turnaround rate, use scenarios, and root cause analysis. Also, software acquisition contracts should include requirements to collect such data. (p. 3)
RECOMMENDATION 3 Each service’s operational or developmental test agency should evaluate the advantages of the use of state-of-the-art procedures to check the specification of requirements for a relatively complex defense software-intensive system. (p. 3)
RECOMMENDATION 6 DoD needs to examine the advantages and disadvantages of the use of methods for obligating software developers under contract to DoD to use state-of-the-art methods for requirements analysis and software testing, in particular, and software engineering and development more generally. (p. 4)
Testing of Defense Systems in an Evolutionary Acquisition Environment (2006)
This study considered the methods that would be applicable to defense systems development that is implemented in stages.
- … revise DoD testing procedures to explicitly require that developmental tests have an operational perspective …,
- require (DoD) contractors to share all relevant data on system performance and the results of modeling and simulation …
Industrial Methods for the Effective Development and Testing of Defense Systems (2012)
This study examined the potential of the use of system and software engineering methods for defense system development.
The performance of a defense system early in development is often not rigorously assessed, and in some cases the results of assessments are ignored; this is especially so for suitability assessments. This lack of rigorous assessment occurs in the generation of system requirements; in the timing of the delivery of prototype components, subsystems, and systems from the developer to the government for developmental testing; and in the delivery of production-representative system prototypes for operational testing. As a result, throughout early development, systems are allowed to advance to later stages of development when substantial design problems remain.
This page intentionally left blank.