Improved Operational Testing and Evaluation Better Measurement and Test Design for the Interim Brigade Combat Team with Stryker Vehicles: Phase I Report (2003) / Chapter Skim
Currently Skimming:
6. Assessing the IBCT/Stryker Operational Test in a Broad Context
6. Assessing the IBCT/Stryker Operational Test in a Broad Context
Pages 87-96
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 87... ...
From a practical point of view, it is clear that several of the ideas put forward in NRC 1998 for improvement of the measures and test design cannot be implemented in the IBCT/Stryker IOT due to various constraints, especially time limitations. However, by viewing the Styker test as an opportunity to gain additional insights into how to do good operational test design and evaluation, our panel hopes to further sharpen and disseminate the ideas contained in NRC 1998.
|
From page 88... ...
systems and the heterogeneity of the performance of these systems across environments of use, users, tactics, and doctrine, operational tests cannot, generally speaking, satisfy this role.1 Instead, the test and evaluation process should be viewed as a continuous process of information collection, analysis, and decision making, starting with information collected from field experience of the Conclusion 2.2 of the NRC 1998 report states: "The operational test and evaluation requirement, stated in law, that the Director, Operational Test and Evaluation certify that a system is operationally effective and suitable often cannot be supported solely by the use of standard statistical measures of confidence for complex defense systems with reasonable amounts of testing resources" (p.
|
From page 89... ...
The DoD milestone process must be rethought, in order to replace the fundamental role that significance testing currently plays in the pass/fail decision with a fuller exploration of the consequences of the various possible decisions. Significance tests and confidence intervals2 provide useful information, but they should be augmented by other numeric and analytic assessments using all information available, especially from other tests and trials.
|
From page 90... ...
This maturation should be expedited through previous testing that incorporates various aspects of operational realism in addition to the usual developmental testing. The role, then, for operational testing would be to confirm the results from this earlier testing and to learn more
|
From page 91... ...
and (6) use of small-scale screening or guiding tests for collecting information on test planning." Also, as mentioned in Chapter 4, it is not yet clear that the test design and the subsequent test analysis have been linked.
|
From page 92... ...
491: "All services should explore the adoption of the use of smallscale testing similar to the Army concept of force development test and experimentation. " Modeling and simulation are discussed in NRC 1998 as an important tool in test planning.
|
From page 93... ...
1051: The Department of Defense and the military services should give increased attention to their reliability, availability, and maintainability data collection and analysis procedures because deficiencies continue to be responsible for many of the current field problems and concerns about military readiness. While criticizing developmental and operational test design as being too focused on evaluation of system effectiveness at the expense of evaluation of system suitability, this recommendation is not meant to suggest that operational tests should be strongly geared toward estimation of system suitability, since these large-scale exercises cannot be expected to run long enough to estimate fatigue life, etc.
|
From page 94... ...
Making use of the information on situation awareness collected during training exercises and in contractor and developmental testing in the operational test design would have helped in the more comprehensive assessment of the performance of IBCT/Stryker. For example, allocating test replications to situations in which previous difficulties in situation awareness had been experienced would have been very informative as to whether the system is effective enough to pass to full-rate production.
|
From page 95... ...
We understand ATEC's view that its operational test designs must allocate, to the extent possible, replications to environments in accordance with the allocation of expected field use, as presented in the OMS/MP. In the panel's judgment, the OMS/MP need only refer to the operational evaluation, and once estimates of test performance in each environment are derived, they can be reweighted to correspond to summary measures defined by the OMS/MP.
|
From page 96... ...
Forwarding an immature system to operational test is an expensive way to discover errors that could have been detected in developmental testing, and it reduces the ability of an operational test to carry out its proper function. System maturation should be expedited through previous testing that incorporates various aspects of operational realism in addition to the usual developmental testing.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.