Improved Operational Testing and Evaluation Better Measurement and Test Design for the Interim Brigade Combat Team with Stryker Vehicles: Phase I Report (2003) / Chapter Skim
Currently Skimming:
5. Data Analysis
5. Data Analysis
Pages 76-86
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 76... ...
This is necessary to ensure that the designed data collection effort will provide enough information of the right types to allow for a fruitful evaluation. Failure to think about the data analysis prior to data collection may result in omitted explanatory or response variables or inadequate sample size to provide statistical support for important decisions.
|
From page 77... ...
The remainder of this chapter reviews the general principles behind the formal statistical procedures used in confirmatory studies and those methods used in exploratory statistical analyses and then presents some specific recommendations for data analysis for the IBCT/Stryker JOT. PRINCIPLES OF DATA ANALYSIS Formal Statistical Methods in Confirmatory Analyses A key component of any defense system evaluation is the formal comparison of the new system with an appropriately chosen baseline.
|
From page 78... ...
If it is rejected, then the main goal of the evaluation is achieved, and the data analysis may move to an exploratory phase to better understand when and why the new system is better. A difficulty with the binary decision is that it obscures information about the size of the improvement afforded by the new system, and it does not recognize the difference between statistical significance and practical significance.
|
From page 79... ...
Multiple comparison procedures allow for control ofthe experiment-wide error rate by reducing the acceptable error rate for each individual comparison. Because this makes the individual tests more conservative, it is important to determine whether formal significance tests are required for the many outcome measures.
|
From page 80... ...
The remainder of this section assumes that this important part of exploratory analysis has been carried out prior to the use of formal statistical tests and procedures. The focus here is on another crucial use of exploratory methods, namely, to identify data patterns that may suggest previously unseen advantages or disadvantages for one force or the other.
|
From page 81... ...
One might identify combinations of measures that appear to correlate well with the ratings of SMEs, in this way providing a form of objective confirmation of the implicit combination of information done by the experts. Reliability and Maintainability These general comments above regarding confirmatory and exploratory analysis apply to all types of outcome measures, including those associated with reliability and maintainability, although the actual statistical techniques used may vary.
|
From page 82... ...
Confirmatory Analysis ATEC has provided us with detailed plans describing the intended analysis of the SME scores of mission outcomes and mission casualty rates. These plans are discussed here.
|
From page 83... ...
Confirmatory analyses are often based on the assumptions that there is a continuous or at least ordered categorical measurement scale (although they are often done with Poisson or binomial data) and that the measurements on that scale are subject to measurement error that has constant variance (independent of the measured value)
|
From page 84... ...
The model would be likely to be of the form Di= difference score for mission i = overall mean + mission type effect + mission intensity effect + location effect + company effect + other desired .
|
From page 85... ...
Statistical significance tests are useful in the confirmatory analysis comparing the Strykerequipped and baseline forces. In general, however, the issues raised by the 1998 NRC panel suggest that more use should be made of estimates and
|
From page 86... ...
This highlights the need for improved data collection regarding vehicle usage. In particular, data should be maintained for each vehicle over that vehicle's entire life, including training, testing, and ultimately field use; data should also be gathered separately for different failure modes.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.