Skip to main content

Currently Skimming:

Executive Summary
Pages 1-6

The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.


From page 1...
... intends to consider further in the remainder of the study. KEY ISSUES Experimental Design The goal of an operational test is to measure the performance, under various conditions, of diverse aspects of a newly developed DoD system to determine whether the system satisfies a number of criteria.
From page 2...
... We therefore recognized the need for special attention to software-intensive systems and have sought to understand how operational testing of such systems is conducted across the military services. On the basis of our work to date, we note several concerns about current practice: · Barriers to effective software testing.
From page 3...
... Therefore, the primary question concerns the extent to which simulations, possibly with some adjustments and enhancements, can be used for testing purposes, with the important objectives of saving limited test funds, enhancing safety, effectively increasing the sample size of a test, and possibly permitting the extrapolation of test results to untested scenarios. Although we applaud efforts made throughout DoD to make operational testing more cost-effective through the use of simulation, we have identified several concerns: 1We use the term "simulation" to mean both modeling and simulation.
From page 4...
... Operational test results can be affected significantly by the degree of training that soldiers receive prior to testing and by player learning that occurs during the test. Therefore, we are examining the potential use of statistical methods, including trend-free and other experimental designs, to address and correct for these confounding effects.
From page 5...
... The panel believes that statistical methods can and should play a role in the testing and evaluation of software-intensive systems, particularly because not every user scenario can be tested. Consequently, in order to understand current practice in operational testing of software-intensive systems and how statistical methods might be applied, we continue to seek answers in this specific context to a set of general questions related to the statistical design and analysis of experiments: How does one characterize the population of scenarios to test and the environments of use?
From page 6...
... In assessing how to make optimum use of best statistical practices in operational testing, we have repeatedly been led to consider various aspects of the larger acquisition process of which operational testing is a part. For example, starting operational testing earlier in the acquisition process an idea that has won support in the DoD community and among members of our pane} has implications for how statistical methods would be applied.


This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More information on Chapter Skim is available.