Skip to main content

Statistical Methods for Testing and Evaluating Defense Systems Interim Report (1995) / Chapter Skim
Currently Skimming:

Appendix C: Selecting a Small Number of Operational Test Environments
Pages 62-71

The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.


From page 62...
... The combination of information from field tests and simulations is not addressed in this interim report, but is expected to be addressed in the panel's final report. Because this problem was suggested to the panel for study by Henry Dubin, Technical Director of the Army Operational Test and Evaluation Command, we refer to it as Dubin's challenge.
Read the entire page →
From page 63...
... Similarly, the defense testing community might gain valuable insights by approaching their duties from a more formal statistical perspective for example, by considering the concepts and tools of experimental design and their implications for current practice. In this spirit, we present the ideas in this appendix to members of both communities.
Read the entire page →
From page 64...
... = ,|~ ~a', - a~i.,'2 Table C-2 presents the resulting distance matrix. Under the constraints on sample size often encountered in operational testing, the tasks of optimally selecting environments for testing and, subsequently, estimating performance in all environments are extremely difficult unless the environments can be characterized as functions of a very small number of core factors.
Read the entire page →
From page 65...
... temperate rural ground level 6. temperate rural hilly 7.
Read the entire page →
From page 66...
... We point out that entries in the distance matrix D may not be as easily obtained as in our artificial example and may depend partly on subjective intuitions of the users. Also, there may be no natural dimension for the reduced environment space; instead, one could consider different values of k to find the lowest dimension in which the Euclidean distances between points are still consistent (monotonically)
Read the entire page →
From page 67...
... 1 -0.89 -2.47 3 5 15 2 -7.88 -6.95 2 3 6 3 7.03 -5.25 1 1 1 4 10.06 -0.08 1 1 1 5 -3.22 0.52 5 5 25 6 -1.81 4.90 3 5 15 7 2.80 5.40 1 2 2 8 -6.08 3.93 5 5 25 NOTE: The second and third columns are the coordinates in the reduced 2-dimensional environment space. The next three columns contain the weights representing the strategic importance and the prior expected frequency of the environment, and the overall weight (product of the strategic and frequency weights)
Read the entire page →
From page 68...
... In creating this stress matrix, one runs the substantial risk of ignoring the possibility of the illuminating surprises that accompany operational testing, i.e., failure modes that are much more typical of operational than developmental testing. These surprises are not incorporated in this model.
Read the entire page →
From page 69...
... We fixed the dimension of the reduced environment space at k = 2. In practice, multidimensional scaling methods suggest a particular dimension k by minimizing a so-called "stress" criterion (different from our previous use of the term "stress"- a type of goodness-of-fit criterion which can be used to measure the quality of the approximation of the high-dimensional space by the lower-dimensional one.
Read the entire page →
From page 70...
... ~2 .25 -I .35 STATISTICAL METHODS FOR TESTING AND EVALUATING DEFENSE SYSTEMS to .6 .8 .4 ~ + 1.5 .45 (if) + 4.0 .6- 2 + O .8 - 1~0 4.0 4 1.5= ·2+5 .45 .3 O1 02 .4 ~35 + + So -5 0 5 10 x1 FIGURE C-1 Optimal test environments as a function of b.
Read the entire page →
From page 71...
... This result is indicated by a "+" in the circle labeled 5 and a "-" in the circle labeled 8. As b is increased, the environments become less comparable, and the algorithm correctly selects experimental environments that are compromises of all eight environments of interest.
Read the entire page →

Key Terms

  • multidimensional scaling
  • core factors
  • test environments
  • euclidean space
  • stress variables

  • generator example
  • strategic importance
  • stress matrix
  • environment space
  • temperate rural

  • highly weighted
  • distance matrix
  • reduced environment
  • particular operating
  • underlying factor

  • defense testing
  • operational test
  • operating environments
  • allocation scheme
  • eight environments

  • average temperature
  • subjective assessments
  • overall weight
  • developmental testing
  • operating environment

  • average performance
  • scenario structure
  • factorial experiments
  • related complexities
  • intuitive interpretations


    • This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
    More information on Chapter Skim is available.