Statistics, Testing, and Defense Acquisition Background Papers (1999) / Chapter Skim
Currently Skimming:
Application of Statistical Science to Testing and Evaluating Software Intensive Systems
Application of Statistical Science to Testing and Evaluating Software Intensive Systems
Pages 124-170
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 124... ...
The term should be understood, however, as the comprehensive application of statistical science, including operations research methods, to solving the problems posed by industrial software testing. Statistical testing enables efficient collection of empirical data that will remove uncertainty about the behavior of the software intensive system and support economic decisions regarding further testing, deployment, maintenance and evolution.
|
From page 125... ...
From a statistical point of view, all of the topics in this paper follow sound problem solving principles and are direct applications of well-established theory and methodology. From a software testing point of view, the applications of statistical science discussed below are not in 125
|
From page 126... ...
This paper is a composite of what is in hand and within reasonable reach in the application of statistical science to software testing. Statistical testing based on usage models can be applied to large and complex systems because the modeling can be done at various levels of abstraction and because the models effectively allow analysis and simulation of use of the application rather than the application itself.
|
From page 127... ...
Usage models are constructed from specifications, user guides, or even existing systems. The "user" might be a human, a hardware device, another software system, or some combination.
|
From page 128... ...
The statistical testing process involves the six steps depicted in Figure 2. Builcling Usage Models An operational usage model is a formal statistical representation of all possible uses of a system.
|
From page 129... ...
The process is based on the systematic enumeration of sequences of inputs and leads to a complete, consistent, correct and traceable usage specification. Usage models are finite state, discrete parameter, time homogeneous, recurrent Markov chains.
|
From page 130... ...
Because transition probabilities represent classes of users, environments of use, or special usage situations, there may be several sets of probabilities for a single mode! structure.
|
From page 131... ...
Table 2 column 3 represents a model based on Figure 3 with uniform transition probabilities across the exit arcs of each state.
|
From page 132... ...
One approach to statistical testing is to estimate the operational profiles first and then create random test cases based on them. The usage model approach is to first build a model of system use (describe the stochastic process)
|
From page 133... ...
Each state of a usage mode! has a probability distribution across its exit arcs to describe the transitions to other states, which appears as a row of the transition matrix.
|
From page 134... ...
It is possible, also, to compare the differences in complexity associated with different environments of use (represented by different sets of transition probabilities, as in Tables 3 and 4~. Complexity analysis can be used to assess the impact on testing of changes in the requirements and system implementation.
|
From page 135... ...
Many well-posed questions involving states, arcs and paths can be expressed in a mathematical model with a closed form solution. Simulation All the analyses mentioned above have closed form solutions and are applicable to all usage models because of their mathematical structure.
|
From page 136... ...
An alternative to the cycle of setting probabilities, analyzing statistics and revising probabilities is to analytically generate models with stochastic matrices guaranteed to have certain statistics, as described in the next section.
|
From page 137... ...
(Walton, ~ 995~. Usage models can be represented by a system of A constraints and the matrix of transition probabilities can be generated as the solution to an optimization problem.
|
From page 138... ...
Most usage models can be defined with very simple constraints. Test Management Constraints Finally, constraints may be used to represent test management controls.
|
From page 139... ...
The problem is tractable when it is restricted to convex programming, the minimization of a convex objective function subject to a finite set of convex constraints. When mathematical programming is used to generate transition probabilities, the solution generated is optimized for some objective function while satisfying all structural, usage and management constraints.
|
From page 140... ...
Furthermore, testing effectiveness can vary greatly from one person to another unless steps are taken to assure uniformly effective testing. Every test is a traversal of a series of arcs through the model; if the scripts are granular and are followed, they will assure uniform testing.
|
From page 141... ...
A testing chain is a representation of testing experience. A testing chain is started by using just the structure of states and arcs (no transition probabilities)
|
From page 142... ...
The testing chain can represent all testing experience, special cases as well as random testing, or it can represent just random testing. It is possible to instrument code to maintain a "testing chain" based on actual field experience as well.
|
From page 143... ...
Usage models together with mathematical programming methods can be used to design testing to minimize the potential of economic loss from field failure (Sherer, 1996)
|
From page 144... ...
Guiding Special Test Situations Application of statistical science includes creating special, non-random, test cases. Such testing can remove uncertainty about how the system will perform in specific circumstances of interest, aid in understanding the sources of variation in the population, and contribute to effectiveness and control over all testing.
|
From page 145... ...
Testers may expect correct results because they have always been correct in the past; testers may learn the test set as well. If testing and the random test sets are independent of the developers and maintenance workers, reuse of the random test sets is a valid statistical testing strategy that can facilitate automated testing and substantial reductions in the time and cost of testing.
|
From page 146... ...
By a combination of additional management constraints and objective functions the resulting model can emphasize aspects ofthe system or ofthe testing process that are important to testers. The following are among the controls that are possible: · costs can be associated with each arc, and one can minimize cost; · value can be associated with each arc, and one can maximize value; · probabilities associated with exiting arcs that control critical flow can be manipulated; · certain long run effects can be regulated by constraints; · some entropy measures can be maximized to increase uncertainty and increase variability in the sequences; · some entropy measures can be minimized to reduce variability.
|
From page 147... ...
Sometimes the testing will be manual the first time as the correct behavior is recorded and then automated on subsequent runs of the test suite. There are several test runners on the market to which scripts can be sent and which will return information for constructing the testing chain.
|
From page 148... ...
. The failure states of the testing chain are made absorbing and the reliability of the system is defined to be the probability of going from the invocation state to the termination state without being absorbed in a failure state.
|
From page 149... ...
As the testing chain converges to the usage model, it becomes less likely that new information will be gained by further testing generated from the usage model. Simulation Testing can be simulated by generating the test cases and marking the testing chain as if the test cases has been run.
|
From page 150... ...
Incremental Development The Cleanroom software engineering process (Poore and Trammell, 1996) uses the testing approach described in this paper.
|
From page 151... ...
The historical chain may be used to review the development and testing processes across increments. The historical testing chain and the collection of testing chains from version to version can be used to assess reliability growth.
|
From page 152... ...
Furthermore, a common basis for describing the testing done and for interpreting the data is necessary. Usage models have the potential to be the common denominator for test planning and evaluation of results in all testing situations in the system life cycle.
|
From page 153... ...
One would have the original usage model and testing records (testing chains and statistics) , plus field use summarized as an estimate of sequences actually run.
|
From page 154... ...
Usage states and arcs can be associated with underlying changes in the technology. Usage models might be used to assess the extent of change and to guide testing; the greater the change, the harder the testing problem.
|
From page 155... ...
Since ~ 992, the IBM Storage Systems Division has developed and applied Markov chain usage models for certification of tape drives, tape controllers, tape libraries, disk drives and disk controllers. Some products are tested based on up to five different usage models, including models of customer use, a data communication protocol model, a model keyed to the injection of hardware and media failures, and a stress model.
|
From page 156... ...
In general, all the time and effort spent in modeling, analysis, and simulation will be recouped with dividends in the avoidance of uninformative testing. Usage models facilitate the application of many aspects of statistical science to software testing, because (~)
|
From page 157... ...
The economics of software failures in the field and the cost of testing complex systems make the use of statistical science imperative to software testing.
|
From page 158... ...
- statistically valid generalization FIGURE ~ Parallel between statistical inference and software testing. ~ Population W statistically correct selection ~ Sample ~ \ All Possible Uses \ of a Software System randomly generated test cases / inference about fielbl performance from testing (Test Cases ~ 158
|
From page 159... ...
FIGURE 2 The statistical testing process. l l Usage Modeling l | | Model Analysis and Valid action Test Planning Testing Product and Process Measurement Software Certification 159
|
From page 160... ...
FIGURE 3 Example usage mode! structure, directed graph format.
|
From page 161... ...
TABLE ~ Example of a Usage Mode' Structure, Transition Matrix Format To State 4 6 From 8 State 9 10 12 13 14 15 16 17 1 2 3 4 5 6 7 8 .
|
From page 162... ...
Mode-2 Service Decision Mode-2 Service Decision (11) Mode-2 Service Decision ~ Mode-2 Service Decision (11)
|
From page 163... ...
Termination Mode-2 Checkpoint 1/7 Mode-2 Service Decision Mode-2 Service Decision Mode-2 Service Decision (16) Mode-2 Checkpoint (11)
|
From page 164... ...
TABLE 3 Usage Statistics for the Mode' with Uniform Probabilities on the Exit Arcs Expected Expected Expected Prob. of Number of Number of Number of State Long Occurrence Occurrences Transitions Sequences Identification Run in a Single in a Single Until Until Number Probabilities Sequence Sequence Occurrence Occurrence 1 0.0449 1.0000 1.00 22.25 1 2 0.1348 1.0000 3.00 1.00 1 3 0.0449 0.5000 1.00 22.25 1 4 0.0449 0.5000 1.00 19.25 1 5 0.0749 1.0000 1.67 9.00 1 6 0.0225 0.3333 0.50 42.50 2 7 0.0449 0.5000 1.00 21.25 1 8 0.0449 0.5000 1.00 22.25 1 9 0.0225 0.3333 0.50 42.50 2 10 0.0674 0.7500 1.50 16.67 1 11 0.2097 1.0000 4.67 10.75 1 12 0.0300 0.4000 0.67 43.12 2 13 0.0300 0.4000 0.67 43.12 2 14 0.0300 0.5000 0.67 36.75 2 15 0.0637 0.6538 1.42 21.53 1 16 0.0449 1.0000 1.00 20.25 1 17 0.0449 1.0000 1.00 21.25 1 164
|
From page 165... ...
Expected number of sequences to cover the least likely state is approximately 3. Expected number of sequences to cover the least likely arc is approximately 7 The log base 2 source entropy is approximately 1.0197.
|
From page 166... ...
TABLE 4 Usage Statistics of Mode] for Specific Environment Prob.
|
From page 167... ...
The log base 2 source entropy is approximately 0.9169. The specification complexity index is approximately 36.68 (or I.!
|
From page 168... ...
Gutjahr, W.~. 997 Importance sampling of test cases in Markovian software usage models.
|
From page 169... ...
1996 Statistical software testing using economic exposure assessments. Software Engineering Journal September:293-297.
|
From page 170... ...
Walton, G.H. 995 Generating Transition Probabilities for Markov Chain Usage Models.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.