Innovations in Software Engineering for Defense Systems (2003) / Chapter Skim
Currently Skimming:
3 Testing Methods and Related Issues
3 Testing Methods and Related Issues
Pages 20-39
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 20... ...
INTRODUCTION TO MODEL-BASED TESTING Two of the leading statistically oriented methods for the selection of inputs for software testing are forms of model-based testing, and therefore it is useful to provide a quick overview of this general approach. Prior to the development of model-based testing, the general approaches used for software testing included manual testing, scripted automation, use of random inputs, finite state models, and production gram20
|
From page 21... ...
One feasible alternative is to efficiently choose test inputs with associated graphical paths that collectively contain every arc between nodes in the graphical model. There are algorithms that carry out this test plan; random walks through the graph 1One side benefit of developing a graphical model of the functioning of a software system is that it helps to identify ambiguities in the specifications.
|
From page 22... ...
When they exist, test oracles are placed at each node of the graph to permit comparison of the functioning of the current system with what the output of the correct system would be. Through use of an oracle, discrepancies between the current state of the system and the correct system at various nodes of the graph can be identified.3 Graphical models are initially developed at a very early stage in system development, during the identification of system requirements.
|
From page 23... ...
Model: Play Billing Prompt Play / billing prompt \ ,,~3 Play billing ~ prompt i~3 | Float call | ~ ~3\ - | Credit card | Collect I Session end I | Release call | ~— | Enter card number | Billing data verified I No input required FIGURE 3-1 Example of a model used in model-based testing of long-distance telephone flows: (a) high-level model and (b)
|
From page 24... ...
, often referred to as combinatorial design.4 MARKOV CHAIN USAGE MODELS Markov chain usage modeling was described at the workshop by Jesse Poore of the University of Tennessee (for more details, see Whittaker and Poore, 1993, and Whittaker and Thomason, 19941. Markov chain usage models begin from the graphical model of a software program described 4We note that SCR can also be extended to automatic test set generation, albeit in a very different form than discussed here.
|
From page 25... ...
of a program. The Markov chain usage model randomly samples paths from this population using the transition probabilities, which are typically obtained in one of three ways: (1)
|
From page 26... ...
However, these two assumptions can often be made approximately true by adjusting the graphical model in some way. Furthermore, even if the time homogeneity and conditional independence assumptions are violated to a modest extent, it is quite likely that the resulting set of test inputs will still be useful to support acquisition decisions (e.g., whether to release the software)
|
From page 27... ...
described some of the tools that have been developed to implement Markov chain usage models. The Model Language (TML)
|
From page 28... ...
The results of applying Markov chain usage-basecl testing to eight major software development projects were a greater than tenfold reduction in clefects per 1,000 lines of cocle, software development costs within buclget, ancl expedited system integration. For Raytheon's systems, it was found that automated test oracles could be developed to facilitate automated testing.
|
From page 29... ...
Finally, while development of an automated test oracle did represent a major expense, it could be justified by the reductions in time and costs of development that resulted from the implementation of this methodology. In addition to Markov chain usage-baseu testing, Raytheon initially augmented its testing with purposive sets of test inputs to cover all nodes and arcs of the graphical model for each system under test.
|
From page 30... ...
For example, in the pairwise case, for input fields Iand I, there will exist in the set of test inputs at least one input that has value i for field Iand value j for field I, for every possible combination of i and j and every possible combination of two fields Iand ~ For invalid values, a more complicated strategy is utilized. A key empirical finding underlying this methodology is that in many applications it is the case that the large majority of software errors are expressed through the simultaneous use of only two or three input values.
|
From page 31... ...
AETG was therefore used to identify a small number of test events that included inputs containing all pairwise combinations of input values for pairs of input fields (while observing the various constraints)
|
From page 32... ...
laying into conslc aeration some ac ( Tonal complications not mentionecl here, the AETG strategy provided an overall 68 percent savings in test duration ancl a 67 percent savings in test labor costs. INTEGRATION OF AETG AND MARKOV CHAIN USAGE MODELS For a graphical model with limited choices at each node ancl for a relatively finite number of nocles, Markov chain usage model testing is a methodology that provides a rich set of information to the software tester along with an efficient method for selecting test inputs.
|
From page 33... ...
Mike Houghtaling of IBM presented a number of automation tools that the company has used in managing its tape systems and libraries. The company uses Markov chain usage models comprising roughly 2,000 states, which is difficult to represent on paper.
|
From page 34... ...
In addition, designers and implementers of test automation tools employing high-level usage models need to be aware of the gap between the usage model vocabulary and the more concrete implementation vocabulary that is meaningful to the developers of the system. Attempts to document justifications for claims about test case failures will need to bridge the gap.
|
From page 35... ...
A session at the workshop addressed this critical issue given its importance in Department of Defense applications; the presentations examined general tools to avoid interoperability problems in system development, and how one might test the resulting system of systems for defects. Amjad Umar of Telcordia Technologies discussed the general interoperability problem and tools for its solution, which he referred to as integration technologies.
|
From page 36... ...
With respect to low-level integration between two systems, interconnection technologies might work as a mediator between the order processing, provided by a Web browser or lava apples, and the inventory, managed by user interfaces, application code, and a data source. Interconnection technologies may involve a remote user connector, a remote method connector, and a remote data connector.
|
From page 37... ...
There were additions and deletions of roughly 30 rules a day, and there was on average one new software component release per day. As a result of these dynamics, 4,000 errors in processing occurred in 6 months, but, using traditional testing methods, only 47 percent of the defects ultimately discovered were removed during testing.
|
From page 38... ...
Finally, possible subsystems causing problems are identified using an automated root cause analysis. This entire 5A tag in XML identifies and delimits a text field.
|
From page 39... ...
Based on the various methods discussed during this session, the panel concluded that model-based testing offers important potential for improving the software testing methods currently utilized by the service test agencies. Two specific methods, Markov chain usage-based testing and AETG, were described in detail, and their utility in defense or defense-type applications was exhibited.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.