Potential Failures and Disasters for DOD's M&S
Despite the dramatic success of M&S and the fact that M&S will surely “take off” in the commercial sector, the potential of M&S for the Department of the Navy and DOD may not be realized in the foreseeable future. Some of the DOD's most important and expensive M&S efforts may fail or—perhaps worse— end up saddling the DOD components with mediocre and inflexible tools that impede innovation and improvement of content. There is also the potential for disasters due to overdependence on M&S that appear more valid than they actually are (the Spielberg effect). This could cost lives and undercut military operations.
This chapter focuses on two causes of concern: (1) an inadequate research base and (2) the inherent complexity of emerging military systems and operations.
INTELLECTUAL AND TECHNOLOGICAL INFRASTRUCTURE
Strong Technological Base
One basic problem is that DOD's high-visibility work on M&S is dominated by content-neutral computer and software technologies such as object-oriented programming, high-performance computing, computer-aided design, and establishing the communications protocols and infrastructure for everything from collaborative, distributed, multidisciplinary, simulation-based engineering design to large-scale distributed war games. It is in these technologies that there has been
dramatic progress, which will assuredly continue because of commercial developments and DMSO efforts.
Inadequate Knowledge Base
In contrast, there has been curiously little investment in the knowledge base determining the substantive content and quality of much M&S—particularly higher-level M&S needed for mission- and campaign-level work. It is an open secret and a point of distress to many in the community that too much of the substantive content of such M&S has its origin in anecdote, the infamous BOGSAT (bunch of guys sitting around a table), or a narrow construction tied to stereotypical current practices of “doctrinally correct behavior.” There is a need for focused research on the phenomena of combat and other military activities, both historical and prospective. This is the realm of military science.
Another shortfall in knowledge relates to theories and methods for conceiving, designing, and building models (as distinct from software). Symptoms of the problem are evident if one observes that DOD's M&S often consists of nothing more than the computer code itself: there is no separable documented “model” to be reviewed and improved, nor any way to readily understand the assumptions generating the simulation 's behavior. This can hardly be a comfortable basis for decision support.
COMPLEX SYSTEMS AND THE NEED FOR HUMILITY
The inherent complexity of the systems and force operations that DOD is attempting to simulate introduces new difficulties ( Appendix B ). Throughout its efforts, the panel was concerned by the degree to which many forecasts are extrapolating unreasonably from the Boeing 777 experience and from M&S successes in lower-echelon training to imagined systems of extraordinary complexity. Generally speaking, the types of complexity being considered break into three distinct, but not wholly unrelated cases:
Localized systems. These are systems that are highly complex, but are designed from inception as one system. Examples could be an aircraft or a ship, as well as a very large scale integrated circuit.
Systems of systems. These are distributed (typically information) systems for which the final overall configuration is not known during design of the individual components. In fact, overall system configuration often varies depending on the particular application or circumstances.
Combat operations. Here the complexity is due to the interaction of multiple combat systems and forces with one another and with the physical environment in which they are operating.
Significant increases in complexity are occurring in each of these cases. For example, the software in modern weapon systems is perhaps 100 to 1,000 times larger than in weapon systems one or two generations removed, now often totaling over 1 million lines of code. Distributed network systems such as the Internet have become so extensive that the causes of network outages often take hours to ascertain. While anticipated combat operations have become smaller than those in cold war scenarios, the operations themselves have become more complex in the sense that far greater quantities of information are being exchanged and interaction between units can be much more coordinated (e.g., in combining the effects of a small number of dispersed forces to have a large cumulative effect).
This increased complexity means that it will be harder to design systems and to predict their behavior. Some might argue that in the engineering domain, modern design tools should overcome these difficulties. While there certainly have been impressive advances (e.g., in computer-aided design), there is a feeling with at least some members of the engineering community interviewed in the course of the study that they have nearly reached the limits of complexity that can be addressed with current tools and methods. Challenging examples that could lie beyond current approaches include future generation networks and very large scale integrated circuits. Indications of the difficulty of building complex engineering systems are given by some of the well-known “disasters ”—explosion of the Ariane missile, inability to build a next-generation air-traffic control system, outages in telephone and power systems, and even the problems with the baggage handling system at the Denver airport.
What this means from the perspective of M&S is that, if the systems are harder to design and build, then it certainly will be harder to model and simulate them. But, on the other hand, this makes the need for M&S all the greater in designing systems and predicting their behavior. Some of the factors that pose particular problems for the M&S of complex systems are as follows:
Scale. As the number of elements in a system increases, or as the dimension or resolution of the space in which it is considered increases, the time to perform relevant calculations (e.g., seek optimal behaviors) can increase as a polynomial or even exponential of the number of these elements.
Nonlinearity. A given component in a system influences a second, which influences a third, which interacts back on the first, and so forth. This behavior obviously becomes highly complicated as the number of components increases. Such interaction can lead to nonintuitive results.
Heterogeneity. The different components are of a fundamentally different nature or the modeling techniques that must be brought to bear to fully understand a system involve different disciplines (e.g., structural mechanics, aerodynamics, human factors, and propulsion in the study of aircraft performance). This means establishing interrelationships among the disciplines, which can be complicated by inconsistent formalisms and assumptions among them.
New approaches in M&S will most likely be required to address these factors. Two particular points are of note. First is the matter of interaction. For systems with large numbers of components, often the sheer nature of the interaction between the components becomes the dominant factor, rather than the detailed properties of the components. An example is given by the transportation simulation (TRANSIMS) developed at Los Alamos National Laboratory. Realistic, large-scale automobile traffic behavior is predicted by using an approach that describes driver behavior with only simple interaction rules. This result is perhaps counterintuitive to traditional approaches that would have focused on describing individual entities (the automobiles and drivers) in great detail to get realistic traffic behavior. The general implication is that high-resolution description of components may not always be the appropriate way to model the behavior of complex systems involving a great many components. Rather, a formalism must be developed that gives equal (and perhaps greater) weight to describing interaction as it does to describing the properties of the individual entities. 1
The second matter is that of uncertainty. No matter how careful one is in modeling a system, there always will be uncertainties. Within single disciplines, extensive effort has been devoted to characterizing and understanding the implications of uncertainties. The matter becomes particularly acute when multiple disciplines are involved, where the results of a model from one discipline are used in conjunction with those from another discipline. The issue is that there is not, in general, an understanding of how the uncertainties from the first model will influence the predictions of the second model, and so forth as even more disciplines are involved. This propagation of uncertainties could lead to highly erroneous results. What is needed is a formalism, preferably a domain-independent one, that would allow the characterization of the propagation of uncertainties.
In an important sense, one can say that complexity and uncertainty are closely related. Namely, it is the desire to reduce uncertainty that can lead to complexity. For example, the complexity of a modern air-to-ground attack missile, in terms of the number and interaction of its sensing and guidance components, results from the need to reduce the uncertainty in the location of the target that is being attacked. Likewise, the complexity of a large-scale intelligence system, in terms of its data collection and processing components, results from the need to reduce uncertainty about the enemy. Many engineering disciplines take this perspective—control theorists construct feedback mechanisms to cope with uncertainties in the performance of actual systems, and statisticians devise sampling schemes to reduce uncertainties in measured properties. Given the fact that treatment of uncertainty is central to many individual disciplines, the hope is that a more
This perspective could have profound implications for combat modeling, where the tendency often is to want to go to greater and greater levels of resolution of the entities. Preliminary explorations of this perspective are given in Ilachinski (1996a,b).
comprehensive and powerful approach to treating it can be established by having these disciplines collaborate in addressing it.
As the situation now stands, these matters of interaction and uncertainty are daunting challenges. But, as noted, the difficulty of designing and predicting the performance of complex systems makes the need for M&S of such systems all the greater.