Behavioral Modeling and Simulation From Individuals to Societies (2008) / Chapter Skim
Currently Skimming:
8 Common Challenges in IOS Modeling
8 Common Challenges in IOS Modeling
Pages 271-328
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 271... ...
, issues that are especially challenging for the modeling of human behavior. Finally, we discuss some of the challenges posed by the data requirements of IOS models in light of the realities of the data and information available to model developers and users.
|
From page 272... ...
, then some aggregation process must be conducted, usu Much of the work described in this section on model integration and interoperability was performed by John Langton and Subrata Das at Charles River Analytics with support from the Air Force Research Laboratory, Information Directorate (AFRL/IF) under contract FA875006-C-0076, and adapted from Langton and Das (2007)
|
From page 273... ...
p(X) Incompatibility Subdomain Gaps Economy Social FIGURE 8-1 Illustration of gaps and incompatibilities between IOS models.
|
From page 274... ...
A number of approaches can be used to resolve the ontological incompatibility, described below. Formalism Incompatibility While ontological incompatibility creates problems due to multiple ways of designating an entity, the formalism incompatibility shown in F igure 8-1 is concerned with multiple ways of instantiating the object entity An ontology, for the purposes discussed here, is "a systematic arrangement of all of the important categories of objects or concepts which exist in some field of discourse, showing the relations between them.
|
From page 275... ...
Conversion between two such formalisms often requires deep understanding of the models and their formalisms, thus breaking the simple I-O black box idea of encapsulation. Specialization of formalism is often appropriate to map one approach to another.
|
From page 276... ...
High Level of Anger Level of Anger 276 Medium Loss Influence Diagram Among Population Among Population Low Model Fragment of Social Model in Town Aligned Drinking Refined Refined Sufficient with the USA Drinking Power Power Water Fuel Fuel Food Supply Water Drinking Power Water Refined Fuel Power Industrial Oil Oil Power Fragment of Substation Water Plant Field Refinery Generators Infrastructure Industrial Refined Power Crude Model Water Fuel High Voltage Power SRO Model Refined Fuel Terrorist Group A Leads Leader X Angers 8-2.eps Attr Attr Attr Concept Graph Fragment of Behavioral Model Information Model in Town with Terrorist Stronghold Aggressive Diplomatic Quick to Anger Attr Attr Attr Causes Imminent Attack in landscape view, smaller type is 5.73 pt Use of Threatening Calling for Inviting Suicide Observable Intelligence Phrases Jihad Bombers FIGURE 8-2 Interoperability of three different PMESII models. SOURCE: Langton and Das (2007)
|
From page 277... ...
The dynamics of the social model are that short supply in any one of these three consumable products will increase the level of anger among the local population. In fact, if a terrorist organization became aware of the mid-layer SRO model sequence in the infrastructure, then the power substation would assume heightened importance in the eyes of the terrorist strategists: an attack on a substation would not only cripple other services in the loop, but would also drive the sentiment of the local population against the coalition.
|
From page 278... ...
would allow users to explicitly modify, add, and remove interface translation functions, as illustrated in Figure 8-3. Users could also specify these translation functions within an ontology or the XML schema of a model, based on specifications derived, for example, from an evolved, global ontology.
|
From page 279... ...
. XML schemas often exist to support model file persistence.
|
From page 280... ...
Dealing with Ontological Incompatibility Ontological incompatibility refers to two models having different structures, including the entities they specify and the relationships between them. For instance, a rules system model may have several pairs of nodes connected by one link (precedent and consequent)
|
From page 281... ...
For example, consider α to be the node airport and N to be the subset of nodes runway, plane, radar, and air traffic control. The question is, which nodes should airport be mapped to for model integration?
|
From page 282... ...
. We can then deduce that these nodes should be mapped for model integration.
|
From page 283... ...
TABLE 8-2 Mappings Between Modeling Formalisms Bayesian Dempster- Fuzzy Possibilistic Certainty Symbolic Probability Shafer Logic Theory Factor Dictionary Bayesian X Generalization Membership Transformations Mapping from Probability probability degree based on consistency probabilities to to-symbolic interpretation of principles certainty factors mapping probabilities Dempster- Bayesian X Via Bayesian Via Bayesian Via Bayesian Belief value Shafer approximation approximation approximation approximation to-symbolic (transferable belief mapping model) Fuzzy Normalization of Via normalization X Possibility measure Mapping from Membership logic membership degrees interpretation of membership degree-to membership degrees degree to certainty symbolic factors mapping Possibilistic Transformations Belief Membership X Mapping from Possibility theory based on consistency interpretation degree possibility measure principles of possibility interpretation measures to to-symbolic measures of possibility certainty factors mapping measures Certainty Normalization Via normalization Via normalization Via normalization X Certainty factor factor-to symbolic mapping Symbolic Symbolic-to- Symbolic-to-belief Symbolic-to- Symbolic-to- Symbolic-to- X dictionary probability mapping value mapping membership possibility measure certainty factor degree mapping mapping mapping SOURCE: Langton and Das (2007)
|
From page 284... ...
In summary, there are no currently agreed-upon and widely used standards for model integration and interoperability. The field of IOS modeling is fragmented, with models being developed from different perspectives, at different levels of detail, and using different theoretical frameworks and architectures.
|
From page 285... ...
frameworks and toolkits that attempt to address some of the more practical issues of model development, verification and validation (V&V; see following section for more complete discussion) , and integration across modeling concepts and instantiated simulations.
|
From page 286... ...
286 8-4.eps bitmap image landscape for legibility FIGURE 8-4 OneSAF product line architecture framework. SOURCE: See http://www.peostri.army.mil/CTO/FILES/SmithR_GeneralFrameworkInterop.pdf, p.
|
From page 287... ...
As can be seen, the IDEs are very specific to each cognitive modeling paradigm; can range from highly generic programming language development environments (e.g., CLOS) to very specific model development environments (e.g., iGEN)
|
From page 288... ...
OmarJ provides most of the features of OmarL with an improved external communication layer that uses Jini for internode communication and the ability to break out of simulation mode and run agents in a non-time-controlled environment EPIC • IDEs associated with original LISP version of EPIC and with current C++ version SAMPLE AgentWorks™ AgentWorks™ consists of: • Perceptual, cognitive, and communications modules including neural networks, fuzzy logic, Bayesian belief networks, expert systems, and argumentation engines • Advanced processing capabilities supporting planning, learning, and distributed applications • Enhanced usability components for construction, validation, and visualization of agent processes Soar SDB • Soar Debugger (SDB) is an XDB-like debugger for the Soar programming language, including functionality, such as deep structure inspection, watches, and breakpoints and a graphical interface to common Soar commands
|
From page 289... ...
, a bundled software environment and methodology for organizational design • CONNECT (http://www.cra.com) , a social network analysis tool for organizational modeling and simulation • DDD (Distributed Dynamic Decision-making; http://www.aptima.
|
From page 290... ...
From left to right, the figure shows a causal analysis in which, given a set of possible DIME actions to be taken, a system of complex and integrated behavior models is used to predict the potential effects those DIME actions may have across the PMESII dimensions. From right to left, the figure shows a diagnostic analysis in which, given a set of desired PMESII effects in the operational domain, the same system of integrated behavior models is used to identify the candidate sets of DIME actions that might be applied to achieve those desired effects.
|
From page 291... ...
As noted above, this is a conceptually difficult problem to solve theoretically, but some progress can be made with the development of sufficiently flexible IDEs. IDE Development Goals and Examples An ideal IOS IDE, especially one targeted for the complex task of developing DIME/PMESII models, would include • an intuitive graphical model development environment support ing the specification of heterogeneous submodels using a variety of modeling formalisms (Bayesian reasoning, fuzzy logic, system dynamics models, rule-based expert systems, etc.)
|
From page 292... ...
To create these models, the modeler can use a variety of modeling methods (e.g., social network modeling, Bayesian belief networks, rule-based systems, fuzzy logic, case-based reasoning)
|
From page 293... ...
In the upper left of the graphic is shown a simple selection tool for the analyst to select from among the range of available models defining the PMESII environment for execution and analysis. The selection of a specific model results in the input fields for that model being captured from the selected model (via its XML schema-based I-O specification)
|
From page 294... ...
294 FIGURE 8-7 An overview of the analyst's interface in the HASMAT environment. SOURCE: Harper et al.
|
From page 295... ...
. The model constructed within the integrated HASMAT framework consists of a social network representation of an organization or loosely connected set of groups or individuals of interest to the counterterrorism analyst.
|
From page 296... ...
. Each node of the Agent components social network is generate responses represented by an to network events agent that reasons captured as agent about the network behaviors (e.g., data.
|
From page 297... ...
These detailed components capture and generate the simulated responses of a modeled individual or group based on injected or simulated stimuli. These simulated responses are then pushed back up to the social network representation as "change events" within the social network itself.
|
From page 298... ...
This allows the user to consider the effects of potential DIME actions on the PMESII models under consideration. The second type, diagnostic reasoning, enables reasoning from effects to causes.
|
From page 299... ...
For complex nonlinear models such as PMESII models, randomized sampling provides an effective approach to approximating model outputs because it is independent of the underlying formalisms being used by the model. Sampling can be used to analyze any model that incorporates both (1)
|
From page 300... ...
In the case of reasoning using DIME/PMESII models, we can use sensitivity analysis to determine which actions or input variables are most relevant in determining the outcome or effect in which we are interested. Once we have identified a subset of relevant actions, we can then perform brute-force means-ends analysis in the manner described above to determine the optimal combination of those actions.
|
From page 301... ...
that diverse frameworks for IOS models be supported and further developed -- it is too early to tell which approaches will be most useful for different purposes. Verification, validation, and accreditation In this section we describe some of the significant issues involved in the VV&A of IOS models: the ways in which they differ from physicsbased models, the special challenges of forecasting human behavior, given the huge number of variables that can combine to determine it, and other thorny issues.
|
From page 302... ...
The first way is to begin with verification, proceed to validation, and then to the intended purpose. This ordering of concerns may result in a model that is verified and validated yet fails to be useful for its intended purpose.
|
From page 303... ...
Consequently, if a simple model serves the intended purpose, then it should be preferred. Action models require some degree of realism for action, but realism is not a good test for action models.
|
From page 304... ...
As stressed by Haefner (2005) , one possibility is that a model of a given phenomenon is incomplete in the sense that it is not capable of explaining all aspects of the phenomenon deemed to be important for an intended purpose.
|
From page 305... ...
Appropriate modeling of action choices will not eliminate the uncertainty inherent in a situation, but it should help to clarify the possible action alternatives and hence provide useful guidance regarding the best action to take. We start by considering the validation of a simple forecasting model with no action domain.
|
From page 306... ...
The farmer might express this belief by postulating an if-then relationship between weather and corn yield of the form "if A, then B." The contingency condition A might be either "rain" or "no rain" and the result B might then be a specific conditional probability distribution Prob(b|A) for the corn yield b conditional on the realization of A
|
From page 307... ...
that could impact corn yield. As Table 8-4 shows, the corn farmer action model has the same general framing as the corn farmer prediction model, except that the if-then rela
|
From page 308... ...
For example, for the problem at hand, the corn farmer might be able to deduce that the addition of fertilizer to his field will profitably increase his corn yield whether or not it rains, because of a government support program that reimburses farmers for all of their fertilizer costs -- that is, fertilizer is free. In this case the farmer's best (most profitable)
|
From page 309... ...
The purpose or goal is simply to occupy important terrain and minimize casualties -- those of both the military force and the villagers. This village deployment action model could be entirely framed in terms of if-then relationships connecting compound contingency conditions to ultimate outcomes, in which each compound contingency condition involves a village entry choice together with a mayor hostility level and the presence or absence of a local resistance group.
|
From page 310... ...
Third, are the if-then relationships mapping the contingency conditions (scenario-action pairs) into possible outcomes appropriately specified and explained?
|
From page 311... ...
Specifically, as seen in Table 8-5, the village deployment action model is assumed to have three important aspects making up each contingency condition: the village entry mode, the village mayor's initial hostility level, and the existence (or not) of a local resistance cell.
|
From page 312... ...
Third, is each if-then relationship connecting a contingency condition to a possible range of outcomes specified with an appropriate level of realism and prediction? For the farmer, the corn yield is important, as is the price.
|
From page 313... ...
Inability to conduct repeated experiments is a key issue. We can observe rainfall and corn yield over many years.
|
From page 314... ...
. VV&A of IOS models is particularly problematic, because of both the lack of clear
|
From page 315... ...
It is now clear that considerably more emphasis has to be given to the development of models that span the space from the individual decision maker, to small groups, to urban populations, and even to entire national and transnational populations. As noted in Chapter 5, we need to account not only for "nominal" human behaviors, but also for those colored by individual differences (e.g., personality traits)
|
From page 316... ...
required to develop simulations of the desired human behaviors. • Identifying processes to ensure reusability of models and model components.
|
From page 317... ...
Validation Issues Specific to Individual Modeling Approaches In this section we review the validation challenges and approaches that are specific to various modeling approaches used for IOS models. Validation of Conceptual Models Verbal conceptual models are sometimes specific enough that they can be tested and plausibly falsified, using empirical field studies or controlled experiments.
|
From page 318... ...
Validation of Cognitive Models While there is increasing emphasis on validation of cognitive architectures, validation remains one of the most challenging aspects of cognitive architecture research and development. "[Human behavioral representation]
|
From page 319... ...
Therefore, typically only small portions of the overall response surface can be estimated at once. The size of the analyzed response surface is thus often dictated by the user's interests and the critical policy or decision-making questions at issue (i.e., the action domain and the scenarios relevant to that domain, as discussed above)
|
From page 320... ...
In addition, ABM researchers are also beginning to explore the potential benefits of conducting parallel experiments with real and computational agents for achieving improved validation of their behavioral assumptions.11 A critical concern is how to attain sufficiently parallel experimental designs so that information drawn from one design can usefully inform the other. Recommendations for Developing and Validating IOS Models We have argued that IOS models should be validated beginning with the purpose and then considering the action set, scenarios, and if-then relations in the specific situation.
|
From page 321... ...
Action models should aid decision makers, not replace them. Examine "What Might Be" as Well as "What Is" "What is" should mimic the real world within limits.
|
From page 322... ...
IOS models are likely to forecast a range of possible outcomes, some more likely than others, and to incorporate many factors that are highly uncertain and, indeed, unknowable at the time the model is developed. How then can such models be validated?
|
From page 323... ...
Triangulation goes beyond docking and involves examining the same action domain using an action model, an expert group using a qualitative approach, and reference to quantitative studies in the domain. An action model validated using multiple approaches is more likely to help the decision maker take actions that meet the purpose.
|
From page 324... ...
This multivariate sensitivity technique can find places where a complex model "breaks," that is, produces results that are outside a range of reasonable predictions. In summary, universal rules about what is the appropriate procedure for validating IOS models are not possible.
|
From page 325... ...
Models using primary sources of data have more flexibility, given that they can determine exactly what type of data needs to be collected. However, primary data collection involves its own set of limitations that are reflected in the factors described below.
|
From page 326... ...
. Toolkit for building hybrid, multi-resolution PMESII models.
|
From page 327... ...
Pew (Eds.) , Modeling human behavior with integrated cognitive archi tectures: Comparison, evaluation, and validation (pp.
|
From page 328... ...
. Panel on Modeling Human Behavior and Command Decision Making: Representations for Military Simula tions.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.