Behavioral Modeling and Simulation: From Individuals to Societies (2008)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Executive Summary T he Air Force and the other military services are increasingly inter- ested in using models of the behavior of humans, as individuals and in groups of various kinds and sizes, to support the development of doctrine, ­strategies, and tactics for dealing with state and nonstate adver- saries, for use in analysis of the current political and military situation, for planning future operations, for training and mission rehearsal, and even for the acquisition of new systems. In this report we refer to this broad class of models as individual, organizational, and societal (IOS) models. There are many lines of research on such models, which span several disciplines, have different goals, and often use different terminologies. The National Research Council was asked by the U.S. Air Force to review relevant IOS modeling research programs in the various research communities, evaluate the strengths and weaknesses of the programs and their methodologies, determine which have the greatest potential for military use, and provide guidance for the design of a research program to effectively foster the development of IOS models useful to the military. The formal statement of task for the study includes the following specific items: • Review the state of the art of the subset of the social sciences per- ceived as having the greatest payoff in terms of informing future computational model developments. • Review the state of the art in societal modeling applications ­serving   n I this study, the committee broadened the scope to include individual and organizational models as well because of the inseparability of all three, given the intended usage.  

 BEHAVIORAL MODELING AND SIMULATION the U.S. Department of Defense (DoD) and related agencies, with special emphasis given to computational modeling and simulation- based approaches. • Review the state of the art in the three computational modeling communities outside DoD (cognitive science and individual behav- ioral modeling, network analysis and multiagent organizational modeling, and multiresolution modeling and simulation) and iden- tify strengths and shortcomings in each. • Identify how gaps in societal behavioral modeling applications s ­ erving DoD and related agencies might be filled by conceptual models in the social sciences; computational modeling approaches now under way in the social science community; and closer l ­inkages between the cognitive science community, the network/­ organizational model­ing community, and the multiresolution model­ing and simulation community. • Develop a research and development roadmap to fill current appli- cation gaps, for the near, mid-, and far term. Today’s military missions have shifted away from force-on-force w ­ arfare—fighting nation-states using conventional weapons—toward com- bating insurgents and terrorist networks in a battlespace in which the atti- tudes and behaviors of civilian noncombatants may be the primary effects of military actions. These new missions call for agile, indigenously sensitive forces capable of switching quickly and effectively from conventional com- bat to humanitarian assistance and able to defuse tense situations without, if possible, the use of force. IOS models are greatly needed for planning, supporting, and training for these forces and for evaluating the technology with which they fight. Models of human behavior in social units—teams, organizations, cultural and ethnic groups, and societies—are needed to understand, predict, and influence the behavior of these social units. For example, models could be used to predict the effects of actions intended to disrupt terrorist networks, to predict the response of insur- gents and the local population to the presence of friendly forces in a given area, or to predict the effects of alternative diplomatic, military, and eco- nomic courses of action on the attitudes and behaviors of the population in a region of interest. Models could also be used in training and mission rehearsal to create simulation environments in which military units could, for example, experience the effects of their actions on the (simulated) behavior of a crowd that might either disperse or turn hostile. Models could also be used to evaluate the likely results of proposed changes intended to make military command and control organizational structures more agile and adaptive, and to assess the effects of introducing new technology capa- bilities on the performance of these organizations. 

EXECUTIVE SUMMARY  CONCLUSIONS We use a framework of modeling pitfalls, lessons learned, and future needs to characterize our major conclusions in a way that will be most use- ful to the sponsors in the design of future research programs. The problems or pitfalls identified by the committee are organized in terms of five major categories: 1. Modeling strategy—matching the problem to the real world: Dif- ficulties in this area are created either by inattention to the real world being modeled or by unrealistic expectations about how much of the world can be modeled and how close a match between model and world is feasible. 2. Verification, validation, and accreditation: These important func- tions often are made more difficult by expectations that verifica- tion, validation, and accreditation (VV&A)—as it has been defined for the validation of models of physical systems—can be usefully applied to IOS models. 3. Modeling tactics—designing the internal structure of a model: Problems are sometimes generated by unwarranted assumptions about the nature of the social, organizational, cultural, and individ- ual behavior domains, and sometimes by a failure to deliberately and thoughtfully match the scope of the model to the scope of the phenomena to be modeled. 4. Differences between modeling physical phenomena and human behavior—dealing with uncertainty and adaptation: Problems arise from unrealistic expectations of how much uncertainty reduction is plausible in modeling human and organizational behavior, as well as from poor choices in handling the changing nature of human structures and processes. 5. Combining components and federating models: Problems arise from the way in which linkages within and across levels of analysis change the nature of system operation. They occur when creat- ing multilevel ­models and when linking together more specialized m ­ odels of behavior into a federation of models. To summarize, IOS modeling is a complex, emerging science with roots in many different disciplines. Its advancement requires that researchers maintain awareness of each other’s work and build on each other’s results, yet the multidisciplinary nature of IOS modeling has created a fragmented field. For the field to advance, researchers need better frameworks and forums in which to compare, discuss, and evaluate their results. The field currently features a multitude of complex models created using different 

 BEHAVIORAL MODELING AND SIMULATION data and different theories to address different problems, making compara- tive analysis nearly impossible. Common datasets and challenge problems are needed in order to learn which modeling approaches and sets of vari- ables are most useful for specific types of problems. It seems clear there is no single right model and probably will never be. The committee thinks that a federated modeling approach, in which different models at different levels are linked together and component sub- models can be swapped in and out, is promising for attacking complex IOS modeling problems. Considerable research needs to be done to make this federated vision a reality, however. Standards, architectures, methods, and tools are needed to lower the barriers for developing, linking, and validat- ing federated models. Different modeling purposes require different types of models. In the committee’s judgment, the purpose of the model should drive the appropri- ate variables to be included in the model. To do this successfully requires a clear specification of model purpose and criteria for usefulness for that purpose, which in turn requires that model developers work closely with the eventual users of the model. The committee also recommends validation for action, in which the purpose of the model drives its validation criteria. IOS models cannot be validated “in general”—they must be validated for a specific use. A cross-disciplinary community of interest needs to establish and promulgate accepted standards for validation of IOS models. Triangulation methods that combine expert judgment, qualitative results and theoretical work, and quantitative results should be further refined and more widely used. Common challenge problems and datasets are needed to facilitate docking of models for comparative purposes. Finally, models of human beings and their individual and collective behaviors necessarily include a large amount of inherent uncertainty. This uncertainty is not a flaw of the model and cannot be designed out of the model. Human behavior is dynamic and adaptive over time, and it is impossible at the moment (and into the foreseeable future) to make reliably exact predictions about it. Researchers need to develop ways to estimate the probability of plausible outcomes and express those estimates in ways that are clear and meaningful to model users, who can then judge whether the results meet their needs. It is important also to avoid raising expecta- tions about the capabilities of IOS models beyond what can realistically be expected. Recommendations Recommendations for an IOS modeling research and development program fall into three broad categories: (1) large-scale, integrated cross- 

EXECUTIVE SUMMARY  disciplinary research programs, focused around representative challenge problems and common datasets; (2) research in six independent areas that will advance the capabilities to address these integrated problems; and (3)  multidisciplinary conferences, workshops, and other information exchange forums, with attendees to include not only model developers but also government program managers and military decision makers. Integrated Cross-Disciplinary Research Programs We suggest the funding of multiple large-scale, multiyear research pro- grams that focus on comparing and, if appropriate, integrating models from different disciplines, different perspectives, and different levels of detail. The goal would be to create a level playing field on which the capabilities of different approaches could be compared and the strengths of each assessed. The ultimate goal is to move IOS modeling science forward through the process of comparison, docking, and integration. It is essential for all participants in each program to focus on the same well-defined challenge problem instantiated in a common testbed and to use a common dataset. At the heart of each program would be a representative problem that is critical for military operations, defined in detail. We have chosen five representative problems as a starting point for choosing the problems to be addressed. The research teams for these efforts should be multidisciplinary, and the program team should also include military users with operational experi- ence in the domain for which the models are to be developed. These users will be ultimate judges of whether model results are useful and will provide advice on how the results can best be presented. The use of a common challenge problem and a common testbed will facilitate docking of the dif- ferent models for purposes of comparison. The development of challenge problems should be a major focus early in the development of research programs. These integrated programs will encourage mutual education between modelers and operational users. Results should be presented at workshops for program participants and other interested parties and at public confer- ences as well as published in the open literature. Independent Research Thrusts In support of the integrated programs we recommend, we have identi- fied six independent areas in which research is needed. Progress in each of these areas could increase the ability to develop the integrated modeling capabilities that are needed to address military problems. In each area, we suggest the funding of multiple research teams from multiple perspec- 

 BEHAVIORAL MODELING AND SIMULATION tives, with periodic workshops for researchers to exchange results. We also suggest that operational users as well as government program managers participate in these workshops. Thrust 1: Theory Development Models should be conceptually correct and grounded in the underly- ing fundamentals of what is known about individual human and group social behavior. However, current theory in this area does not answer all of the questions needed to structure models that address relevant issues. Basic research is needed for theory development, especially for the low- level social behaviors that are the building blocks for larger scale social behavioral patterns. This theory development work must involve multiple disciplines and perspectives with periodic workshops to exchange results. Theory development challenge problems should be defined to guide the work, but these can be nonmilitary and need not involve the level of military detail necessary for the integrated problems discussed above. A series of workshops should be conducted with researchers to identify key theory gaps. Academic institutions are key players for theory development, but they need information, incentives, and funding to address these theoretical issues. There is a need to educate researchers in military domains, establish conferences and journals in which their results can be presented, provide postdoctoral support, and provide funding that allows researchers to spend time learning about military domains in depth. Thrust 2: Uncertainty, Dynamic Adaptability, and Rational Behavior Models must deal with the inherent uncertainty and the dynamic adap- tation that characterizes human behavior. Models must also be capable of modeling both rational and nonrational behavior. Basic research is needed in each of these areas. Issues include • How should models capture the “uncertainty-in-the-small” associ- ated with individuals and small groups? How can model structures and parameters capture this variability, and how much of this vari- ability must be included for the purposes of the model? • How should models capture the “uncertainty-in-the-large” associ- ated with populations and variations in population distributions? How much variability must be included for the purposes of the model? • How can models capture adaptation and learning over time and as the results of actions by others? For example, people have multiple 

EXECUTIVE SUMMARY  overlapping identities and allegiances. How can these be captured in a model, and how can one estimate the effects of actions and events on the primacy of these multiple allegiances as they affect decisions and actions? • What are the factors that contribute to rational, adaptive behavior, and what factors induce behavior that appears irrational? Models of both rational and nonrational behavior must capture all the key factors—cognitive, affective, cultural, and contextual—that motivate and shape behavior of specific individuals in specific situations. Better techniques are needed for understanding the implications of diversity and variability for model-based sensitivity analysis. Better auto- mated technology is needed to put the model through its paces to explore the parameter space effectively and produce robust results. Thrust 3: Data Collection Methods The difficulty of obtaining data is an ongoing challenge for IOS model- ing. Research is needed to develop better data collection processes through field studies, experiments, and potentially massively multiplayer online games (MMOGs). Although a variety of ethnographic data collection techniques are cur- rently in use, they need to be better tailored to the needs of IOS models. For field data collection, it is necessary to bring modelers and data collectors together to develop data ontologies, joint specifications, and data collection methodologies and tools that are specifically tuned to IOS models. MMOGs are a potential untapped resource for collecting social and behavioral data on a large scale. We recommend the creation of a MMOG facility and the funding of basic research to determine if MMOGs can be used to test, verify, and validate IOS models. We recommend that fund- ing be put into developing the science of MMOGs. We note that funding MMOGs is a risky endeavor, but we think that the potential benefits out- weigh the risks. Thrust 4: Federated Models It is a fundamental conclusion of the committee that no single model- ing approach can provide all the capabilities needed by DoD. We recom- mend a federated approach in which modeling components are created to be interoperable across levels of aggregation and detail. For example, a federated model might include a detailed representation of a few key individuals, linked to group-level models of different cultural groups and 

 BEHAVIORAL MODELING AND SIMULATION terrorist organizations, linked to geographic sector–level models of the level of unrest in a city. This approach is flexible and extensible, allowing the addition or subtraction of models at different levels of detail as needed for the problem to be addressed. Combining model components to create federated models in the sense being recommended requires deep semantic interoperability (i.e., theoreti- cal consistency) and presents difficult challenges. To create semantic inter­ operability, it is necessary to recognize that the links among components are themselves elements of the model. Research is needed on: • How to ensure that the models being federated embrace compat- ible assumptions regarding concept abstractions, entity resolution, time scale resolution (tempo), uncertainty, adaptability, docking standards, input-output, semantics, etc. • How the components of the federated model should be encapsu- lated and which elements must be exposed to other components. • How specific classes of models should be linked (e.g., cognitive models to social network models). • How to ensure dynamic extensibility. In addressing these issues, IOS modelers should maintain awareness of research and development in model federation in the larger modeling and simulation community. Thrust 5: Validation and Usefulness Current VV&A concepts and practices were developed for the physical sciences, and we argue that different approaches are needed for IOS models. Specifically, we recommend that a “validation for action” approach be used that assesses the usefulness of a model for the specific purposes for which it was developed. It is thus very important that the purpose(s) and criteria for judging success be clearly stated a priori for all models. We recom- mend organizing national workshops to agree on appropriate processes for VV&A of IOS models and to outline a roadmap for developing improved processes and standards. On the basis of the results of this workshop, we recommend that a DoD-wide authority develop and disseminate VV&A processes and standards for IOS models. Basing model validation on the usefulness of the model for specific problems requires that model purposes be clearly stated by model users and clearly understood by model devel- opers. We suggest that, as part of developing a VV&A standard for IOS models, clear guidelines be developed for specifying model purpose. 

EXECUTIVE SUMMARY  Thrust 6: Tools and Infrastructure for Model Building It is important to reduce the barrier to entry for developing models, modeling tools, frameworks, and testbeds. Scientists should be able to build and validate models without the large overhead currently associated with many DoD modeling and simulation investments. It should be possible to easily tailor existing models for specific purposes. Sharing of IOS modeling knowledge across disciplines, as facilitated by the conferences and workshops recommended below, will support this goal. Work is also needed in developing an infrastructure for IOS modelers, including a national network of possible collaborators, common databases for model development and testing, and frameworks and toolkits for rapid model development. The limited data that exist for IOS models are often not accessible to model developers. We recommend national web-accessible data repositories that are open to researchers who seek to inform and test models. For mili- tarily relevant domains in which some data are classified, we recommend an investment in automated tools to sanitize the data. We also recommend the development and maintenance of an online web-based catalog of general approaches, models, simulations, and tools. The notion is to develop something along the lines of the Defense Model- ing and Simulation Office’s Modeling and Simulation Resource Repository or the clearinghouse at Carnegie ­Mellon’s CASOS site (http://www.casos. cs.cmu.edu). To be effective, the envisioned site needs careful consideration in terms of organization, content, currency, and usability. This cannot be a one-time effort. It needs significant startup funding and continued support over its lifetime. Multidisciplinary Conferences and Workshops A number of the issues and problems identified by the panel were the results of the failure of different disciplines to exchange information, or they resulted from misunderstandings among government funders of model devel- opment efforts, military users of models, and model developers. Because of the diversity of this group, there is no natural forum for them to exchange information. We recommend the organization of special-­purpose workshops around the integrated research programs recommended above, as well as workshops for the independent research thrusts described above. IOS modelers need to be educated on: • The nature of the military decisions for which models are relevant. • Desired model functionality. 

10 BEHAVIORAL MODELING AND SIMULATION • The most useful form for presenting model results. • The value of work performed by others outside their discipline. • Feasible and appropriate VV&A approaches for IOS models. Operational users and managers need to be educated on: • The value of multidisciplinary approaches and the need for review of models from multiple perspectives. • The inherent uncertainty associated with model predictions. • The value of models for sensitivity and trade-off analysis (versus the one right answer). • The design of virtual experiments to assess results over a range of conditions. • Reasonable definitions of validation for IOS models, feasible approaches for VV&A testing, and why these approaches differ from those used for physics-based models. The recommended workshops should involve model developers, opera- tional military users of the models, and government personnel making funding decisions regarding model development. Roadmap for Future Research and Development The committee recommends a use-driven research program to extend the state of the art in IOS modeling, focused around a series of challenge problems—clear specifications of the uses to which the model is to be put, defined to be relevant to military needs, and expanded over time as progress is made in modeling approaches, tools, and technologies. The purpose of the model, as captured in the challenge problems, drives the theory to be applied, the data to be used, and the model development. Model develop- ment is made easier by modeling tools and infrastructure and relies on federation standards to ensure the interoperability of model components. Once the model is developed it is validated by asking the question: Is the model useful for its intended purpose? The recommended program proceeds in a cyclical fashion. Based on the answers to the question “Is the model useful?” new models may need to be developed, new theory and new data (and new types of data) may be needed, and new interoperability standards, tools, and infrastructure may be required. Depending on the results, the problem itself may need to be ­redefined, clari- fied, or expanded. These challenge problems, combined with periodic work- shops and conferences to compare and exchange results, serve as a unifying force and a common ground for the fragmented field of IOS modeling, pro- viding a foundation on which scientific progress can be made.