National Academies Press: OpenBook
« Previous: 5 Modeling and Simulation Research and Development Topics
Suggested Citation:"6 Conclusions and Recommendations." National Research Council. 2002. Modeling and Simulation in Manufacturing and Defense Acquisition: Pathways to Success. Washington, DC: The National Academies Press. doi: 10.17226/10425.
×
Page 103
Suggested Citation:"6 Conclusions and Recommendations." National Research Council. 2002. Modeling and Simulation in Manufacturing and Defense Acquisition: Pathways to Success. Washington, DC: The National Academies Press. doi: 10.17226/10425.
×
Page 104
Suggested Citation:"6 Conclusions and Recommendations." National Research Council. 2002. Modeling and Simulation in Manufacturing and Defense Acquisition: Pathways to Success. Washington, DC: The National Academies Press. doi: 10.17226/10425.
×
Page 105
Suggested Citation:"6 Conclusions and Recommendations." National Research Council. 2002. Modeling and Simulation in Manufacturing and Defense Acquisition: Pathways to Success. Washington, DC: The National Academies Press. doi: 10.17226/10425.
×
Page 106
Suggested Citation:"6 Conclusions and Recommendations." National Research Council. 2002. Modeling and Simulation in Manufacturing and Defense Acquisition: Pathways to Success. Washington, DC: The National Academies Press. doi: 10.17226/10425.
×
Page 107
Suggested Citation:"6 Conclusions and Recommendations." National Research Council. 2002. Modeling and Simulation in Manufacturing and Defense Acquisition: Pathways to Success. Washington, DC: The National Academies Press. doi: 10.17226/10425.
×
Page 108
Suggested Citation:"6 Conclusions and Recommendations." National Research Council. 2002. Modeling and Simulation in Manufacturing and Defense Acquisition: Pathways to Success. Washington, DC: The National Academies Press. doi: 10.17226/10425.
×
Page 109
Suggested Citation:"6 Conclusions and Recommendations." National Research Council. 2002. Modeling and Simulation in Manufacturing and Defense Acquisition: Pathways to Success. Washington, DC: The National Academies Press. doi: 10.17226/10425.
×
Page 110
Suggested Citation:"6 Conclusions and Recommendations." National Research Council. 2002. Modeling and Simulation in Manufacturing and Defense Acquisition: Pathways to Success. Washington, DC: The National Academies Press. doi: 10.17226/10425.
×
Page 111
Suggested Citation:"6 Conclusions and Recommendations." National Research Council. 2002. Modeling and Simulation in Manufacturing and Defense Acquisition: Pathways to Success. Washington, DC: The National Academies Press. doi: 10.17226/10425.
×
Page 112
Suggested Citation:"6 Conclusions and Recommendations." National Research Council. 2002. Modeling and Simulation in Manufacturing and Defense Acquisition: Pathways to Success. Washington, DC: The National Academies Press. doi: 10.17226/10425.
×
Page 113
Suggested Citation:"6 Conclusions and Recommendations." National Research Council. 2002. Modeling and Simulation in Manufacturing and Defense Acquisition: Pathways to Success. Washington, DC: The National Academies Press. doi: 10.17226/10425.
×
Page 114
Suggested Citation:"6 Conclusions and Recommendations." National Research Council. 2002. Modeling and Simulation in Manufacturing and Defense Acquisition: Pathways to Success. Washington, DC: The National Academies Press. doi: 10.17226/10425.
×
Page 115
Suggested Citation:"6 Conclusions and Recommendations." National Research Council. 2002. Modeling and Simulation in Manufacturing and Defense Acquisition: Pathways to Success. Washington, DC: The National Academies Press. doi: 10.17226/10425.
×
Page 116
Suggested Citation:"6 Conclusions and Recommendations." National Research Council. 2002. Modeling and Simulation in Manufacturing and Defense Acquisition: Pathways to Success. Washington, DC: The National Academies Press. doi: 10.17226/10425.
×
Page 117
Suggested Citation:"6 Conclusions and Recommendations." National Research Council. 2002. Modeling and Simulation in Manufacturing and Defense Acquisition: Pathways to Success. Washington, DC: The National Academies Press. doi: 10.17226/10425.
×
Page 118
Suggested Citation:"6 Conclusions and Recommendations." National Research Council. 2002. Modeling and Simulation in Manufacturing and Defense Acquisition: Pathways to Success. Washington, DC: The National Academies Press. doi: 10.17226/10425.
×
Page 119
Suggested Citation:"6 Conclusions and Recommendations." National Research Council. 2002. Modeling and Simulation in Manufacturing and Defense Acquisition: Pathways to Success. Washington, DC: The National Academies Press. doi: 10.17226/10425.
×
Page 120
Suggested Citation:"6 Conclusions and Recommendations." National Research Council. 2002. Modeling and Simulation in Manufacturing and Defense Acquisition: Pathways to Success. Washington, DC: The National Academies Press. doi: 10.17226/10425.
×
Page 121
Suggested Citation:"6 Conclusions and Recommendations." National Research Council. 2002. Modeling and Simulation in Manufacturing and Defense Acquisition: Pathways to Success. Washington, DC: The National Academies Press. doi: 10.17226/10425.
×
Page 122
Suggested Citation:"6 Conclusions and Recommendations." National Research Council. 2002. Modeling and Simulation in Manufacturing and Defense Acquisition: Pathways to Success. Washington, DC: The National Academies Press. doi: 10.17226/10425.
×
Page 123
Suggested Citation:"6 Conclusions and Recommendations." National Research Council. 2002. Modeling and Simulation in Manufacturing and Defense Acquisition: Pathways to Success. Washington, DC: The National Academies Press. doi: 10.17226/10425.
×
Page 124
Suggested Citation:"6 Conclusions and Recommendations." National Research Council. 2002. Modeling and Simulation in Manufacturing and Defense Acquisition: Pathways to Success. Washington, DC: The National Academies Press. doi: 10.17226/10425.
×
Page 125
Suggested Citation:"6 Conclusions and Recommendations." National Research Council. 2002. Modeling and Simulation in Manufacturing and Defense Acquisition: Pathways to Success. Washington, DC: The National Academies Press. doi: 10.17226/10425.
×
Page 126

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

6 Conclusions and Recommendations The committee makes four overall recommendations—one in each of the four major areas that it considers current impediments to the widespread use of simulation-based acquisition (SBA) and related concepts in manufacturing and acquisition: · Technology and research; · Infrastructure for modeling and simulation (M&S); · Modeling and simulation in manufacturing and acquisition, including developing experience in the use of M&S, learning lessons from that experience, and institutionalizing those lessons; · Culture and the human issues inherent in any major change. Within these overall recommendations are subsidiary recommendations addressing the four sets of issues. Complicating this picture are the breadth and depth of SBA. Application of M&S to a single component is quite different from its application to the life cycle of a system-of-systems. The goal of the recommendations is to move DOD toward this all-encompassing objective. The current state of M&S technology applicable to systems acquisition is mixed. Some is ready for full-scale implementation. Some is ready for prototype testing, evaluation, and improvement. Some is not available at all. To move toward DOD's ultimate goads will require departing from approaches that dominate today's M&S and rely on single- point solutions—single-resolution models and "stove-piped" simulations that cannot be reused and integrated with others. These departures will 103

104 MODELING AND SIMULA TION IN MANUFACTURING necessitate significant improvements in multiresolution modeling; model integration; model reuse; verification, validation, and accreditation (VV&A); and multiresolution modeling among others. To achieve DOD's ultimate goals, the whole modeling paradigm may need to be rethought. These methodological and technological issues must be addressed to form a firm scientific basis for M&S in manufacturing and acquisition of military systems (or systems-of-systems). DOD must gain experience with modeling and simulation in the context of SBA. Carefully chosen projects should use M&S in real, important applications. This first class of applications should not carry significant technological risk, but should use proven technology to demonstrate the value of M&S in acquisition even when applied in limited situations. All of these experiences need to be positive. To continue, a second set of carefully chosen projects should use M&S in SBA in a prototyping mode. These projects should carry moderate technological risk, but low programmatic risk. The prototypes should 'not be on the critical path of major acquisitions. This set of projects should be used to understand the application of new M&S technology in real-world application, and the results should indicate the strengths and weaknesses of M&S technology in SBA. Successes should be moved into the mainstream of SBA. Areas that prove difficult, but important, should drive M&S R&D funding. A third area of importance is in infrastructure development, both in M&S and in information technology. Lacking infrastructure, every M&S application will start over without building on past applications. The infrastructure is required in order to achieve continuous improvement in the use of M&S in acquisition. Infrastructure includes shared processes, databases, standards, and architectures. This infrastructure must largely evolve from practice. Externally imposed standards, for example, are rarely effective. Fourth, improvement in all of the above areas will prove ineffective without a change in the DOD acquisition culture. One of the committee's recommendations is for an SBA center of excellence. This resource, which could be geographically distributed, would help create the new culture of using M&S in SBA applications. It would also help the defense SBA community reach out to the academic community and integrate knowledge and insights from that community into the DOD acquisition world. Finally, there must be leadership from the top in DOD to encourage the appropriate use of M&S in acquisition and manufacturing. The risk for program managers must be changed from worrying about "deviating from the status quo" to worrying about "not mindfully using new technology to improve the state of acquisition practice."

CONCLUSIONS A ND RECOMMENDATIONS 105 An integrated view of the committee's recommendations is critical. Enhancement of the technology enables the process. Use of that technology provides experience that guides further use, as well as pointing out important opportunities for further R&D. infrastructure allows improvement over time and an ability to be more consistent and integrative. Finally, people and culture are the bottom line. If the people and the business culture do not trust and embrace M&S in manufacturing and acquisition, use of SBA will not advance. As policy makers read these recommendations, they need to consider the synergy among these areas. Each area needs to be addressed to be successful in achieving DOD's goals. TECHNOLOGY AND RESEARCH Overall Recommendation. Long-term research and development should be funded, conducted, and applied to increase the science and technology base for M&S in areas in which current knowledge falls short ofthat required for manufacturing, acquisition, and life-cycle support of military systems. . Recommendation. In order to realize DOD's vision for the use of M&S in manufacturing and acquisition generally, and for SEA in particular, DOD should conduct or support basic research and development in the following areas: Modeling methods: scalability, multiresolution and multiviewpoint modeling, agent-based modeling, semantic consistency of models, model complexity, fundamental limits of models and computation, and characterization of uncertainty and risk in models; Model integration: interoperability, composability, integration of heterogeneous processes, and linking of engineering and effectiveness simulations; Model correctness: domain knowledge, including phenomenology of warfare, physics-based modeling, and human behavior modeling; and general verification, validation, and accreditation methods; Standards: M&S standards for interoperability and modeling; general software standards; and higher-layer standards, including enterprise engineering; Methods and tools: for assistance in the translation of system requirements into system functionality.

106 MODELINGANDSIMULATIONINMANUFACTURING · Domain-specific models: including models for emerging areas such as information operations and operations other than war. The current state of scientific knowledge in M&S falls short, in several areas, of the level needed to realize DOD's vision for the use of M&S in manufacturing and acquisition generally and for SBA in particular. Basic scientific research and development (R&D) is required in those areas to address the knowledge shortfalls. Those areas, identified and discussed in Chapter 5, are summarized here. Research in modeling methods is needed. Scalability is an essential M&S capability to support the range of M&S applications needed. Multiresolution and multiviewpoint modeling contribute to providing scalability and flexibility, but they are still understood primarily from an experimental point of view. Agent-based modeling can support emerging requirements for systems-of-systems modeling. Models must be semantically consistent if they are to be composed in a simulation system. Abstraction and multiresolution families of models can help deal with the increasing complexity of models. Theoretical limits of modeling and computation apply to the use of M&S for manufacturing and acquisition, but the implications ofthose limits are generally not considered in that context. They must be studied, both to determine the limits and to develop methods to deal with them. Uncertainty is present in most of the real- world systems of interest; its influence must be modeled and measured. Both characterization of uncertainty and risk in models and the development of models that assist in evaluating uncertainty and risk are important. Further advances in model integration are required. Model interoperability and composability, have the potential to support the flexible use of models for different phases of the manufacturing and acquisition process, but they present both theoretical and practical problems. Substantial effort has been put into interoperability and composability, with the goal of achieving both reductions in development cost and increases in credibility, but those goals have generally not yet been fully achieved. This is especially true with respect to integrating heterogeneous processes and linking engineering and effectiveness simulations. Increased attention must be given to model correctness. improvements to model correctness will depend on both domain knowledge in crucial areas and general VV&A methods. More domain knowledge relevant to model development is needed in the areas of the phenomenology of warfare, physics-based modeling, and human behavior. The credibility and utility of M&S for all applications, including i

CONCLUSIONS AND RECOMMENDATIONS 107 manufacturing and acquisition and SBA, depend on validation of models and simulations, yet the degree and bounds of validity of many models are difficult to quantify or even qualify. The development of supporting standards must continue. Standards for interoperability have achieved significant success, but work remains in this area. Standards for modeling will directly support interoperability and composability. General software standards have made simulation development more predictable and reliable, and further application of them to simulation would be beneficial. It is important that higher-layer standards, such as enterprise engineering, develop in a manner compatible with M&S. Recommendation. M&S capabilities should be enhanced so that systems-of-systems have the following capabilities: . · To represent possible design variations, operational use patterns, and engagement scenarios; · To contain and make available a library of composable sensor, weapon, and C4ISR (command, control, communications, computers, intelligence, surveillance, and reconnaissance) models; · To manage interactions among component systems efficiently; and To support analytic and optimization usage modes with visualization, experiment definition, and statistical analysis capabilities. Military force modernization in the future may depend on the introduction of novel system and operational concepts to which the present acquisition process is poorly adapted. Accelerated development and demonstration of new systems prior to full funding or fielding should be conducted in order to assess their expected mission effectiveness. M&S technologies are essential for performing such assessments in complex warfighting environments. This is especially true for new operational concepts that involve systems-of-systems. Systems-of-systems simulation depends on several of the basic M&S areas listed above for research—especially, multiresolution modeling, integration and interoperation, and validation. It is therefore a cross-cutting and integrative application of those basic research areas. Because of its difficulty, research on systems-of-systems simulation should proceed in parallel with work in the basic areas, since the systems-of-systems

108 MODELING AND SIMULA TION IN MANUFACTURING simulations research may reveal requirements to consider when studying the basic areas. Recommendation. A research initiative should be created at multiple universities to attract academic attention and expertise to the M&S needs of DOD. Such an initiative, commonly known as a multiuniversity research initiative (MURI), may be useful in attracting academic expertise to the M&S needs of DOD. An increasing number of universities have programs that focus on M&S. Each MURI should be configured to include strong industrial participation, not only in terms of dollars but with time of key personnel to work with academic researchers. Such programs can focus efforts and provide assistance in addressing the most critical M&S technology, infrastructure, and programmatic shortfalls. By encouraging research with direct funding and by stipulating active involvement of both industry and academia in shared research, DOD should be able to advance the state of the art in M&S technologies directly. The Defense Modeling and Simulation Office (DMSO) should establish mechanisms to acquire feedback from DOD program offices concerning shortfalls in M&S research. This information should be used to drive the requirements process for direct funding within the MURI program. Recom mendation. Transitioning of research into applications should be planned and executed as an integral part of the development process. A continuing problem has been the transitioning of results from M&S research into application. Active involvement of industry in M&S research, together with carefully crafted demonstration projects implementing research results, will provide lessons learned in real-world application. Remaining difficulties are to be cycled back into research programs, while successes should be harvested and developed for widespread use in robust manufacturing and acquisition pursuit. Appropriate members of the application community must be made aware of results via active promotion as well as through papers, workshops, and symposiums. Desired End State To reach the potential benefits M&S could have for manufacturing and acquisition in general and SEA in particular, research is required, and

CONCL USIONS AND RECOMMENDS TIONS 109 clearly a desired outcome is that it be funded and conducted. This is not, however, simply a general call for M&S research of all types. Rather, specific research topics have been identified as crucial to the applications that are of concern here, and it is research into those topics that is sought. That research should be conducted through one or more research initiatives with relatively broad university participation, but with active coordination from a responsible organization to ensure its relevance. Finally, it is not enough that the research be funded and conducted; it must also be applied to the issues of manufacturing and acquisition. The test of success in this technology and research area will be the practical application ofthe results of the recommended research in actual manufacturing and acquisition processes. That transition from research into application will likely depend on cooperation between universities and industry. INFRASTRUCTURE FOR MODELING AND SIMULATION Overall Recommendation. DOD should invest in "common good" activities to encourage adequate standards and a strong infrastructure for M&S. Recommendation. DOD should institute incentives for program managers to develop M&S elements that contribute to the general infrastructure, including an annual competition for the best infrastructure contributions. A handbook that illustrates and discusses how M&S can be integrated into program planning documents should be developed. In current practice, investments by program offices in M&S may often result in data, models, tools, and environments that have the potential to be reused across DOD acquisition programs. However, such generally useful outcomes are incidental to program manager goals and objectives, and there is no institutionalized process for continually infusing these results into the common infrastructure. A process of this kind should be defined and implemented. DOD should support creation of SBA-related M&S infrastructure. For example, resources could be set aside annually to reward program managers in a competition for the best infrastructure contributions. In this way, program offices would increase the value of their work to the DOD by developing M&S and information technology applications useful to the immediate program and readily reused by other DOD program offices.

110 MODELING AND SIMULA TION IN MANUFACTURING DOD should develop a guide or handbook that not only defines systems engineering as practiced within DOD but that goes on to illustrate and discuss how M&S may be integrated into program planning documents, such as the integrated master plan (IMP). This guide should be made available online and should provide templates and other tools necessary to support development and implementation of an acquisition program M&S plan. Defining systems engineering at a high level and further defining how M&S applies to systems engineering activities within a total systems engineering framework will provide clearer guidance for the integration of M&S into systems engineering activities. Providing this as a Web-based system also provides direct linkage of guidance to templates and tools within the DMSO and Defense Acquisition Deskbook infrastructure. The Deskbook is an excellent repository of information, but it lacks useful products for supporting SBA and use of M&S in the . . . acquisition process. Recommendation. DOD should exploit common elements of M&S to develop a common infrastructure capable of supporting consistency and interoperability across programs. The M&S infrastructure that the committee recommends creating includes the following: . . Common repositories. These repositories should contain data, models, tools, and environments that can support multiple phases of a program and that persist from program to program. Standards should allow different developers to interoperate for the common good, while retaining competitive advantage and property rights where appropriate. Knowledge base. This knowledge base underlies the right formulation of M&S infrastructure and workable standards and represents a well-organized information resource in the theory, science, engineering, and craft required for successful M&S development. Properly archived in electronic database form, with helpful searchability attributes, the knowledge base supports continued advances though basic and applied research and development. Trained M&S workforce. This workforce contains the cadre of professionals, ranging from specialists in M&S infrastructure to M&S researchers, needed to support the wide array of activities and programs that SBA entails. More and better-trained workers with the knowledge and skills to jump-start industrial and DOD assimilation of M&S and sustain its development are needed.

CONCL USIONS AND RECOMMENDA TIONS . 111 More detailed discussion of this area is included under the heading "Culture and Human Issues," in this chapter. Information technology infrastructure. This infrastructure contains the computing and networking technology that processes and routes the flow of information, much as the national highway system enables the flow of physical goods and services. M&S technologies are necessary to design and test current and next- generation computing and networking technologies that promise exponential increases in power. Conversely, the information technology structure will drive the advance of M&S infrastructure, making it possible to tackle issues in greater depth with increasing confidence in the outcomes. Related to this infrastructure are standards for its modeling components. Recommendation. A collaborative effort should be stimulated among members of DOD, industry, and the academic community to advance the emergence of standards for performance simulation and product modeling. · DOD should remain actively engaged in commercial standards efforts to ensure that DOD needs are considered in the standards development process. · DOD should take the lead in the development of standards that lack commercial interest. · DOD should develop standard semantics for the data elements used in DOD acquisition-related models and simulations, such as standard nomenclature, definitions, and units of measure. In its review of previous studies related to M&S in military acquisition, the committee found a general consensus that standards play an important role and can be applied to make better use of M&S in acquisition. The continuing need for a collaborative effort in this area among DOD, industry, and academia must be reiterated. As robust standards emerge, they will enhance the interoperability and reuse of models and simulations for system acquisition. As a cautionary note, ill- thought-out, mandated standards inhibit progress rather than encourage it. Standards must emerge from the joint efforts of the user community. Some areas exist in which significant industry effort outside the DOD community can be leveraged. For example, although progress has been relatively slow, significant effort in the international standards community has been devoted to the development of the Standard for the Exchange of Product Model Data (STEP) for computer-aided design (CAD)/computer- aided manufacturing (CAM) and other types of product data. DOD and relevant industry groups must remain actively engaged in such commercial

112 MODELING AND SIMULATION IN MANUFACTURING standards efforts to ensure that DOD needs for such things as enhanced CAD/CAM standards are considered in the standards development process. Significant progress has also been made on standards for information exchange, such as the extensible markup language (XML). Although standards are often a moving target during their evolutionary development, it is important for the DOD community to participate in their development and attain maximum advantage from commercial efforts. Because of lack of commercial interest, there will be some standards for which the DOD community must take the lead in development. Examples of this in the performance simulation community include the high level architecture (HLA), now approved as IEEE Standard 1516, and the synthetic environment data representation and interchange specification (SEDRIS), both developed with leadership and financial support by DMSO. Existing standards organizations, such as the Simulation Interoperability Standards Organization (SISO), can be leveraged to evolve simulation-oriented standards. Although there is hope that standards such as these might become predominantly supported and pervasively adopted by the commercial marketplace, their importance to the acquisition of DOD systems dictates that DOD be sufficiency involved to ensure their sustenance. An area in which less progress has been made is standard semantics for data elements used in DOD acquisition-related models and simulations. Work on standard nomenclature, definitions, and units of measure is needed to ensure valid substantive interactions among models and simulations as use of federated M&S becomes more prevalent. Standards that build from current standards are emerging in severa] M&S areas. Development of such standards should be encouraged. Related software and system engineering standard developments, such as those described in Chapter 5, should be monitored closely. Desired End State A significant government role is needed to nurture common models and tools that industry on its own would not develop. The benefits derived from proactive leadership by DOD in developing such infrastructure for the common good would include these: · Fully exploiting the potential of modeling and simulation to greatly improve products, perfect processes, reduce design-to- manufacturing-to-fielding cycle time, and reduce system realization costs;

CONCLUSIONS AND RECOMMENDATIONS 113 Developing and managing the knowledge of theory, science, engineering, and craft required for successful M&S developments; Growing and training the large numbers of professionals, ranging from specialists in M&S infrastructure to M&S researchers, who are needed to support the wide array of activities and programs that SBA entails. The various components of infrastructure potentially constitute a self- sustaining or autocatalytic process. Each of these constituents is necessary to make the whole cycle work, and if they are present, each reinforces the other. A test of the successful implementation of the recommended infrastructure elements is a noticeable increase in the number, and capabilities, of M&S professionals; a significant increase in the theory, methods, and best practices available for M&S projects; and ultimately, greatly reduced times to develop and acquire the best systems for the nation's defense. USE OF MODELING AND SIMULATION IN ACQUISITION AND MANUFACTURING Overall Recommendation. Process improvements should be undertaken to better support integration of M&S within DOD's system acquisition process. Recommendation. M&S use should be expanded in the concept exploration phase. M&S and SBA in DOD must have a scope that includes not just "building the thing right," but also "building the right thing." Approaches to SBA to date have focused on building systems once the need for those systems has been identified, to the exclusion of identifying what system should be built. In initial DOD applications of the principles of SBA, attention has been focused on selected programs (e.g., Joint Strike Fighter (JSF), or the Crusader artillery). In each of these activities, the government/industry team has creatively employed integrated process teams (IPTs) and M&S tools to facilitate intraprogram coordination for example, by linking the design and test and evaluation (T&E), the operations and maintenance (O&M), and the training communities. Although results are only preliminary, there is some evidence to suggest that the quality performance of these programs is

114 MODELING AND SIMULA TION IN MANUFACTURING enhanced through the application of SBA principles (i.e., "building the thing rightly. Recently, the Secretary of Defense submitted the 2001 Quadrennial Defense Review (QDR) to Congress to identify the appropriate strategy for DOD to use in dealing with the rapidly changing security environment (OSD, 2001) by acquiring systems that can be brought to bear rapidly against adversaries that employ antiaccess and area denial tactics. This suggests that there is a premium on acquiring systems that are interoperable, relatively light, and readily transportable (e.g., on C-130 transport aircraft) and aircraft with sufficient flying range to operate from bases that are relatively far from the theater of action (i.e., "building the right things. Unfortunately, as the world situation changes, thus changing the needed capabilities for systems, application of SBA principles merely to "build the thing right" is no longer sufficient. This suggests the importance of beginning the SBA process at the broad conceptual stage, while the mix of future systems and their broad properties are still being explored in the context of evolving national security strategy. At this point, M&S should be employed creatively to broadly characterize needed attributes of the key systems (i.e., to determine what "the right thing" is). At this early stage, M&S can be used most effectively to promote cooperation between government and industry in making trade-offs, such as cost or weight versus performance. institutionally, this should be done during the requirements process in production of capstone requirements documents and Joint Requirements Oversight Council (JROC) deliberations. In addition, the answer to the question "What is the right thing?" may change during the acquisition of a system as the geopolitical context changes. For example, the Crusader may have been the right thing when NATO faced a massive array of Soviet armor in Central Europe, but it is unlikely to remain the right thing in the future when speed and ease of transport to remote theaters are of paramount concern. Accordingly, it would be desirable for the Secretary of Defense to designate an organization in his office to assess periodically whether selected systems (or systems-of- systems) remain the right thing to acquire. Once it has been determined (and reaffirmed) what the right thing is, the principles of SBA should be employed to ensure that individual weapons systems are "built right." Use of appropriate processes, including M&S, to ensure that there is appropriate intra- and interprogram communication and sharing is a critical part of building the thing right. Recommendation. A set of guidelines and best practices should be developed concerning model, simulation, algorithm, and data ownership rights among DOD and industry organizations

CONCL USIONS AND RECOMMENDA TIONS involved in system acquisition to enhance the potential for collaboration and facilitate reuse of models and software components. 115 Collaborative acquisition of systems involves a much closer relationship between government and industry than has traditionally existed. It also relies on a greater degree of model, simulation, and data sharing. In reviewing previous studies, the committee found that issues of data and model ownership, proprietary information, and intellectual property represent significant obstacles to SBA-type processes. These significant issues have not been resolved in any coordinated fashion. Additionally, increasing international interest in SBA, together with international acquisition programs (e.g., the JSF), result in even more complex issues in this area. Industrial organizations have legitimate concerns regarding the protection of proprietary data whenever a competitive acquisition process is used or anticipated. Conversely, to ensure that it obtains an effective system, the government needs significant insight into system characteristics and may want to preserve the ability to compete subsequent modifications. Currently, issues that involve such model and data ownership rights are essentially resolved on a program-by-program, case- by-case basis. As system-of-systems interoperability (within DOD, across agency lines, and internationally) becomes increasingly emphasized, case-by-case negotiation of data rights on models, simulations, algorithms, and data could have an adverse impact on the ability to represent the constituent systems in a valid manner. Another issue is the protection of information within electronically connected collaborative environments, in which many stakeholders work collaboratively on different aspects of system development using common databases. Given the importance of model, simulation, algorithm, and data ownership issues, a set of guidelines and/or best practices should be developed. These guidelines and best practices should include, at a . . mlnlmum: · Guidelines for government sharing of models, simulations, and data with industry during competitive procurements, and with international partners; Best practices for definition of government ownership rights to models, simulations, and data developed during system acquisitions; and

116 MODELING AND SIMULATIONIN MANUFACTURING . Industry agreements on acceptable practices for protection of electronically stored proprietary data that are made selectively available to the government. Development of such guidelines and best practices is probably most effectively done by a working group of DOD and industry acquisition professionals, with international representation as appropriate, convened specifically for this purpose. Recommendation. A deliberate effort should be undertaken to define how M&S is to be integrated into the DOD systems acquisition process, including use of the maturity of the simulation support plan (SSP) as an element in milestone decision reviews and establishing specific evaluation criteria. No single acquisition program has yet demonstrated a comprehensive use of SBA processes. While program-specific approaches to SBA are in use in some DOD programs, no comprehensive, cross-program approach is yet in use. SBA literature addresses the use of M&S in five areas: 1. To define and analyze the requirement for the system, 2. To engineer the system, 3. To define the system development process, 4. To test the system, and 5. To support system training. In support of SBA, the program manager is required to develop a simulation support plan (SSP) to define precisely how M&S will be used in the five areas listed above. The maturity of an acquisition program may be evaluated by its use of M&S to support these areas. The maturity of the SSP and its implementation should be made an element of the milestone decision reviews, along with the achievement of key performance parameters. Specific criteria should be established that will serve as SSP evaluation criteria to be applied at the milestone reviews. The criteria should address the manner in which M&S has been applied to establish the acquisition program baseline. The criteria should establish SSP maturity metrics that are reported into a system such as the consolidated acquisition reporting system (CARS) for cross-program review. The Director, Defense Research and Engineering (DDR&E) should charge the Functional Acquisition Area Council of the DOD Executive Council on Modeling and Simulation (EXCIMS) to oversee development of an SBA maturity guide to be applied to major defense acquisition

CONCLUSIONS AND RECOMMENDATIONS 117 programs. The SBA maturity guide should establish the criteria to assess SSP maturity and M&S integration and should use exit criteria to assess the status of weapons system programs that will be evaluated at each program milestone. The exit criteria will assess the degree to which M&S integration planning has been accomplished and the degree to which M&S integration has occurred in the system acquisition process. Application of M&S to the system acquisition process will be evaluated against known areas of opportunity and success in applying M&S to system acquisition activities where its use was planned. The committee found that performance modeling or system effectiveness simulations are not sufficiently integrated with product modeling or engineering simulations, which are used to determine how to build a system. The DOD acquisition instruction document' requires that the program manager apply a systems engineering process to translate operational needs and/or requirements into a solution that includes design, manufacturing, test and evaluation, and support processes and products. The key systems engineering activities that must be performed include requirements analysis, functional analysis/allocation, design synthesis and verification, and system analysis and control. The document goes on to define 20 areas that must be considered as part of the systems engineering process. This high-level discussion ofthe systems engineering process, to be applied for major systems being acquired by DOD, should serve as the basis for tailored integration of M&S into the systems engineering processes applied to acquisition programs. DOD has developed the Handbook of Work Breakdown Structures for Major Defense Acquisition Programs and Major Automated information System Programs. This handbook is out of date and is not in accord with guidance provided by the previously discussed document. The handbook should be updated and should provide greater detail concerning the integration of M&S into the systems engineering portion of the performance work breakdown statement. DOD has undertaken an effort, supported by the Software Engineering institute (SEI), to develop and promulgate the capability maturity model integration (CMMI) for application to the development of software-intensive systems. A similar effort should be undertaken to define the structure of systems engineering processes with detailed descriptions of how M&S may be integrated into these processes. Systems engineering maturity levels should be defined across the other six domains of DOD systems engineering in addition to the definition for software-intensive systems. Furthermore, a library of process artifacts, perhaps an addition to ' Available at <http://www.acq.osd.mil/ap/dodi_5000_2_final_version_april_05_2002 Instruction.doc>. Accessed June2002.

118 MODELING AND SlMULA TION IN MANUFACTURING the Defense Acquisition Deskbook, should be established to serve as a repository for M&S integration plans and related systems engineering documents that may be referenced by new or existing programs. Recommendation. Incentive should be created and implemented for DOD program managers to adopt best practices for the use of M&S in acquisition and throughout the life cycle of military systems. The committee found that DOD program managers have no incentives to apply resources to interprogram aspects of M&S and SBA- type processes. In fact, they currently have disincentives to apply resources to M&S activities that might benefit later phases of their own programs as weld as other programs. Because the typical tenure of a program manager is relatively short compared with the life of the program, investments in M&S made early in a program (such as those that might reduce total ownership cost or improve supportability) will not produce returns until years or even decades dater, long after the program manager has departed. In an environment with severely limited funds, a natural tendency exists not to invest in activities whose benefit will not be realized during the program manager's tenure. This is even more the case when the benefits wild be realized by a different, perhaps yet uninitiated, program. Several prior M&S/SBA studies have recognized the need for incentives, although few workable solutions have emerged. The problem is undoubtedly difficl]t, and this report does not offer detailed recommendations for its resolution. Perhaps the most necessary incentive is for enlightened senior acquisition leadership to encourage and provide positive recognition for individual program managers for adopting M&S/SBA best practices that are emerging but are not yet institutionalized. This could be realized by utilizing the aforementioned acquisition maturity mode] to ensure use of these best practices before a program is a]]owed to proceed to the next phase of the DOD system . . . acquisition process. Recommendation. Pilot efforts should be defined and undertaken as a part of advancing the use of and experience in M&S. They should be sponsored at the ]eve] of the Office of the Secretary of Defense (OSD) to explore and document the benefits of co]]aborative acquisition of systems enabled by advances in M&S and information technologies. A sma]] number of well-defined pilot efforts should be undertaken. These pilot efforts should place special emphasis on exploring potential

CONCLUSIONS AND RECOMMENDATIONS 119 cross-program benefits of M&S and information-technology-enabled collaborative acquisition, and should be set up in a sequence with time phasing that leads to exploration of system-of-systems issues. This may lead to the selection of pilot efforts that are aligned with programs comparatively close in the relative phasing of their acquisition schedules. Each pilot effort should be constructed to permit the collection of data on specific metrics in order to estimate potential benefits in performance, cost, and schedule that could result from more widespread application. They should also be constructed so as to guide technical development of M&S and SBA concepts, permitting the necessary risk resulting from an emphasis on learning, and must persist long enough to ensure that the desired learning is realized. Because the nature of activities in DOD increasingly involves joint services, these pilot efforts should be sponsored at the OSD level. Although the precise mechanism for managing programs should be decided by the Under Secretary of Defense for Acquisition, Technology and Logistics (USD (AT&L)), it is recommended that DMSO and the Defense Systems Management College (DSMC) participate in an oversight role to ensure that lessons learned from the pilot efforts are shared effectively with the M&S and acquisition communities. The committee found that no single acquisition program has demonstrated comprehensive use of SBA, and that inadequate resources have been allocated to support the vision for use of M&S in military systems. Pilot efforts, if properly focused, can provide the following: A means of exploring new concepts before large-scale investments are made, Opportunities for transition of research, Identification of areas in which additional research is needed, and Evidence of benefits and other lessons learned to subsequent users. The committee notes that recommendations for pilot programs or other forms of experimentation were made in at least five previous M&S/SBA-related studies, none of which appear to have been fully acted on. The recommendation for pilot programs remains relevant today. The committee found evidence that some individual acquisition programs, such as the JSF and Future Combat Systems (FCS) programs, are planning to make investments in M&S and information-technology- enabled collaborative acquisition efforts, which can be expected to provide some valuable lessons learned to other programs. These efforts are to be encouraged. Necessarily, individual programs must focus on efforts

120 MODELING AND SIMULA TION IN MANUFACTURING expected to directly benefit their own system performance, cost, and schedule objectives, making it difficult to explore cross-program issues and to develop and collect data on well-defined metrics. Desired End State The successful implementation of these recommendations on the use of M&S in acquisition and manufacturing will only be recognized by long- term efficiencies gained in the acquisition of systems. The near-term implementation of pilot efforts should result in the establishment of metrics and areas for further work. The metrics so established will aid in defining best practices to be followed during acquisition, the use of which can be promoted and rewarded. Success of the expansion of M&S use in the concept exploration phase to "build the right thing" will be measured over time by the degree of reduction in cancellation of systems because of decreasing relevance. Finally, the better integration of M&S into the acquisition process and the resolution of ownership rights to facilitate collaboration and reuse can be measured by the realized decrease in system acquisition times and by improved avoidance of cost overruns. CULTURE AND HUMAN ISSUES Overall Recommendation. DOD must provide leadership to initiate, support, and sustain a cultural change in the acquisition process if it is to be enabled by modeling, simulation, information technologies, and development of appropriate intellectual capital in these fields. Recommendation. Concerted actions should be taken to fundamentally transform the current acquisition culture in DOD into one characterized by collaboration, cumulative learning, agility, risk tolerance, teaming from failure, and appropriate rewards and penalties. The following steps should be taken: · DOD's Senior Executive Council should set the direction by creating a vision of the desired acquisition culture and formulating and issuing policy consistent with that vision. · DOD's Business Initiative Council should institute appropriate incentives for program managers; address issues of data and model ownership, proprietary information, and intellectual property; identify and address policy, legal, and organizational barriers that inhibit SBA activities; identify and address policy issues associated with the potential international dimensions of

CONCLUSIONS AND RECOMMENDATIONS . 121 SBA; provide needed resources to implement SBA programs, and ensure consistency among service implementations of SBA. DOD's Business Initiative Council should appoint agents of cultural change to develop and implement a strategy to bring about the needed change in culture by implementing and enforcing rewards, creating a best practices manual, training stakeholders, and convening conferences. Based on numerous reports from respected sources, as described in Appendix B. the committee reaffirmed that the current acquisition culture in DOD is a fundamental impediment to effective implementation of SBA. The committee's visits, briefings, discussions, and assessments verified that the current cultural climate in DOD acquisition is characterized by organizational and functional stovepipes; a failure to share cross-phase, cross-program knowledge; bureaucratic inertia; risk aversion; limiting of goals to minimize the probability of failure; and a lack of appropriate incentives. There is a need to transform this culture into one that is characterized by cross-function and cross-program collaboration, cumulative learning, agility, risk tolerance, learning from failure, and appropriate rewards and penalties. The committee is under no illusion that such a cultural transformation will be either easy or rapid. However, if the right individuals in DOD play appropriate leadership roles and initiate an appropriate mix of actions, meaningful cultural change can be initiated and sustained. First, the highest level of senior policy makers must set the direction. The appropriate organization to perform this role is DOD's Senior Executive Council. The council is led by the Deputy Secretary of Defense and consists ofthe service secretaries and the USD(AT&L) (OSD, 2001~. The council was created in June 2001 to function as a business board of directors for DOD (DOD, 2001 a). This organization must create a vision of the culture that it wants to inculcate in the acquisition community, and then formulate and issue policy consistent with that vision. At the next level in the hierarchy, DOD has announced the creation of the Business Initiative Council, which is headed by the USD(AT&L) and composed of the service secretaries and the Vice Chairman of the Joint Chiefs of Staff. The organization is to "recommend good business practices and implement cost savings that could offset the funding requirements for personnel programs, infrastructure recapitalization, equipment modernization and transformation initiatives" (DOD, 2001b). Consistent with that charge, the Business Initiative Council should institute appropriate incentives for program managers; address and ameliorate issues of data and model ownership, proprietary information, and inte]]ectua] property; identify and address policy, legal, and organizational barriers that inhibit SBA activities; identify and address

122 MODELING AND SIMULA TION IN MANUFACTURING policy issues associated with the potential international dimensions of SBA; and provide necessary resources to implement SBA programs. In addition, the council should ensure that individual service implementations of SBA are sufficiently consistent that it will not be necessary for industry to deal with four (or more) distinctly different SBA infrastructures and processes. It would be desirable if the Business Initiative Council could formulate a grand challenge for the acquisition community that would encourage the military branches to push the boundaries of SBA. The Business Initiative Council should also appoint, anoint, and empower agents of cultural change at all levels in the SBA process. These agents would serve as champions and perform such functions as developing strategies and plans for SBA, capturing lessons learned from SBA activities, and convening meetings to support articulation of goals and to stimulate sharing of lessons learned. These agents of cultural change should develop and implement a strategy to bring about the necessary change in culture. The following represents a partial enumeration of actions that might be taken to implement such a strategy: · Reward structure. Implement and enforce rewards that encourage adherence to SBA principles. These might include promoting successful practitioners and providing adequate resources for innovative applications of SBA. Best practices. Create a living manual of best practices for SBA. This manual would capture lessons learned from both successes and failures. Education and training. Institute appropriate education and training for all of the stakeholders in the process. This would include creating and delivering short courses on SBA principles for senior decision makers (drawing on the evolving list of best practices) and ensuring that both military personnel and civilians in the acquisition process are exposed to SBA education and training at all phases in their careers. Annual conferences. Currently, multiple conferences on SBA are convened irregularly. Over time, these should evolve to an annual joint service/combined conference using the interservice/industry training, simulation, and education conference (I/ITSEC)2 as a model. Consistent with the I/ITSEC model, this conference should feature participation by senior decision makers (to articulate policy, learn state-of-the-art practice); include all stakeholders in the acquisition process (e.g., users, developers, manufacturers, support personnel, trainers); demonstrate key acquisition 2 Information concerning this conference is available at <http://iitsec.org>.

CONCL USIONS AND RECOMMENDA TIONS technologies and infrastructure; provide tutorials to educate participants on the state of the practice in SBA; and facilitate extensive networking among the participants. Recommendation. DOD should take the lead in collaborating with academia and industry to build the intellectual capital needed to implement SBA. Currently, the acquisition community lacks the intellectual capital required to implement SBA. To redress this deficiency, a number of orchestrated steps must be taken. Recommendation: Create a center of excellence for M&S in SBA. This resource would help create and promulgate the desired acquisition culture and enhance DOD's ties to the academic community. Additional steps include: · Academic degree programs in M&S. Ongoing efforts to develop academic degree programs in M&S should be supported by way of research funding related to SBA-type topics and the involvement of university faculty in commercial and DOD- sector M&S. Multiuniversity consortium. A multiuniversity consortium should be set up to continue the development of M&S education applicable to SBA. These efforts should be made in concert with the university research efforts described in these recommendations. Mentoring program. A mentoring program should be established to take advantage of the insights developed by experienced practitioners of SBA. These might include individuals in the center of excellence. Career-long learning. Individuals should be encouraged to maintain and expand their proficiency and expertise in M&S through continuing education programs and attendance at relevant conferences, meetings, and workshops. . . . 123 Although a variety of courses related to M&S is offered on most campuses, these are largely in the context of particular disciplines. For example, in a typical university, senior undergraduate and graduate courses are offered by departments such as electrical and computer engineering, management information systems, and industrial engineering. Unfortunately, these courses are too narrow in viewpoint and scope to serve the needs of M&S as a discipline capable of meeting the challenges

124 MODELING AND SIMULA TION IN MANUFACTURING of SBA. Therefore, at this time, offerings in education and training are not capable of meeting the current and future demand, and there must be significant developments in this regard. Previous panels (NRC, 1997a,b; SBAISG, 2000) have concluded that: What it is to be an M&S professional must be defined, and accreditation mechanisms must be developed; University degrees at the undergraduate and graduate levels must be defined and institutionalized; and Professional development, distinct from university degree programs, must be an essential component of the full education and training package. A small but growing number of academic institutions have recognized the situation described above and the opportunities it presents for increasing student enrollments (Fujimoto, 2000b; Sarjoughian and Cellier, 2001~. Individuals and small groups in these institutions have usually had to overcome the resistance of colleagues and the indifference of university administrations to establish new centers and degree programs centered on M&S as a discipline in its own right. The Defense Acquisition University is also taking steps to integrate M&S more fully into the education of acquisition personnel and program managers.3 An accreditation certificate program, Certified Simulation Professional, has been established that will be available to help assess the degree of capability that M&S students and working professionals have achieved.4 The M&S professional certification program may be an important motivator for developing M&S instruction in post-university professional programs and even more widely in restructuring traditional graduate and undergraduate curricula to better address the need for M&S professionals. The stability of such initiatives is not guaranteed. A healthy enrollment of students and a significant input of research funding will be required to enable these programs to flourish and to spread to other institutions. Thus, the committee endorses the following recommendations of other panels.5 . Universities need to characterize the discipline of M&S and clearly delineate the discipline from the neighboring ones such as systems engineering and computer science/ engineering; 3 Randy Zittel, Defense Systems Management College. 2001. Presentation to this study committee. 4 Information on this effort is available at <http://www.simprofessional.org> s These are described more fully in Appendix B.

COlICL USIONS AND RECOMMENDS LIONS 125 Universities must work with other sources of professional training to work out areas in which each should concentrate and identify combinations of offerings that work as a coherent whole; and Universities must work with funding agencies to establish programs of research and education needed to advance the field and to obtain adequate funding for their implementation. Further, the committee recommends that industry, in the form of M&S-based companies and corporations using M&S, should do the following: . Work with universities to characterize the current and future types of M&S professionals that they will hire and what the educational background of these professionals should be (Yurcik and Silverman, 2000~; Coordinate their education and training programs with those of universities for a coherent set of offerings; and Collaborate with universities to establish research teams that can respond to requests for proposals from government funding sources. Desired End State The desired culture for realization of substantial gains in use of M&S in manufacturing and acquisition is one in which government, academia, and industry would seek regularly to collaborate in defining needs, identifying and funding research programs to further M&S technologies to fulfill those needs, and implementing M&S in SEA. In that desired culture, individuals would be encouraged to take risks in application of M&S and would share [earnings from both successes and failures in regular forums. It would be the responsibility of program managers to push the envelope in developing and implementing M&S technologies. Academic degree programs would exist recognizing M&S as a discipline, with curricula meeting the needs of M&S in manufacturing and acquisition. Centers of excellence would provide the community with resources and leadership.

Next: References »
Modeling and Simulation in Manufacturing and Defense Acquisition: Pathways to Success Get This Book
×
Buy Paperback | $80.00 Buy Ebook | $64.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

The Committee on Modeling and Simulation Enhancements for 21st Century Manufacturing and Acquisition was formed by the NRC in response to a request from the Defense Modeling and Simulation Office (DMSO) of DOD. The committee was asked to (1) investigate next-generation evolutionary and revolutionary M&S capabilities that will support enhanced defense systems acquisition; (2) identify specific emerging design, testing, and manufacturing process technologies that can be enabled by advanced M&S capabilities; (3) relate these emerging technologies to long-term DOD requirements; (4) assess ongoing efforts to develop advanced M&S capabilities and identify gaps that must be filled to make the emerging technologies a reality; (5) identify lessons learned from industry; and (6) recommend specific government actions to expedite development and to enable maximum DOD and U.S. commercial benefit from these capabilities. To complete its task, the committee identified relevant trends and their impact on defense acquisition needs; current use and support for use of M&S within DOD; lessons learned from commercial manufacturing; three cross-cutting and especially challenging uses of M&S technologies; and the areas in which basic research is needed in M&S in order to achieve the desired goals for manufacturing and defense acquisition.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  6. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  7. ×

    View our suggested citation for this chapter.

    « Back Next »
  8. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!