DoD Engagement with Its Manufacturing Innovation Institutes Phase 2 Study Final Report (2021) / Chapter Skim
Currently Skimming:
Appendix C: Interim Report Main Text
Appendix C: Interim Report Main Text
Pages 153-190
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 153... ...
C Interim Report Main Text This appendix is a reprint of the main text of the National Academies Interim Report: DoD Engagement with Its Manufacturing Innovation Institutes: Phase 2 Study (The National Academies Press, Washington, D.C., 2021) by the Committee on DoD Engagement with Its Manufacturing USA MIIs Phase 2 Study.
|
From page 154... ...
The Committee on DoD Engagement with Its Manufacturing USA MIIs Phase 2 Study was formed to provide strategic guidance in the following topics that were identified in the 2019 report for further exploration: protocols for conducting long term engagement assessments of the MIIs including evaluation metrics, best practices for MII education and workforce development (EWD) programs, and development of strategies for better connecting MIIs to the broader DoD community.
|
From page 155... ...
Finding 2: DoD has a sound framework for conducting 5-year assessments of each institute to support such decisions. The four issue areas in the Joint Defense Manufacturing Council (JDMC)
|
From page 156... ...
and the Joint Defense Manufacturing Council should expand the evaluation framework within its four topic areas to consider additional strategic considerations focused on the Manufacturing Innovation Institute (MII) key objectives of advancing research, education and workforce development, and ecosystem.
|
From page 157... ...
Recommendation: As the Department of Defense starts the 5-year Manufacturing Innovation Institute evaluation process in 2021, it is recommended that a process be established to catalog lessons learned and assess the effectiveness of the Joint Defense Manufacturing Council evaluation process to facilitate continuous improvements. A meta-evaluation can be accomplished by appropriate groups in professional societies or within suitable not-for-profit organizations.
|
From page 158... ...
The DoD Manufacturing Innovation Institutes (MIIs) are focused on specific technology areas that have been identified as critical for the future of DoD and for which manufacturing technology maturation is an important need for technology implementation.
|
From page 159... ...
.5 DoD is initiating formal 5-year evaluations of the DoD MIIs in 2021. In order to be responsive to DoD needs, OSD ManTech asked that an interim report focused on Task 1 be provided in a timeframe that would allow the National Academies' perspectives to be used in the execution of the first MII evaluations.
|
From page 160... ...
In order to be supportive of DoD's plans for the initiation of the MII evaluations, this interim report on Task 1 will be completed within a 6-month pe riod of performance that started on October 1, 2020. Data gathering in support of the three study tasks is being performed using a combination of document reviews, one-on-one phone or video interviews, committee video meetings with DoD and MII personnel who actively support the MIIs, as well as the broader DoD com munity.6 The committee also provided a request for input questionnaire to relevant groups from DoD, industry, and the general public.
|
From page 161... ...
Ideally, the industrial base would be equipped with advanced manufacturing technologies, which could deliver critical products and systems affordably and rapidly. To better support the need for timely, effective manufacturing technology development and transition, DoD has established nine Manufacturing Innovation Institutes through its Defense-wide Manufacturing Science and Technology (DMS&T)
|
From page 162... ...
The mission of the DoD MIIs is consistent with the responsibilities for manufac turing technology assigned to the OUSD(R&E) and overseen by the OSD ManTech office.5 That office recognized the need for a coherent and effective strategy to guide the establishment and sustainment of MIIs, and also to ensure their value to DoD and the nation, by setting five strategic goals for its Manufacturing USA institutes: • Goal 1: Drive impactful advanced manufacturing research and development.
|
From page 163... ...
. The success of the evaluation framework is demonstrated by the fact that DoD has used the framework to complete 5-year, strategic evaluations of 24 DoD FFRDC and UARC organizations over a more than 20-year period.
|
From page 164... ...
5-year assessment process for the Manufacturing Innovation Institutes (MIIs)
|
From page 165... ...
DoD has produced the "2020 DoD Metrics List" and the "Final Metrics List for the DoD MIIs."14 These documents, representing the output from the latest activity on metrics development for the DoD MIIs, began with activities sponsored by the Advanced Manufacturing National Program Office,15 which examined metrics for all of the manufacturing institutes sponsored by federal agencies. As the MIIs have matured, the metrics used 14 Excel books: "Final DoD MII Performance Metrics_cleared" and "2020 MII Complete Metrics List_cleared," both received from ManTech on December 11, 2020.
|
From page 166... ...
to assess performance have grown from the initial, organizational development-fo cused metrics to the output focused-metrics and impact-focused metrics Based on these metrics lists, OSD ManTech currently collects a considerable amount of data on the MIIs on a weekly, quarterly, semi-annual, or annual basis. The information topics include: Ecosystem (e.g., patents filed and licensed, successful technology transitions, extent of member partnering, state involvement in MII)
|
From page 167... ...
Appendix C 167 Reprinted from Interim Report FIGURE 2.4 An example of a section of the metrics for evaluating the Ecosystem currently gathered by OSD ManTech. SOURCE: Final DoD MII Performance Metrics received from ManTech on December 11, 2020.
|
From page 168... ...
Federal program evaluation processes are often based on a set of core principles that are consistent across agencies. A good example is NSF's evaluation policy, which was updated in 2020.3 NSF notes that assessments can have different pur poses, such as monitoring progress, guiding improvement efforts, or determining the effectiveness or efficiency of a program.
|
From page 169... ...
The first important consideration of the evaluation process is the selection of the evaluation team. The review team leads are selected based on their recognized leadership in the community and their relevant expertise in the technology area.
|
From page 170... ...
The earliest activity in the NASA process is a planning session between the review team leads and program leadership, which sets the expectations for the data and information that will be required in the review process. NASA has formal data delivery requirements set at 60 and 20 days before the site visit.
|
From page 171... ...
Guidance to DoD MII Evaluation Process Even though the DoD MIIs have significantly different goals than the Agency programs described above, the DoD 5-year evaluation of the MIIs can benefit from insights and lessons learned provided by these guides and earlier studies. First, the selection of the review team lead is obviously critical.
|
From page 172... ...
Collaborative planning activities and the development of a TOR-like agreement among the JDMC team lead, OSD ManTech, and MII leadership will enable the development of a transparent evaluation process that is balanced between DoD's needs for a strategic assessment and the time and cost resource requirements placed on the MII. In terms of information and data provided prior to the site visit, these materials would include OSD ManTech's per spectives on specific attributes of the MII being evaluated and the MII-prepared responses to the relevant metrics-driven questions developed by OSD ManTech.
|
From page 173... ...
In addition, the repository of the 5 year evaluation results for all of the MIIs will enable the identification of lessons learned and best practices from individual MIIs which can be incorporated into future reviews and which can support the general improvements within the set of MIIs. STRATEGIC EVALUATION CRITERIA FOR DOD MIIS The JDMC evaluation of the DoD Manufacturing USA institutes provides an opportunity to better insure and document the value of the MIIs for DoD.
|
From page 174... ...
As noted in the section "Evaluation Process for DoD MIIs" above, in anticipation of the MII evaluation, the JDMC team lead will need to work with OSD ManTech and MII leadership to develop a department assessment of the ap propriate characteristics that then can be evaluated and the relevant metrics and discussion topics that also can be applied to the specific MII. As has been done by other Agencies, the DoD should have a broad set of evaluation topics within the framework which can be used to guide the design of an evaluation process for a particular MII.
|
From page 175... ...
4. Does DoD see a need for an ecosystem to drive advanced manufacturing technology implementation activities (e.g., road mapping, community convening, standards and regulations development, etc.)
|
From page 176... ...
The unique organizational characteristic of the MIIs is that they are industry-led, public–private partnerships, unlike other sources available to DoD. MIIs were chartered8 and organized for establishing and growing a manufacturing ecosystem, advancing research and technology, and securing human capital.
|
From page 177... ...
economy and national security.11,12 Support is needed for emerging technologies such as directed-energy weapons, hypersonics quantum and cybersecurity, all of which are vital to national defense.13,14 A strong manufacturing sector not only ensures a ready supply of defense and commercial goods and services, but also ensures the integrity and safety of these goods, such as electronics and control systems. Advanced manufacturing encompasses all aspects of manufacturing, including the ability to quickly respond to customer needs through innovations in product 10 As provided by ManTech through the Excel books: "Final DoD MII Performance Metrics_ cleared" and "2020 MII Complete Metrics List_cleared," both received from ManTech on December 11, 2020.
|
From page 178... ...
Hence, establishing and growing manufacturing ecosystems is one of the three chartering principles of Manufacturing USA MIIs. Since DoD MIIs vary in the nature, maturity, and intended use of their focused technology, each MII and its stakeholders need to identify the shared challenges and shared opportunities that their ecosystem wishes to develop, grow, and maintain.
|
From page 179... ...
4. Assess evidence of MII impact on the robustness and resilience of DoD supply chains (e.g., multiple sources for technology, technology companies linked to system providers or government supply chain, sufficiently trained workers, foundation provided in emerging technology by institutes)
|
From page 180... ...
During an MII evaluation process, a key question that can be addressed is, What is limiting the rate of maturation of the technology or a resilient ecosystem for this advanced manufacturing field? Are there technology gaps, a lack of standards, inadequate equipment and facilities, a lack of qualified suppliers, insufficient demand, untrained workforce, etc.?
|
From page 181... ...
While numerical data provides a useful metric for MII performance, the committee has done a detailed review of the kinds of EWD programs undertaken by MIIs and developed a list of "best practices" that various MIIs are undertaking. While these best practices are less amenable to metrics, evidence that best practices are being applied represents an important indicator of program quality.
|
From page 182... ...
The data collected from the annual reviews, along with the trends in the data, will be very useful to the JDMC evaluation committee performing the 5-year assessment. In addition, the JDMC review can address the MIIs progress toward achieving DoD's strategic goals for the MIIs, as well as DoD's overall mission needs for manufacturing tech nology.
|
From page 183... ...
STRATEGIC QUANTITATIVE METRICS All of the topics in the Strategic Evaluation Criteria for DoD MIIs section above represent qualitative metrics which are directly relevant to the three key strategic objectives for the MIIs (advancing research and technology, establishing and growing a manufacturing ecosystem, and securing human capital)
|
From page 184... ...
The institutes also categorize these metrics in total, by non-DoD government entity, by DoD service, by state or local government, and by commercial entity. In addition to the metrics currently gathered, the committee suggests additional, strategic assessment should include: MIIs must develop a process to track demonstrated outcome impact of completed projects on cost, lead time, and/or platform performance enhancements, and MIIs should track their impact on the roadmaps for the various technologies they tasked with and how these impacts fit within the larger community contributions. The additional metrics suggested for the research and technology objective are: 1.
|
From page 185... ...
This includes tracking the number of successful technology transitions, number of patents filed and used, number of members on funded projects, level of member cash to inkind contributions, level of ecosystem participation, number of DoD subject matter experts engaged, number of states involved and level of state contributions. For the long-term strategic evaluation, the committee suggests that the institutes define the advanced manufacturing ecosystem which the MII and its stakeholders envision to serve the needs of its community.
|
From page 186... ...
The MII should be asked to develop trendlines of this data for the MII's period of operations, applied against workforce education goals set out in the MII's skill roadmaps, to indicate progress in meeting strategic, overall skills train ing demands in the MII's manufacturing technology sector. The additional metric suggested for the securing human capital (EWD)
|
From page 187... ...
4. The JDMC review team's assessment of the information provided at the site visit.
|
From page 188... ...
. FINDINGS Finding 1: As noted in Fiscal Year 2020 Industrial Capabilities Report to Congress,1 support for the DoD MIIs remains high among senior leadership at DoD, but the declining budget request for core funding will require difficult decisions.
|
From page 189... ...
to support decisions on renewal of agreements and to provide budget justification for future years commensurate with the value of the MII program. Recommendation: OSD ManTech and the Joint Defense Manufacturing Council should expand the evaluation framework within its four topic areas to consider additional strategic considerations focused on the Manufacturing Innovation Institute (MII)
|
From page 190... ...
Recommendation: The Joint Defense Manufacturing Council 5-year assessment should address strengths, weaknesses, and opportunities for improvement of the Manufacturing Innovation Institutes (MIIs) relative to non-Department of Defense (DoD)
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.