C

Interim Report Main Text

This appendix is a reprint of the main text of the National Academies Interim Report: DoD Engagement with Its Manufacturing Innovation Institutes: Phase 2 Study (The National Academies Press, Washington, D.C., 2021) by the Committee on DoD Engagement with Its Manufacturing USA MIIs Phase 2 Study.

Reprinted from Interim Report

Summary

To better support the need for timely, effective manufacturing technology development and transition, the Department of Defense (DoD) has established nine Manufacturing Innovation Institutes (MIIs) through its Defense-wide Manufacturing Science and Technology (DMS&T) program element within the Office of the Secretary of Defense (OSD) Manufacturing Technology (ManTech) program. The Institutes are considered by DoD to be important facilitators that bring together innovative ecosystems in key technology and market sectors in the United States. The DoD MIIs are industry-led, with dual, public–private benefit, providing large commercial market potential while also meeting key U.S. defense industrial needs. The mission of the nine DoD-established Institutes addresses both defense and commercial manufacturing needs within specific, defense-relevant technology areas.

DoD’s commitment of over $1.12 billion to date for these nine Institutes demands a coherent and effective strategy to guide their establishment and sustainment, and also to ensure their value to DoD and the nation.

The National Academies of Sciences, Engineering, and Medicine was asked to perform a follow-on study to its 2019 report Strategic Long-Term Participation by DoD in Its Manufacturing USA Institutes,¹ which examined DoD’s strategic relationship with the DoD Manufacturing Innovation Institutes. The Committee on DoD Engagement with Its Manufacturing USA MIIs Phase 2 Study was formed to provide strategic guidance in the following topics that were identified in the 2019 report for further exploration: protocols for conducting long term engagement assessments of the MIIs including evaluation metrics, best practices for MII education and workforce development (EWD) programs, and development of strategies for better connecting MIIs to the broader DoD community. The sponsor of the work at OSD ManTech asked the National Academies to provide an interim report focused on the protocol assessment topic in a timeframe that would allow its perspectives to be used in the execution of the first DoD 5-year MII evaluations planned for 2021. This interim report is focused on that topic. The final report from this study will document findings and recommendations relevant to EWD best practices and DoD engagement strategies. The statement of task for this study is reprinted in Appendix A.

___________________

¹ National Academies of Sciences, Engineering, and Medicine, 2019, Strategic Long-Term Participation by DoD in Its Manufacturing USA Institutes, Washington, DC: The National Academies Press, https://doi.org/10.17226/25417.

KEY FINDINGS AND RECOMMENDATIONS

Findings

Finding 1: As noted in Fiscal Year 2020 Industrial Capabilities Report to Congress,² support for the DoD MIIs remains high among senior leadership at DoD, but the declining budget request for core funding will require difficult decisions. At the 5-year review point for each institute, DoD will need to decide whether to continue core funding support, possibly at a reduced level or at a higher level, or decline to renew the agreement in order to apply available budget to higher priorities.

Finding 2: DoD has a sound framework for conducting 5-year assessments of each institute to support such decisions. The four issue areas in the Joint Defense Manufacturing Council (JDMC) evaluation plan³ are based on an assessment process which is familiar to DoD and will facilitate the assessment of the MIIs within the context of DoD’s mission needs. The planned evaluation process is similar to good practices the committee has seen in other agencies. The JDMC, based on its charter, is well suited to conduct the 5-year assessments. Over time, the committee believes the 5-year reviews can improve the coupling between public and private interests in each MII and move all MIIs toward collective implementation of best practices. Equally important, the JDMC can play a key role in the integration of the MIIs into DoD’s technology delivery pipeline, which will be further explored in this committee’s work as described by the third task in the statement of task for this study, and presented in the final report.

Finding 3: The metrics needed for strategic 5-year assessments are not limited to the metrics used for ongoing management and oversight of the MIIs. DoD has a robust set of metrics for quarterly reviews that are important to assessing progress toward annual performance objectives. The 5-year reviews are more strategic in nature. Additional topics that are appropriate in the 5-year assessment would include the continued need, appropriateness of the public–private partnership, and long-term performance trends of the MII. Strategic metrics would also be appropriate for the 5-year reviews.

Finding 4: The unique public–private partnership aspect of the MIIs requires strategic consideration of the health of the “private” component of the institutes. To the

___________________

² Department of Defense, 2021, Fiscal Year 2020 Industrial Capabilities Report to Congress, https://media.defense.gov/2021/Jan/14/2002565311/-1/-1/0/FY20-INDUSTRIAL-CAPABILITIES-REPORT.PDF.

³ The MII Strategy and Assessment_ManTech_Oct2020_cleared PowerPoint, received on November 9, 2020, from ManTech.

extent that DoD views these cost-shared private–public partnerships as essential, an additional consideration would be input from MII members and non-members from the ecosystem.

Finding 5: Greater involvement of the DoD acquisition and sustainment communities are needed for DoD to fully benefit from the technology development, EWD, and manufacturing ecosystem of DoD MIIs. The JDMC has an opportunity to use the 5-year assessments and the growth of this engagement over time as a key measure for the impact of the MIIs on DoD advanced technology development and implementation.

RECOMMENDATIONS

Recommendation: The Department of Defense should phase in formal 5-year assessments of the Manufacturing Innovation Institutes (MIIs) to support decisions on renewal of agreements and to provide budget justification for future years commensurate with the value of the MII program.

Recommendation: The Office of the Secretary of Defense Manufacturing Technology Program (OSD ManTech) and the Joint Defense Manufacturing Council should expand the evaluation framework within its four topic areas to consider additional strategic considerations focused on the Manufacturing Innovation Institute (MII) key objectives of advancing research, education and workforce development, and ecosystem. Design of the evaluation team and details of the evaluation tasks should consider the best practices of other agencies discussed in Chapter 3, including provisions for access to outside technical experts as needed. The process should be tailored to the unique characteristics of the MII public–private partnerships and should be made available to the MIIs and the rest of the manufacturing community.

Recommendation: The Office of the Secretary of Defense Manufacturing Technology Program (OSD ManTech) and the Joint Defense Manufacturing Council should amend the planned evaluation process to make provisions for additional strategic metrics and trend analysis, and for additional sources of input beyond those currently planned, such as the following: other stakeholder survey results, assessment of the evaluation criteria, site visit results, and terms of reference planning agreements. In assessing effectiveness of governance and management, specific attention should be given to the effectiveness of government and industry leadership in achieving desired outcomes.

Recommendation: The Joint Defense Manufacturing Council 5-year assessment should address strengths, weaknesses, and opportunities for improvement of the Manufacturing Innovation Institutes (MIIs) relative to non-Department of Defense (DoD) components of the public–private partnership to inform decisions on whether a public–private partnership is the best way to meet DoD needs. The Office of the Secretary of Defense Manufacturing Technology Program (OSD ManTech) should also perform a stakeholder survey as a component of the 5-year MII assessment. The survey should include all stakeholders such as large, medium, and small businesses, academia, state governments, and foundations that are within the MII ecosystem.

Recommendation: As the Department of Defense starts the 5-year Manufacturing Innovation Institute evaluation process in 2021, it is recommended that a process be established to catalog lessons learned and assess the effectiveness of the Joint Defense Manufacturing Council evaluation process to facilitate continuous improvements. A meta-evaluation can be accomplished by appropriate groups in professional societies or within suitable not-for-profit organizations.

1

Introduction

Since 2012, sixteen Manufacturing USA Institutes have been established by the federal government. Nine of these organizations have been sponsored by DoD, under OSD ManTech. The DoD Manufacturing Innovation Institutes (MIIs) are focused on specific technology areas that have been identified as critical for the future of DoD and for which manufacturing technology maturation is an important need for technology implementation. The DoD MIIs are as follows:

• AFFOA (Advanced Functional Fabrics of America) (2016) is working to enable a manufacturing-based revolution by transforming traditional fibers, yarns, and fabrics into highly sophisticated, integrated, and networked devices and systems.
• AIM Photonics (American Institute for Manufacturing Integrated Photonics) (2015) is working to accelerate the transition of integrated photonic solutions from innovation to manufacturing-ready deployment in systems spanning commercial and defense applications.
• America Makes (National Additive Manufacturing Innovation Institute) (2012) is a national accelerator and the nation’s leading collaborative partner for technology research, discovery, creation, and innovation in additive manufacturing and 3D printing.
• ARM (Advanced Robotics Manufacturing) (2017) has the mission to create and then deploy robotic technology by integrating the diverse collection of industry practices and institutional knowledge across many disciplines—sensor technologies, end-effector development, software and artificial intelligence, materials science, human and machine behavior modeling, and quality assurance—to realize the promises of a robust manufacturing innovation ecosystem.
• BioFabUSA (Advanced Regenerative Manufacturing Institute) (2017) has the mission to make practical the large-scale manufacturing of engineered tissues and tissue-related technologies to benefit existing industries and grow new ones.
• MxD (Manufacturing x Digital Institute) (2014) is working to enable factories across the United States to deploy digital manufacturing and design technologies so that those factories can become more efficient and cost competitive.

• LIFT (Lightweight Innovations for Tomorrow) (2014) is working to develop and deploy advanced lightweight materials manufacturing technologies.
• NEXTFLEX (America’s Flexible Hybrid Electronics Manufacturing Institute) (2015) has the mission of adding electronics to new and unique materials that are part of everyday lives in conjunction with the power of silicon integrated circuits to create conformable and stretchable smart products.
• BioMADE (Biomanufacturing and Design Ecosystem) (2020) seeks to advance sustainable and reliable bio-industrial manufacturing technologies.

In 2019, the director of OSD ManTech asked the National Academies of Sciences, Engineering, and Medicine to perform a study on how DoD should engage with DoD’s MIIs in the future. The National Academies’ study made a series of recommendations and suggestions for further study.⁴ In 2020, OSD ManTech asked the National Academies to perform a second fast-track study to provide strategic guidance in three important suggestions from the first study: protocols for conducting long term engagement assessments of the MIIs including evaluation metrics (Task 1), best practices for MII EWD programs (Task 2), and development of strategies for better connecting MIIs to the broader DoD community (Task 3).⁵ DoD is initiating formal 5-year evaluations of the DoD MIIs in 2021. In order to be responsive to DoD needs, OSD ManTech asked that an interim report focused on Task 1 be provided in a timeframe that would allow the National Academies’ perspectives to be used in the execution of the first MII evaluations. The final report from this Phase 2 study will document findings and recommendations relevant to Tasks 2 and 3.

STUDY OBJECTIVE

The statement of task (SOT) for this study is shown in Appendix A. The charge to the committee was to provide general strategic guidance in three topical areas, as follows:

• Protocols for conducting long-term engagement assessments of the MIIs, including evaluation criteria;
• Best practices in education and workforce development (EWD) for the MIIs, including scale-up methodologies for collaborative efforts by the MIIs in EWD; and

___________________

⁴ National Academies of Sciences, Engineering, and Medicine, 2019, Strategic Long-Term Participation by DoD in Its Manufacturing USA Institutes, Washington, DC: The National Academies Press, https://doi.org/10.17226/25417.

⁵ See Appendix A for the statement of task.

• Steps for improving MII linkages with other parts of the DoD and with the broader federally funded research enterprise;

The output of this fast-track project will be a short interim consensus report addressing the first bullet of the SOT and a final consensus report addressing the remaining two bullets of the SOT that draws from the information gathered by the committee and other public information sources, as needed, and provides findings, options, and, if the information gathered justifies them, recommendations for DoD to consider in developing its long-term role with existing and potential future MIIs sponsored by DoD in the three areas identified.

STUDY APPROACH

This Phase 2 study is a “rapid response” study with a 10-month period of performance. In order to be supportive of DoD’s plans for the initiation of the MII evaluations, this interim report on Task 1 will be completed within a 6-month period of performance that started on October 1, 2020. Data gathering in support of the three study tasks is being performed using a combination of document reviews, one-on-one phone or video interviews, committee video meetings with DoD and MII personnel who actively support the MIIs, as well as the broader DoD community.⁶ The committee also provided a request for input questionnaire to relevant groups from DoD, industry, and the general public. The committee convened to discuss these inputs and to formulate its consensus findings and recommendations. A final report will be submitted that fully addresses all three tasks, with a focus on Tasks 2 and 3.

___________________

⁶ See Appendix B for the list of briefers.

2

Background

DOD MANUFACTURING INSTITUTES

DoD has a strong tradition of innovation development and implementation in support of next-generation defense systems. Within DoD’s technology development enterprise, there is constant attention to the maturation of technologies that support the challenging performance goals of future military systems. It is recognized that, in order to effectively mature and transition DoD science and technology advances into production, access to a robust and responsive U.S. industrial base is essential. Ideally, the industrial base would be equipped with advanced manufacturing technologies, which could deliver critical products and systems affordably and rapidly.

To better support the need for timely, effective manufacturing technology development and transition, DoD has established nine Manufacturing Innovation Institutes through its Defense-wide Manufacturing Science and Technology (DMS&T) program element within the DoD Manufacturing Technology (ManTech) program. The institutes are considered by DoD to be crucial facilitators that bring together innovative often non-traditional ecosystems in key technology and market sectors in the United States, enabling their vibrancy and robustness.¹ The DoD Manufacturing Innovation Institutes are industry-led, with a dual public–private benefit, providing large commercial market potential while meeting key U.S. defense needs. The mission of the nine DoD-established institutes addresses both commercial and defense manufacturing needs within specific, defense-relevant technology areas.

As is shown in Figure 2.1, DoD has committed approximately $1.12 billion to date for these nine institutes,² and non-DoD partners have contributed approximately $1.93 billion. As noted in Fiscal Year 2020 Industrial Capabilities Report to Congress,³ DoD views the MII public–private partnerships as a means to achieve significant advances in important manufacturing technologies in a cost-constrained environment. However, budgeted core funding for the institutes is

___________________

¹ See Department of Defense (DoD), 2017, Department of Defense Manufacturing USA Strategy, Version Date September 8, 2017, Director DoD Manufacturing Technology Program, OUSD(R&E) Strategic Technology Protection and Exploitation.

² DoD, 2021, Fiscal Year 2020 Industrial Capabilities Report to Congress, p. 157, https://media.defense.gov/2021/Jan/14/2002565311/-1/-1/0/FY20-INDUSTRIAL-CAPABILITIES-REPORT.PDF.

³ Ibid.

planned to decrease from the FY 2020 appropriated level of $167 million to about $64 million in FY 2022-2025,⁴ which will require difficult decisions by DoD and the institutes on how to accommodate the drop in future funding.

The mission of the DoD MIIs is consistent with the responsibilities for manufacturing technology assigned to the OUSD(R&E) and overseen by the OSD ManTech office.⁵ That office recognized the need for a coherent and effective strategy to guide the establishment and sustainment of MIIs, and also to ensure their value to DoD and the nation, by setting five strategic goals for its Manufacturing USA institutes:

• Goal 1: Drive impactful advanced manufacturing research and development.
• Goal 2: Encourage the creation of viable and sustainable institute business plans.
• Goal 3: Maintain an optimal program design to maximize value delivery.
• Goal 4: Maximize stakeholder understanding of DoD’s Manufacturing USA institutes.
• Goal 5: Effectively support a capable workforce.⁶

The OSD ManTech office provides oversight of the DoD institutes and has performed annual reviews of all of the MIIs since the inception of the institutes.

Evaluation Criteria and Metrics for the DoD MIIs

The federal government has long recognized the need for periodic program assessments to assure the performance, cost, and effectiveness of taxpayer-funded programs. Government agencies, such as the Office of Management and Budget

___________________

⁴ Budget Exhibit R-2 for PE 060368D8Z: Defense Wide Manufacturing Science and Technology, Feb 2020

⁵ DoD, 2018, “DoD Directive 5200.14: Manufacturing Technology (ManTech) Program,” Change 1, October.

⁶ Ibid.

and the Government Accountability Office, and professional societies, such as the American Evaluation Association and the National Academies, have performed a number of studies on best practices and strategies for assessment of a wide variety of federal programs.^7,8,9 DoD was an early adopter and strong proponent of formal program evaluation processes. In 1961, DoD established the Office of Systems Analysis to perform the program evaluation function. Since then, requirements for program evaluation have been included in DoD policies and guidebooks for a wide range of defense programs and organizations. One example is the Comprehensive Review Guidelines for DoD federally funded research and development centers (FFRDCs).¹⁰

The evaluation framework that has been developed by DoD for the MIIs is based on one that DoD has used successfully in review of FFRDCs and University Affiliated Research Centers (UARCs). The success of the evaluation framework is demonstrated by the fact that DoD has used the framework to complete 5-year, strategic evaluations of 24 DoD FFRDC and UARC organizations over a more than 20-year period. The evaluations performed, based on this framework, are used by senior DoD officials as the basis for decisions to renew the FFRDC or UARC, to reopen the competition in the core competency areas, or to end the activity. This framework is consistent with the models of evaluation processes that have been proposed and used by many federal agencies as well. For example, a National Academies study that examined major National Science Foundation (NSF) program recompetitions and suggested that the two key questions for reviewers to assess were the following:

1. Is this still a worthwhile activity or has it become obsolete in the face of scientific and technological advances?; and
2. If the activity is deemed worthwhile, is the current grantee doing a good job that merits continuation or should the award be opened to competition?¹¹

___________________

⁷ Phase 4 Implementation of the Foundations for Evidence-Based Policymaking Act of 2018: Program Evaluation Standards and Practices, https://www.whitehouse.gov/wp-content/uploads/2020/03/m-20-12.pdf. Accessed on 11/23/2020.

⁸ An Evaluation Roadmap for a More Effective Government, https://www.eval.org/evaluationroadmap, accessed November 23, 2020.

⁹ NASEM, 2017, Principles and Practices for Federal Program Evaluation: Proceedings of a Workshop, Washington, DC: The National Academies Press, https://doi.org/10.17226/24831.

¹⁰ USD(AT&L) Policy Memo, Subject: FFRDC Management Plan and Associated “How To” Guides, May 2, 2011.

¹¹ National Academy of Sciences, National Academy of Engineering, and Institute of Medicine, 1994, Major Award Decisionmaking at the National Science Foundation, Washington, DC: The National Academies Press, https://doi.org/10.17226/2268. See Chapter 5, “Recompetition of Awards,” p. 110.

The OUSD(R&E) evaluation process takes a similar approach but has additional considerations which are relevant for the more mission-focused DoD MIIs. The DoD will assess the MIIs in four key question areas, as seen in Figure 2.2, which is taken from the MII Strategy and Assessment document.¹²

OUSD(R&E) plans to use the Joint Manufacturing Defense Council (JDMC) as an independent source to perform the 5-year strategic evaluations of the MIIs, as seen in Figure 2.3.¹³ The JDMC, chartered in 2020, is an OSD-led organization with 18 organizational members within DoD; it is tasked with coordination and strategic guidance of advanced manufacturing activities across DoD. The mission of the organization is:

___________________

¹² Table from the MII Strategy and Assessment_ManTech_Oct2020_cleared PowerPoint, received on November 9, 2020, from ManTech.

¹³ JDMC Charter, https://www.DoDmantech.com/JDMTP/Files/JDMTP_SubPanels_Charter_2011.pdf, accessed January 15, 2021.

Important goals of the JDMC include the following: alignment of defense research, manufacturing, maintenance, and supply capabilities; increased collaboration and efficiencies in DoD manufacturing activities; development of strategies for a robust organic and commercial manufacturing base; and development of DoD’s human capital plans in coordination with other national programs. Since the MIIs address these goals, it is appropriate that the JDMC serve as the evaluator for this substantial DoD investment in advanced manufacturing. In addition, the JDMC is well positioned to not only assess the performance of the MIIs but also to facilitate increased visibility of the MIIs within DoD and enhance the impact of the institutes for DoD. The JDMC has been formally charged with organization and execution of the periodic, 5-year reviews of the MIIs. The JDMC will put together an evaluation team to perform the review of an MII. The team will be responsible for the assessment of the value and performance of the MII in support of Department mission goals. The team membership will include all of the expertise required to complete the assessment of performance in the critical elements of the MII (e.g., manufacturing technology, education and workforce development, and ecosystem). The team will primarily consist of personnel from the JDMC organizations but may be supplemented with additional experts as needed. The deliverable from the JDMC evaluation team will be a recommendation to OSD ManTech and OUSD(R&E) on the continuation of core funding to the MII. Senior DoD leadership will then incorporate JDMC evaluation team’s recommendations into broader strategic considerations within the department and make the decision on the future of the MII. OSD ManTech will manage the evaluation process for all of the MIIs. These reviews will be performed on a rotating basis and are scheduled to start in 2021.

In addition to the development of a strategy and approach for periodic independent evaluation of the institutes, DoD has made significant progress in the development of metrics for assessing MII performance. DoD has produced the “2020 DoD Metrics List” and the “Final Metrics List for the DoD MIIs.”¹⁴ These documents, representing the output from the latest activity on metrics development for the DoD MIIs, began with activities sponsored by the Advanced Manufacturing National Program Office,¹⁵ which examined metrics for all of the manufacturing institutes sponsored by federal agencies. As the MIIs have matured, the metrics used

___________________

¹⁴ Excel books: “Final DoD MII Performance Metrics_cleared” and “2020 MII Complete Metrics List_cleared,” both received from ManTech on December 11, 2020.

¹⁵ National Network of Manufacturing Institutes, 2015, Guidance on Institute Performance Metrics, https://www.manufacturingusa.com/reports/guidance-institute-performance-metricsnational-network-manufacturing-innovation.

to assess performance have grown from the initial, organizational development-focused metrics to the output focused-metrics and impact-focused metrics Based on these metrics lists, OSD ManTech currently collects a considerable amount of data on the MIIs on a weekly, quarterly, semi-annual, or annual basis. The information topics include: Ecosystem (e.g., patents filed and licensed, successful technology transitions, extent of member partnering, state involvement in MII), EWD (e.g., EWD projects underway and completed, number of manufacturing workforce participants in EWD activities, number of participants in Institute-aligned training, apprenticeship, or certification programs), Financial (e.g., core and non-core funding levels (from DoD, federal agencies, state agencies, industry, Foundations, etc.)) and Operations (e.g., projects completed, projects meeting objectives, number of small and large companies involved in projects, number of Government directed projects), An example of the information gathered via the metrics lists is shown in Figure 2.4. The JDMC will benefit from all of the data gathered by OSD ManTech in the completion of their independent 5-year assessment of the MIIs.

3

Discussion

EVALUATION PROCESS FOR DOD MIIS

Best Practices from Evaluation Processes in the Federal Sector

As noted above, numerous federal agencies perform formal independent assessments of major programs and use well-defined procedures in the reviews.^1,2 The agencies’ processes often start from a generic set of principles and procedures and then develop into a tailored process that reflects the unique nature of the organization or program being reviewed. The committee’s purpose in reviewing the practices of other agencies is to identify useful guiding principles that could underpin the DoD unique evaluation process.

Federal program evaluation processes are often based on a set of core principles that are consistent across agencies. A good example is NSF’s evaluation policy, which was updated in 2020.³ NSF notes that assessments can have different purposes, such as monitoring progress, guiding improvement efforts, or determining the effectiveness or efficiency of a program. The five evaluation principles in the policy, along with some relevant text from the policy sections, are reproduced here:

1. Relevance and Utility: Address questions of importance, address information needs of the stakeholders.
2. High Quality and Rigor: Provide credible evidence while balancing time and resources for obtaining it, use appropriate evaluation designs to address key questions, promote proper interpretation of findings.
3. Independence and Objectivity: Evaluations strive for objectivity in planning, conducting, interpreting and disseminating findings, and evaluators shall be insulated from influence which might affect their objectivity.
4. Transparency and Reproducibility: NSF promotes transparency in the planning, implementation, and reporting phases, to the extent possible,

___________________

¹ Government Accountability Office (GAO), 2021, Technology Assessment Design Handbook, GAO-21-347G, February, https://www.gao.gov/products/GAO-21-347G.

² See National Science Foundation (NSF), 2020, Business Systems Review (BSR) Guide, NSF 21-046, October, https://www.nsf.gov/pubs/2021/nsf21046/nsf21046.pdf.

³ See NSF, 2020, “Evaluation Policy,” September, https://www.nsf.gov/od/oia/eac/PDFs/nsf_evaluation_policy_september_2020.pdf.

4. decisions regarding the specification of the evaluation—such as objectives, design and methods, and dissemination of findings, will be documented ahead of time.
5. Ethics: Evaluations shall be planned and implemented to safeguard the dignity, rights, safety, and privacy of participants, stakeholders, and other affected entities, and be equitable, fair, just, and consider contextual factors which could influence findings or their interpretation.⁴

In addition to these core principles, some agencies have published detailed guidance to support the development and execution of evaluation processes. For instance, NSF has a guide to support major facility program reviews;⁵ likewise, the National Aeronautics and Space Administration (NASA) has the NASA Standing Review Board Handbook for mission reviews.⁶ These are broad guideline documents intended to cover a wide range of agency programs. Both documents assert that specific program reviews in their agencies would need to tailor a review for the unique attributes of the program being evaluated from the larger set of evaluation processes and related metrics. It is useful for DoD’s MII evaluation process to consider some of the common themes from these guides.

The first important consideration of the evaluation process is the selection of the evaluation team. The review team leads are selected based on their recognized leadership in the community and their relevant expertise in the technology area. The review team lead is also responsible for planning and scheduling review activities as well as for assuring consistent implementation of agency policies and processes in the evaluations. NASA suggests a separate review manager, who works with the review team lead, to address these additional responsibilities. The review team lead is responsible for developing a candidate list of team members. The evaluation process requires a diverse team of experts with knowledge on all of the essential elements of the program. Depth of knowledge in the technology area is important, but capabilities in management, programmatics, testing, or integration competencies might also be relevant. NASA suggests that having a review team that supports all of the formal evaluations of a program over its life has benefits in terms of continuity and familiarity with the program’s purpose and history. Also, a best practice suggested is to keep the team size to the minimum required to cover all important aspects of the evaluation.

Any evaluation process needs to be tailored for the specific characteristics of the program reviewed. NASA’s guide mentions the importance of factoring in

___________________

⁴ Ibid.

⁵ See https://www.nsf.gov/bfa/lfo/docs/BSR_Guide_October2020_Draft.pdf.

⁶ NASA, 2014, NASA Standing Review Board Handbook, NASA/SP-2014-3706, REV A, https://ntrs.nasa.gov/citations/20140008530.

program maturity in developing the review process. Also, both NASA and NSF’s guides stress the need for sensitivity to the cost and organizational burdens of the review process for the program being reviewed.

Each of the guides divide activities in the review broadly into pre-site visit, site visit, and post-site visit phases. The pre-site visit phase is where all of the information necessary to support the evaluation process is gathered. For both NSF and NASA, the pre-site visit activities start many months prior to the site visit. NSF’s guide mentions activities starting 4 months in advance, and NASA’s is earlier at 6 months in advance of the site visit. The earliest activity in the NASA process is a planning session between the review team leads and program leadership, which sets the expectations for the data and information that will be required in the review process. NASA has formal data delivery requirements set at 60 and 20 days before the site visit.

A related activity is the development of and the agreement to the terms of reference (TOR) for the evaluation. The review team lead is responsible for drafting the TOR, which covers the nature, scope, schedule, and ground rules for the review. The team lead then works with the government convening authorities and the program leadership to collaboratively develop a TOR that meets the agency’s assessment expectations. NSF’s process also includes formal scheduling, planning, and scoping activities. As the site visit draws near, there are reviews made of the progress in assembling the supporting information and data that are needed for the formal evaluation and coordination meetings between the review team and the program leadership. The site visit is typically 1 to 3 days in length, depending on the size and complexity of the program. The review team lead is responsible for developing the agenda in concert with the program leadership. During the pre-site visit phase and at the site visit, the review team—both individually and as a group—perform an assessment of the program’s status against a set of core requirements or strategic criteria. The NSF guide provides a comprehensive catalog of assessment questions organized in focus area modules to support the activities of the review team. The team uses the set of modules and questions to tailor the assessment of the specific program. In the NASA process, the team develops an assessment of performance against a set of criteria and reports status as successful, partially successful, or unsuccessful.

The post-site visit activities include report generation and briefing schedules, which again are tailored for different types of programs. Interestingly, in NASA’s process, there is a requirement that the review lead provide a “Snapshot Report” 24 to 48 hours after the site visit. This report is a one-page summary of the program review, which provides a summary of the review team’s findings and a discussion of key issues and risks.

Another useful source of insights for the DoD MII evaluation process is the National Institute of Standards and Technology (NIST)-led Manufacturing Exten-

sion Partnership (MEP) program.⁷ MEPs are public–private partnerships that are dedicated to support small- and medium-sized manufacturing companies. MEPs and MIIs share a focus on development and growth of manufacturing ecosystems. The MEP program has been successfully providing support to industry for more than 30 years. In 2018, the MEP Advisory Group created a Performance and Research Development Working Group. The working group focused on providing input and guidance to the MEP program to examine performance measurement, management, and evaluation processes that support the MEP National Network. The objective of the activity was to provide improved center evaluation processes and approaches, to promote National Network learning, and to suggest additional data and analysis capabilities for the MEP centers. Some of the findings and observations of the group are of interest. First and foremost, the goal of the MEP evaluation process is to provide accountability to all of the program stakeholders and to demonstrate that the MEPs are making a meaningful difference to their clients and that they also have a broader economic impact. The working group noted that the MEP has had a consistent focus on measuring performance and impact, which has driven improvements in the program and in the centers. A key component of the evaluation is the NIST MEP Client Survey, which captures relevant information such as sales, cost savings, hiring, etc., from industry. Also, an IMPACT metrics report is produced that assesses impact and market penetration. There is a panel review process performed that focuses on trends in center performance. This information allows center-to-center comparisons to be made and best practices to be identified.

There have been many studies over the 30-year life of the MEP program that have examined the effectiveness of these PPPs, improvements to the MEP National Network, and improvements to the MEP evaluation processes. Although the MEP program has been in existence for a long time, the working group suggested areas for research to further refine and improve the assessment processes used for the MEPs in the future.

Guidance to DoD MII Evaluation Process

Even though the DoD MIIs have significantly different goals than the Agency programs described above, the DoD 5-year evaluation of the MIIs can benefit from insights and lessons learned provided by these guides and earlier studies. First, the selection of the review team lead is obviously critical. This individual is responsible for all of the planning, execution, and reporting steps in the evaluation. There will

___________________

⁷ MEP Advisory Board Performance & Research Development Working Group, 2019, Performance FrameworkFinalReport, https://www.nist.gov/system/files/documents/2019/05/08/report_wgperformanceframework_final.pdf.

be a substantial time commitment for this individual for the period of the evaluation process. Regarding the review team, the JDMC review team could benefit from members who have knowledge of the technical field, knowledge of the EWD needs in the field, knowledge of DoD’s needs in the technical field, knowledge of the ecosystem in the field, and knowledge of DoD’s management in similar major technology development programs. The Joint Defense Manufacturing Technology Panel (JDMTP) is an important source of manufacturing technology expertise in DoD and can support the MII reviews.

Although across the set of DoD MIIs there are many organizational similarities, the individual MIIs have some unique characteristics. Also, the MIIs are at different levels of maturity (institute age). There will be a need to develop a tailored evaluation process for individual MIIs. Collaborative planning activities and the development of a TOR-like agreement among the JDMC team lead, OSD ManTech, and MII leadership will enable the development of a transparent evaluation process that is balanced between DoD’s needs for a strategic assessment and the time and cost resource requirements placed on the MII. In terms of information and data provided prior to the site visit, these materials would include OSD ManTech’s perspectives on specific attributes of the MII being evaluated and the MII-prepared responses to the relevant metrics-driven questions developed by OSD ManTech. As with DoD’s review of UARCs and FFRDCs, the results of surveys of customer satisfaction needs to be included in such material. Since the MIIs are PPPs, the customers include both department organizations that have sponsored work at the MII as well as industry members and other stakeholders from the ecosystem in the technology area. Finally, the assessment of MII performance would include consideration of responses provided to evaluation criteria questions in the strategic areas of importance defined for the DoD MII review process as the four evaluation questions. This topic is discussed further in the section “Strategic Evaluation Criteria for DoD MIIs” below.

At the site visit, the team would interact with MII leadership and institute members and receive detailed briefings on technology developments and transitions, EWD programs, ecosystem development, and the MII business plan’s status and direction. The team will meet to develop the set of findings and recommendations from the evaluation. The team lead can then provide a recommendation to OSD ManTech and OUSD(R&E) on the future of the MII. The reports developed by the review team will also be critical for the department’s tracking of trends in performance of individual MIIs over time.

Finally, it should be noted that as the JDMC MII evaluation process starts in FY 2021, it is expected that an effort will be made to capture lessons learned from the process so that improvements can be incorporated as the 5-year evaluations progress over time. It can also be expected that refinements to team structure, data

and information sets, timelines for their delivery prior to the site visit, and suitable reporting requirements will be made as the process matures. A formal meta-evaluation of the first two MII reviews planned for 2021 would be useful to the JDMC. This would be a collaborative effort among OSD ManTech, the JDMC, and MII leadership, which would lead to an effective and efficient evaluation process. In addition, the repository of the 5 year evaluation results for all of the MIIs will enable the identification of lessons learned and best practices from individual MIIs which can be incorporated into future reviews and which can support the general improvements within the set of MIIs.

STRATEGIC EVALUATION CRITERIA FOR DOD MIIS

The JDMC evaluation of the DoD Manufacturing USA institutes provides an opportunity to better insure and document the value of the MIIs for DoD. The nature of the JDMC evaluation is different from the annual review of MII performance, in that the JDMC has an opportunity to assess the broader impact and value of the institutes in meeting DoD’s mission needs in advanced manufacturing technologies. The metrics previously developed by DoD are very valuable to the JDMC evaluation process. Although the four question evaluation framework is appropriate to accomplish the strategic evaluation of the MIIs, there are a number of

topics which should be considered which would expand the scope of the questions to better reflect an evaluation process designed for the MIIs.

There are three chartering principles, as seen in Figure 3.1, which are common to all DoD MIIs:

1. Perform advanced manufacturing research and scale-up,
2. Develop and sustain a resilient domestic ecosystem, and
3. Develop a sufficient, well-trained workforce.

The combination of these principles underlines the uniqueness of MIIs which necessitates the development of a similarly unique set of evaluation criteria. Assessments within these areas serve as the starting point for the evaluation. There are a number of considerations that will require the JDMC to tailor the MII evaluation based on the specific institute being reviewed. DoD has started nine MIIs in a wide variety of technical fields from 2012 to 2020. Within the set of MIIs, there are institutes that have a focus on a facility (or facilities) with advanced technology equipment and resident expertise (e.g., the Manufacturing Innovation Institute NextFlex), institutes that serve as convening bodies for expertise and capabilities that are distributed among the members (e.g., the Manufacturing Innovation Institute AM), and institutes that have combinations of these characteristics. Over time, the institutes will have a natural evolution from their initial, start-up phase to a more mature, execution phase. The technology readiness level (TRL) and manufacturing readiness level (MRL) of technologies will vary with the portfolio mix of breakthroughs, developments, applications, and new concepts from the pipeline of ideas coming to an institute. The number of transitions to DoD applications will also vary over time based on opportunities, viable solution candidates, and application funding. Demonstrations of expansion of the DoD supply chain or enhancements to supply-chain resilience would increase over time as well. The EWD needs of DoD will evolve as technologies mature and are transitioned to DoD systems. As noted in the section “Evaluation Process for DoD MIIs” above, in anticipation of the MII evaluation, the JDMC team lead will need to work with OSD ManTech and MII leadership to develop a department assessment of the appropriate characteristics that then can be evaluated and the relevant metrics and discussion topics that also can be applied to the specific MII. As has been done by other Agencies, the DoD should have a broad set of evaluation topics within the framework which can be used to guide the design of an evaluation process for a particular MII. The evaluation questions, along with suggested additional critical topics for consideration for the JDMC evaluations, are shown below.

Question 1: Is There a Continuing Need for the MII?

This question is the DoD’s assessment of the importance of the MII being evaluated in the context of DoD’s overall needs in advanced manufacturing technologies. MIIs have unique combinations of organizational attributes that provide advanced technologies, an ecosystem of importance to DoD and the nation, and a knowledgeable workforce in a specific manufacturing technology area. The programmatic focus of the MIIs is in advanced technology development and demonstration, with a concentration on programs in TRLs 4 to 7. MIIs typically engage in pre-competitive technology development projects and also perform technology demonstration projects. Other organizations within DoD focus on development of technologies up to TRL 4 and also on transition to implementation in TRL 7-9. The question of continuing need for the MII depends on both DoD’s long-term plans for the MII technology area and how the MII is integrated into DoD’s strategy for advanced technology implementation. As has been noted before, the MII assessment acknowledges that within the set of MIIs, there are technologies that are at different maturities and at different levels of penetration into DoD programs and systems. Furthermore, MIIs are at different points within the start-up and follow-on stages when the 5-year reviews are performed. Therefore, a long-term view of the continuing need is required.

In these strategic reviews, DoD assesses the MII in the context of emerging needs of DoD agencies and military services. The JDMC review process will need input from these organizations and DoD stakeholders. Relevant topics to examine would include the following:

1. Does DoD leadership view the MII technical field as critical to the future of DoD? For example, does this MII cover any of the needs identified in the Industrial Capabilities Report (A&S) to Congress?
2. Is there evidence that science and technology work in OSD and the agencies and services supports the need for technology maturation in an MII?
3. For mature MIIs, is there evidence that the DoD Research, Development, Test and Evaluation (RDT&E), acquisition, and sustainment communities are engaged in the development, transition, and implementation of MII technologies?
4. Does DoD see a need for an ecosystem to drive advanced manufacturing technology implementation activities (e.g., road mapping, community convening, standards and regulations development, etc.)?
5. Does DoD see a need for sustainment and growth of a workforce to support this technology area for DoD and the nation?
6. Does DoD view the MII as essential so that DoD and the nation will benefit from the expansion of the ecosystem in this technology area (e.g., resilient

6. supply chain, versatile production capability, globally competitive position in the technology area, etc.)?
7. What is the DoD’s sunsetting strategy for the MII?

Question 2: Is an MII the Best Alternative?

With this question, DoD will assess whether the organizational characteristics of the MII still afford the best approach to achieving DoD’s goals in a specific technology field. The unique organizational characteristic of the MIIs is that they are industry-led, public–private partnerships, unlike other sources available to DoD. MIIs were chartered⁸ and organized for establishing and growing a manufacturing ecosystem, advancing research and technology, and securing human capital. These outcomes support OSD ManTech’s congressionally mandated mission to support the warfighter while also enhancing U.S. manufacturing base capabilities, expertise, and intellectual property. In assessing whether continuation of the MII is the best way to achieve these outcomes, the JDMC is expected to assess the strengths and weaknesses of available alternatives in the MII field of discipline. Relevant topics to examine would include the following:

1. Are alternative sources available in the MII technology area with capabilities for delivering the needed outcomes in technology maturation, EWD, and manufacturing ecosystem? Alternatives might include existing R&D consortia, contractors, FFRDCs, UARCS, national laboratories, other agency programs, etc.
2. What are the strengths and weaknesses of alternative sources in comparison to those of the MII public–private partnership model that could be used to validate that the MII is still the best alternative?
3. What evidence exists of continued “private” commitment to the public–private partnership to ensure MII viability for renewing the 5-year agreement?

The committee note that DoD has well established best practices for analysis of alternatives in the acquisition of defense systems.⁹ A much more streamlined process is called for in the 5-year review of an MII, focused on the decision as to whether to continue an agreement that was considered the best alternative at the time of original award. The primary emphasis should be on updating the identification and qualitative assessment of alternative sources to include both those originally considered and new alternatives that may now be available to DoD, Stan-

___________________

⁸ As outlined in the MII Strategy and Assessment document MII Strategy, received on November 9, 2020, from ManTech.

⁹ DoD Instruction 5000.84, “Analysis of Alternatives,” August 4, 2020.

dard qualitative techniques such as strengths, weaknesses, opportunities, threats (SWOT) analysis may be appropriate. If the qualitative evaluation determines a viable competing source may be equally effective in meeting DoD needs, a more detailed quantitative analysis of costs and benefits is required.

Question 3: Has the MII Performed Well?

This question assesses the performance of the MII against DoD’s goals for the institutes. Consistent with DoD’s chartering principles for the MIIs, there are three components of the assessment—technology development, ecosystem development, and EWD. Since the formation of the institutes, DoD has had an active effort to develop and improve the metrics it uses to perform annual assessments of the MIIs. The metrics provided in the 2020 DoD Metrics List and Final Metrics List¹⁰ are of value to the 5-year assessment process in that they provide key inputs on trends in the MII. Being a 5-year strategic evaluation, the evaluation needs to examine trajectories toward a desired steady state of the MII. In some cases, this will entail trend analysis on the data collected by DoD and the institute. In other cases, additional topics that relate to DoD mission needs and national stature, as well as future plans and directions of the institutes, need to be considered.

Advanced Manufacturing Ecosystem

The defense manufacturing supply chain is critical to both the U.S. economy and national security.^11,12 Support is needed for emerging technologies such as directed-energy weapons, hypersonics quantum and cybersecurity, all of which are vital to national defense.^13,14 A strong manufacturing sector not only ensures a ready supply of defense and commercial goods and services, but also ensures the integrity and safety of these goods, such as electronics and control systems.

Advanced manufacturing encompasses all aspects of manufacturing, including the ability to quickly respond to customer needs through innovations in product

___________________

¹⁰ As provided by ManTech through the Excel books: “Final DoD MII Performance Metrics_ cleared” and “2020 MII Complete Metrics List_cleared,” both received from ManTech on December 11, 2020.

¹¹ Executive Office of the President, 2017, National Security Strategy of the United States of America, Washington, DC, p. 55.

¹² Fiscal Year 2020 Industrial Capabilities Report to Congress, OSD A&S Industrial Policy, January 2021. https://media.defense.gov/2021/Jan/14/2002565311/-1/-1/0/FY20-INDUSTRIALCAPABILITIES-REPORT.PDF.

¹³ Ibid.

¹⁴ See https://www.nist.gov/mep/manufacturing-infographics/defense-manufacturing-supplychain.

design and production processes and innovations in the supply chain. As manufacturing advances, it is becoming increasingly complex and knowledge-intensive, relying on diverse data streams and partner networks to accelerate the pace of new products and services delivered.

Advanced manufacturing ecosystems can provide speed in technology development while enabling partner networks to co-innovate around common challenges.¹⁵ This ecosystem approach requires deliberate coordination among and between various parties to solve shared challenges and meet shared objectives. In the case of smart manufacturing initiatives, the Deloitte and MAPI Smart Manufacturing Ecosystem Study identified four primary ecosystems that support advanced manufacturing ecosystems, see Table 3.1.

Given the pace of change, maintaining vibrant and resilient manufacturing ecosystems is of paramount importance to U.S. national security and the DoD. Hence, establishing and growing manufacturing ecosystems is one of the three chartering principles of Manufacturing USA MIIs.

Since DoD MIIs vary in the nature, maturity, and intended use of their focused technology, each MII and its stakeholders need to identify the shared challenges and shared opportunities that their ecosystem wishes to develop, grow, and maintain.

As public–private partnerships, MIIs provide DoD with an opportunity to broaden and strengthen DoD’s supply chains for advanced technologies through the ecosystem. The character of the ecosystem that best suits the needs of DoD

TABLE 3.1 Four Primary Advanced Manufacturing Ecosystems and the Focus Areas of Shared Objectives

___________________

¹⁵ Deloitte, 2020, “Accelerating Smart Manufacturing: The Value of an Ecosystem Approach,” October, https://www2.deloitte.com/us/en/insights/industry/manufacturing/accelerating-smart-manufacturing.html; Deloitte, “Manufacturing Goes Digital: Smart Factories Have the Potential to Spark Labor Productivity 2019,” Deloitte and MAPI Smart Factory Study, https://www2.deloitte.com/us/en/insights/industry/manufacturing/driving-value-smart-factory-technologies.html.

will be dependent on the specific technology field of the MII. Each MII therefore needs to describe the advanced manufacturing ecosystem that it envisions building, based on the vision of the needs of the DoD and what will ultimately be acquired by DoD acquisition and sustainment from the manufacturing community. The MII also needs to explain its role in growing and improving the ecosystem and state why no other entities can play that role better than it can. The health of the ecosystem of the MIIs is therefore an important aspect of the value of the institutes to DoD. Over time, it is natural for members to join and depart from the MII ecosystem based on its own interests. However, it is important for DoD to monitor support for the MIIs with industry representatives in those relevant fields. Strategically, DoD needs to continue to assess the health and growth of the ecosystem over time.

Possible assessment activities for health and growth of the ecosystem are as follows:

1. Assess the MII’s plan and progress in development and growth of an ecosystem for its technology area.
2. Assess what programs are in place to enable development of a robust: 1) supply chain ecosystem, 2) production ecosystem, 3) customer ecosystem, and 4) talent ecosystem
3. Assess the distribution and number of partners from: a) industry, b) economic development, c) workforce development, d) education, e) government, f) professional organizations, and g) philanthropy.
4. Assess evidence of MII impact on the robustness and resilience of DoD supply chains (e.g., multiple sources for technology, technology companies linked to system providers or government supply chain, sufficiently trained workers, foundation provided in emerging technology by institutes).
5. Assess if effective marketing and communications systems are in place to attract new members, as well as support and retain existing members.
6. Examine whether Tier 1 industry members and other DoD original equipment manufacturers (OEMs) use the MII in platform system-specific manufacturing technology development activities.
7. Assess how the MII has helped summarize and promote sharing of common manufacturing processes in the form of sharable SOPs and helped SMEs adopt these best practices.
8. Assess how the MII has supported standards development.
9. Assess the level of state and local entities participating in economic development, EWD, etc.

10. Assess the health of the MII ecosystem.
    1. Have new industry collaborations emerged from MII activities?
    2. What number of jobs are directly attributed to efforts of the MII?
    3. Are there new start-up companies which have been enabled by the MII ecosystem?
    4. For MIIs that have a facility base, are there examples of new start-up companies incubating at the MIIs?
    5. For MIIs that have a facility base, what is the blend of small, medium, large companies that have used the facilities (e.g., are the user organizations technology creators or adaptors?)?
11. Assess evidence (e.g., market survey, etc.) of MII impact on global competitiveness.

Technology Development

One important reason for DoD’s support of MIIs is that they offer effective delivery of manufacturing technologies that support DoD needs. The OSD ManTech metrics list contains the topics that document the contribution of the MIIs to technology development and transition. An additional topic relevant to MII performance is the efficiency of technology transition. Determining the rate-limiting steps in developing an advanced technology is critical to addressing acceleration and maximizing impact. The MIIs are responsible for finding innovative approaches to address technology development challenges. During an MII evaluation process, a key question that can be addressed is, What is limiting the rate of maturation of the technology or a resilient ecosystem for this advanced manufacturing field? Are there technology gaps, a lack of standards, inadequate equipment and facilities, a lack of qualified suppliers, insufficient demand, untrained workforce, etc.? Once identified, the evaluation can focus on the MII strategy for addressing these rate-limiting issues and address if adequate resources are appropriately prioritized and aligned to efficiently and effectively close these gaps. The 5-year evaluation would ideally assess the trends in the effectiveness of the MIIs technology development and implementation activities. Also, as the MIIs mature, it is to be expected that the degree of engagement with DoD’s acquisition and sustainment communities and opportunities for technology insertion onto DoD systems and platforms will increase.

Possible assessment activities include the following:

1. Assess evidence of technology maturation, technology scale-up, technology transfer, technology implementation, and trends in the data.
2. Assess evidence of technology development acceleration (the transitioning efficiency).

3. Assess evidence that DoD RDT&E, acquisition, and sustainment communities are actively engaged in the technology (and EWD) road-mapping activities.
4. Assess evidence of MII engagement in addressing standards development and regulatory issues in support of DoD and national needs.
5. Assess the amount of engagement with the DoD RDT&E, acquisition, and sustainment communities, which supports acceptance of the MII as a preferred technology provider.
6. Assess how the MII has helped summarize and promote sharing of common manufacturing processes in the form of sharable SOPs and helped SMEs adopt these best practices.

Education and Workforce Development

EWD has been a core mission of the MIIs from the outset, based on an understanding that new production technologies will not be adopted unless there is a workforce ready to apply them. Currently, MIIs collect institute-specific data on the numbers of those trained and reached in their EWD programs. While numerical data provides a useful metric for MII performance, the committee has done a detailed review of the kinds of EWD programs undertaken by MIIs and developed a list of “best practices” that various MIIs are undertaking. While these best practices are less amenable to metrics, evidence that best practices are being applied represents an important indicator of program quality. It is also indicative of the reach, both potential and ongoing, of the MII’s EWD programs in improving workforce education skills in the MII’s technology area. While no single institute is undertaking all of the program elements delineated below, the ability to perform a number, with quality efforts, can be an important indicator of program strength. Questions seeking evidence from the MII of application of best practices appear below, with a more detailed explanation about the need for and significance of each question set out in Appendix C.

Possible questions for information gathering on EWD issues are set out below, with indicators likely to be developed earlier in the MII’s term listed before later indicators:

1. Is the MII adequately staffed with personnel experienced in EWD practices?
2. Has the MII dedicated adequate funding resources to support significant EWD programs, from both its core and outside funding?
3. Has the MII developed and circulated, with industry and education institution involvement, a detailed set of knowledge, skill, and assessment (KSA) elements and corresponding competencies in its technology area?

4. Is the MII engaged in both mapping skill demand and in developing skill roadmaps, both regionally and nationally, and using these to inform its programs?
5. Has the MII evaluated available credential offerings in the MII’s technology area then worked with industry to fill in credential gaps to either develop or apply industry-recognized credentials?
6. Are the MII’s education materials being developed with and used by its education and industry ecosystem?
7. Has the MII formed significant programs for regional engagements around EWD needs?
8. Has the MII developed online education materials in its technology area available to industry and education institutions?
9. Has the MII engaged effectively with DoD depots, arsenals, shipyards and/or defense contractors in its EWD programs or through efforts that reach veterans?
10. In collecting data on numbers that the MII educated and trained, is there information on whether those educated entered positions or performed work based on the training?
11. Is the MII engaging with DoD acquisition and sustainment to identify and support internal DoD EWD needs and opportunities? How many programs have been executed? Have the programs successfully met the DoD’s needs? Is the MII engaging with DoD acquisition and sustainment to identify internal EWD needs?

Current and former MII stakeholders should be consulted in the development of this information. Supplemental material relevant to and discussing these questions appears in Appendix C.

Question 4: Is the Governance and Management Effective?

Similar to Question 3, the performance of the MII in establishing and operating the institute has been reviewed on an annual basis by DoD. There is a section in the 2020 DoD Metrics list directed at operations performance. The data collected from the annual reviews, along with the trends in the data, will be very useful to the JDMC evaluation committee performing the 5-year assessment. In addition, the JDMC review can address the MIIs progress toward achieving DoD’s strategic goals for the MIIs, as well as DoD’s overall mission needs for manufacturing technology. The examination of the adequacy of MII leadership in addition to business plans and processes in supporting the achievement of these mission needs could be relevant to the effectiveness of the governance and management of the MIIs.

Possible assessments for governance and management are as follows:

1. Provide evidence of leadership and organizational effectiveness in advancing and transitioning technology, EWD, and manufacturing ecosystem enhancement and use.
2. Assess performance relative to MII Business Plan; examine how the MII business plan has evolved as the MII matures.
3. Assess the MII’s strategy for engagement with DoD RDT&E, acquisition, and sustainment communities.
4. Assess the percentage of budget (revenue and costs) in technology programs, EWD, outreach, operations.
5. Assess funding levels and trends from Intellectual Property (IP).
6. Assess funding levels and trends from DoD customers.
7. Assess funding levels and trends from commercial customers.
8. Assess funding levels and trends from non-DoD federal agencies.
9. Assess level, quality, and trends of funding or cost share from non-federal sources (states, foundations, etc.).
10. Assess user facility utilization rate and nature of use and of the user group (e.g., govt., OEM, SME, etc.).
11. Assess plans for DoD cyber compliance.
12. Assess partner retention.
13. Assess funding level for technology transition and implementation at member facilities.
14. Assess the level of utilization of the MII by government laboratories, universities, and other industry groups.

The goals and expected outcomes of the MII key objectives (advancing research and technology, establishing and growing a manufacturing ecosystem, and securing human capital) in the four critical areas of the evaluation framework are summarized in Table 3.2.

STRATEGIC QUANTITATIVE METRICS

All of the topics in the Strategic Evaluation Criteria for DoD MIIs section above represent qualitative metrics which are directly relevant to the three key strategic objectives for the MIIs (advancing research and technology, establishing and growing a manufacturing ecosystem, and securing human capital). In addition, the trend analysis mentioned, which will be performed based on data collected by OSD ManTech and the Institutes on the MIIs over time will serve as quantitative assessments of progress against these key objectives. These quantitative measures are described below.

TABLE 3.2 Summary of Goals and Expected Outcomes for Evaluation Criteria

Advancing Research and Technology

As has been noted, OSD ManTech and the MIIs already measure and collect data on a number of metrics important to technology development and operations. This includes tracking the number of projects that are being executed, progress to meeting technical objectives in timely manner, and how many projects are started or get completed over a given time period. The institutes also categorize these metrics in total, by non-DoD government entity, by DoD service, by state or local government, and by commercial entity. In addition to the metrics currently gathered, the committee suggests additional, strategic assessment should include: MIIs must develop a process to track demonstrated outcome impact of completed projects on cost, lead time, and/or platform performance enhancements, and MIIs should track their impact on the roadmaps for the various technologies they tasked with and how these impacts fit within the larger community contributions. The additional metrics suggested for the research and technology objective are:

1. Describe the process used to track demonstrated outcome impact of completed projects on cost, lead time, and/or platform performance enhancements.
2. Assess impact on the roadmaps for the relevant MII manufacturing tech

1. nologies and how these MII contributions interface with the larger community contributions.

Establishing and Growing a Manufacturing Ecosystem

OSD ManTech and the institutes measure and collect data on a number of quantitative metrics relevant to their manufacturing ecosystem. This includes tracking the number of successful technology transitions, number of patents filed and used, number of members on funded projects, level of member cash to in-kind contributions, level of ecosystem participation, number of DoD subject matter experts engaged, number of states involved and level of state contributions. For the long-term strategic evaluation, the committee suggests that the institutes define the advanced manufacturing ecosystem which the MII and its stakeholders envision to serve the needs of its community. The institutes should identify the ecosystem programs and partnerships, which are required to establish robust and resilient DoD supply chains and track and report trendlines in 1) the number and growth of successful tech transitions into relevant DoD industrial bases, 2) growth in number or percentage of IP portfolio which contributed to the competitiveness of its ecosystem, 3) number of MII programs focused on establishing robust and resilient DoD supply chains, and 4) growth in number and value of regional/state/local economic development programs and partnerships focused on nurturing, growing, and maintaining a vibrant and resilient ecosystem. The institutes should address how the ecosystem metrics have progressed or changed over time to put their impact in context on technical development, talent development, production capability, standards development, regulatory requirements, customer engagement and overall operations. The additional metrics suggested for the Ecosystem objective are:

1. Identify the ecosystem programs and partnerships, which are required to establish robust and resilient DoD supply chains and track and report trends in 1) the number and growth of successful tech transitions into relevant DoD industrial bases, 2) growth in number or percentage of IP portfolio which contributed to the competitiveness of its ecosystem, 3) number of MII programs focused on establishing robust and resilient DoD supply chains, and 4) growth in number and value of regional / state / local economic development programs and partnerships focused on nurturing, growing, and maintaining a vibrant and resilient ecosystem.
2. Address how the ecosystem metrics have progressed or changed over time to put their impact in context on technical development, talent development, production capability, standards development, regulatory requirements, customer engagement and overall operations.

Securing Human Capital

The Institutes are currently required to collect quantitative data on such items as numbers of students participating in institute projects or institute internship or training programs, numbers completing institute-related certificates or apprenticeships, teachers or trainers completing institute training programs, number of EWD projects and their funding showing longer-term and shorter-term projects, funding to the institutes from outside sources for EWD, and the level and depth of learning experiences for the different EWD participants identified. Aside from extensive qualitative metrics, the committee suggests additional metrics regarding whether those trained completed programs and entered positions or performed work based on the training. The MII should be asked to develop trendlines of this data for the MII’s period of operations, applied against workforce education goals set out in the MII’s skill roadmaps, to indicate progress in meeting strategic, overall skills training demands in the MII’s manufacturing technology sector. The additional metric suggested for the securing human capital (EWD) objective is:

1. Can the Institute provide evidence of trends of EWD metrics data for the MII’s period of operations, applied against workforce education goals set out in the MII’s skill roadmaps, to indicate progress in meeting strategic, overall skills training demands in the MII’s manufacturing technology sector?

MODIFIED EVALUATION PROTOCOL

When these additional assessment topics are taken into account, it becomes clear that additional sources of input for the MII 5-year evaluation will be needed beyond those shown in Figure 2.2. The committee believes that the JDMC 5-year review would be significantly improved if inputs from the sources shown on the left in Figure 3.2 are obtained. Any data collection efforts will need to be designed in advance to adequately capture and evaluate quantitative data. The additions to the JDMC evaluation process are the following:

1. An initial terms of reference-like agreement that specifies the nature, scope, schedule, and ground rules for the evaluation process, which is tailored to reflect the specific MII being reviewed. Tailoring would take into account the unique characteristics of individual MIIs, such as a foundry-style MII or a networking-style MII, an emerging/revolutionary technology MII or an evolutionary technology MII.
2. The results of the survey of the non-DoD stakeholders that will provide an assessment of the health of the MII ecosystem. Examples of non-DoD

2. stakeholders could include NSF and the Small Business Innovation Research (SBIR) communities feeding technology ideas to the MII, the stakeholders who are requesting/paying for results from the MII, the recipients of EWD training, and those hiring of EWD-trained employees.
3. The results of the JDMC review team’s assessment of the MII’s performance relative to the key evaluation questions.
4. The JDMC review team’s assessment of the information provided at the site visit.

4

Findings and Recommendations

Based on the information gathered from all sources, the committee has developed the following list of findings regarding the DoD MMIs. In this interim report, the committee confines its comments to the first task of the study’s statement of task: evaluation criteria and metrics (the statement of task is reprinted in Appendix A).

FINDINGS

Finding 1: As noted in Fiscal Year 2020 Industrial Capabilities Report to Congress,¹ support for the DoD MIIs remains high among senior leadership at DoD, but the declining budget request for core funding will require difficult decisions. At the 5-year review point for each institute, DoD will need to decide whether to continue core funding support, possibly at a reduced level or at a higher level, or decline to renew the agreement in order to apply available budget to higher priorities.

Finding 2: DoD has a sound framework for conducting 5-year assessments of each institute to support such decisions. The four issue areas in the Joint Defense Manufacturing Council (JDMC) evaluation plan² are based on an assessment process which is familiar to DoD and will facilitate the assessment of the MIIs within the context of DoD’s mission needs. The planned evaluation process is similar to good practices the committee has seen in other agencies. The JDMC, based on its charter, is well suited to conduct the 5-year assessments. Over time, the committee believes the 5-year reviews can improve the coupling between public and private interests in each MII and move all MIIs toward collective implementation of best practices. Equally important, the JDMC can play a key role in the integration of the MIIs into DoD’s technology delivery pipeline, which will be further explored in this committee’s work as described by the third task in the statement of task for this study, and presented in the final report.

___________________

¹ Department of Defense, 2021, Fiscal Year 2020 Industrial Capabilities Report to Congress, https://media.defense.gov/2021/Jan/14/2002565311/-1/-1/0/FY20-INDUSTRIAL-CAPABILITIES-REPORT.PDF.

² The MII Strategy and Assessment_ManTech_Oct2020_cleared PowerPoint, received on November 9, 2020, from ManTech.

Finding 3: The metrics needed for strategic 5-year assessments are not limited to the metrics used for ongoing management and oversight of the MIIs. DoD has a robust set of metrics for quarterly reviews that are important to assessing progress toward annual performance objectives. The 5-year reviews are more strategic in nature. Additional topics that are appropriate in the 5-year assessment would include the continued need, appropriateness of the public–private partnership, and long-term performance trends of the MII. Strategic metrics would also be appropriate for the 5-year reviews.

Finding 4: The unique public–private partnership aspect of the MIIs requires strategic consideration of the health of the “private” component of the institutes. To the extent that DoD views these cost-shared private–public partnerships as essential, an additional consideration would be input from MII members and non-members from the ecosystem.

Finding 5: Greater involvement of the DoD acquisition and sustainment communities are needed for DoD to fully benefit from the technology development, EWD, and manufacturing ecosystem of DoD MIIs. The JDMC has an opportunity to use the 5-year assessments and the growth of this engagement over time as a key measure for the impact of the MIIs on DoD advanced technology development and implementation.

RECOMMENDATIONS

Recommendation: The Department of Defense should phase in formal 5-year assessments of the Manufacturing Innovation Institutes (MIIs) to support decisions on renewal of agreements and to provide budget justification for future years commensurate with the value of the MII program.

Recommendation: OSD ManTech and the Joint Defense Manufacturing Council should expand the evaluation framework within its four topic areas to consider additional strategic considerations focused on the Manufacturing Innovation Institute (MII) key objectives of advancing research, education and workforce development, and ecosystem. Design of the evaluation team and details of the evaluation tasks should consider the best practices of other agencies discussed in Chapter 3, including provisions for access to outside technical experts as needed. The process should be tailored to the unique characteristics of the MII public–private partnerships and should be made available to the MIIs and the rest of the manufacturing community.

Recommendation: OSD ManTech and the Joint Defense Manufacturing Council should amend the planned evaluation process to make provisions for additional strategic metrics and trend analysis, and for additional sources of input beyond those currently planned, such as the following: other stakeholder survey results, assessment of the evaluation criteria, site visit results, and terms of reference planning agreements. In assessing effectiveness of governance and management, specific attention should be given to the effectiveness of government and industry leadership in achieving desired outcomes.

Recommendation: The Joint Defense Manufacturing Council 5-year assessment should address strengths, weaknesses, and opportunities for improvement of the Manufacturing Innovation Institutes (MIIs) relative to non-Department of Defense (DoD) components of the public–private partnership to inform decisions on whether a public–private partnership is the best way to meet DoD needs. The Office of the Secretary of Defense Manufacturing Technology Program (OSD ManTech) should also perform a stakeholder survey as a component of the 5-year MII assessment. The survey should include all stakeholders such as large, medium, and small businesses, academia, state governments, and foundations that are within the MII ecosystem.

Recommendation: As the Department of Defense starts the 5-year Manufacturing Innovation Institute evaluation process in 2021, it is recommended that a process be established to catalog lessons learned and assess the effectiveness of the Joint Defense Manufacturing Council evaluation process to facilitate continuous improvements. A meta-evaluation can be accomplished by appropriate groups in professional societies or within suitable not-for-profit organizations.