2

Protocols for Long-Term Engagement Assessments for the DoD MIIs

The main body of work by the committee in this area has been documented in the interim report submitted by the National Academies of Sciences, Engineering, and Medicine to the sponsor in April 2021.¹ The main text of the interim report is reprinted in Appendix C. As the study progressed beyond the interim report, the committee primarily focused on the remaining two tasks and continued to consider additional topics related to Manufacturing Innovation Institute (MII) assessments that were complementary to those covered in the interim report. In this final report, the committee provides additional clarification on the following topics discussed in the interim report: specific roles and responsibilities for activities recommended by the committee, additional assessment topics on the commercial role of the MIIs, appropriate topics for stakeholder surveys, and ecosystems and assessment of their development in the MIIs. The committee thought these areas deserved consideration, and these topics are discussed in the following two sections. It should be noted that additional findings and recommendations for Task 1 have been added at the end of this chapter.

___________________

¹ National Academies of Sciences, Engineering, and Medicine, 2021, DoD Engagement with Its Manufacturing Innovation Institutes: Phase 2 Study Interim Report, Washington, DC: The National Academies Press, https://doi.org/10.17226/26149.

STAKEHOLDER SURVEYS

One important recommendation from the interim report focused on the need for partner and customer feedback to help validate, assess, and improve MII performance relative to the dual Department of Defense (DoD) and national economic goals for the institutes. The health of the private part of the public-private partnership (PPP) models of the MIIs was discussed as being essential to the success of the organizations in accomplishing DoD’s and the nation’s vision for the MIIs. As discussed in the interim report, the Joint Defense Manufacturing Council’s (JDMC’s) performance of an independent evaluation of the health of the non-DoD components of the PPP during the information gathering, pre-site visit phase of the review process would provide critical input to DoD’s ability to assess each institute’s contributions to the department and to national economic goals of these organizations. As the 5-year assessments of the institutes within the set of MIIs are accomplished, the relative performance of individual DoD MIIs in meeting the unique dual mission requirements can be evaluated by the department

Depending on the particular MII, the non-DoD components and partners can include industry—large, medium and small, state organizations, and foundations. The Office of the Secretary of Defense (OSD) Manufacturing Technology (ManTech) program has established metrics to assess the number of industry partners on an annual basis. While useful, metrics or numbers are not sufficient for the true measurement of the health of these other ecosystem members. DoD and the MIIs need insight into the aspects of the specific MII organizations and structure that are important to their partners; aspects of the organizations that work well for the partners; and equally importantly, aspects of the MII that need to be improved. It is incumbent on the JDMC to perform an independent appraisal of the MII’s performance relative to the health of its ecosystem and to use that appraisal in the overall assessment of the MII’s performance and potential for the future. The committee suggests that JDMC perform a stakeholder survey to gain this insight. As was stated numerous times in the interim report, the assessment protocol applied to a specific MII undergoing a review needs to be tailored for the technology field, for the maturity of that technology, and for the age of the MII. DoD’s assessment of MII performance will be based on the department’s strategic plan for the advancement of the specific field of technology. For example, some MIIs focus on technologies in which DoD is a traditional driving force for technology development, such as structural materials, while other MIIs are working in fields in which DoD’s impact in technology development will be secondary, such as electronics or biotechnology. The evaluation processes, the assessment of ecosystem development, and the development of education and workforce development (EWD) capabilities within the set of MIIs should reflect this breadth of MII technology fields. The evaluation of the information obtained in the stakeholder survey should be viewed similarly.

The types of partners and their relative numbers in the ecosystem will be assessed based on the perspective of DoD’s plan for the development of the MII ecosystem in the particular field. In addition, all the DoD MIIs are expected to benefit DoD—through the development and transition of technology products for department systems and platforms and by supporting the growth of commercial industry in the specific technology field. The stakeholder survey will provide insight into the national impact of the MII within the manufacturing community. A suggested set of questions and discussion topics, parsed by organization type, is provided in Box 2.1. The JDMC team can select from these questions a set that best suits the specific MII being evaluated, as well as others relevant to the specific MII, and produce a tailored stakeholder survey.

DoD organizations that have participated with the MII (e.g., in road mapping activities) or have directly sponsored projects are another source of important insights for the JDMC evaluation process. The DoD MIIs’ impact within the department could increase through the knowledge and engagement of non-OSD ManTech organizations with the MIIs. Documenting the performance of the MII relative to the needs of department organizations through a DoD customer survey (Box 2.2) can help to demonstrate the value of the MII to the broader DoD community as well as to identify issues or impediments to engagement with DoD. The 5-year evaluation process for the DoD MIIs is modeled on the review process used by DoD to assess federally funded research and development centers (FFRDCs) and University Affiliated Research Centers (UARCs). A central component of DoD reviews of those organizations is a customer survey that measures DoD sponsoring organization satisfaction with the work performed for them by the FFRDC or UARC being evaluated. Such a survey would be useful in the JDMC evaluations for the MIIs. An example of the types of information solicited by the DoD organization responsible for the UARC in these customer surveys is relevant for the MII review process.² The survey would address the quality, efficiency, and cost of the work performed by the UARC for the DoD sponsor. Finally, DoD sponsors could be queried on the importance and value of the MIIs for the development of robust supply chains as well as on the development and delivery of training to support their organization’s needs.

All of the information generated in support of the 5-year JDMC MII evaluations would be curated by OSD ManTech. The aggregation of this information over time would enable DoD to perform reviews focused on trends in MII performance.

___________________

² D. Sivillo, NAVY UARC Deputy Program Manager, Naval Sea Systems Command 106, 2008, “Navy UARCs,” presentation to the 2008 University Affiliated Research Center (UARC) Administrative Challenges Meeting, May 8-9, https://uarc2008.files.wordpress.com/2008/05/dave_sivillo_navsea-106-uarc-business3.pdf.

This information could also provide critical inputs to support institute-to-institute comparisons and identification of best practices and institute performance issues.

ADVANCED MANUFACTURING ECOSYSTEMS

Next is a discussion of the importance of manufacturing ecosystems for the achievement of DoD’s and the nation’s manufacturing capability goals for the MIIs, which expands on material on this subject in Chapter 3 of the interim report in the section “Question 3: Has the MII Performed Well?” It is important to note that establishing and growing manufacturing ecosystems is one of the three chartering principles of Manufacturing USA MIIs. In the briefing to the sponsor on the interim report, the sponsor expressed a desire to have additional discussions on the important subject of manufacturing ecosystems and assessment of their development in the MIIs.

At the national level, manufacturing is increasingly seen by many as a critical sector for economic prosperity and national security. Indeed, many experts hold that U.S. economic prosperity and national security depends on its ability to competitively produce advanced technology goods and services in the United States.

However, the manufacturing base in the United States continues to show evidence of gaps and weaknesses. The “hollowing out” of the U.S. industrial commons that took place in the past several decades has resulted in huge trade deficits year after year and a loss of over 5 million manufacturing jobs since 2000. U.S. trade deficits continue to grow as evidence of continuing weakness. Particularly alarm-

ing is the trade deficit for advanced manufactured goods, which started to appear in 2000, reached $130 billion in 2019, and then reached $190 billion in 2020, indicating a significant threat to national security and future economic prosperity.³ Many policymakers are increasingly convinced that what is invented in the United States should also be produced in the United States, so that the nation can directly benefit—in the form of economic prosperity, quality employment opportunities, and national security—from basic research and development (R&D) investment.

From this perspective, it is important for the United States to have globally competitive advanced manufacturing ecosystems, including but not limited to the following characteristics: resilient supply chains; strong regional industrial clusters and strategic partnerships; a skilled workforce; infrastructure; a robust industrial policy framework—including a regulatory, intellectual property, and tax structure that promotes globally competitive innovation; and strong investment capabilities to nurture and grow high-tech manufacturing startups.

Advanced manufacturing ecosystems can provide effectiveness in technology development while enabling the ecosystem actors to co-innovate around common challenges. This ecosystem approach requires deliberate coordination among and between various parties to solve shared challenges and meet shared objectives.

With the advancement of Industry 4.0, more manufacturers are increasingly adopting new digital technologies and transforming their business models and practices. In the case of digital manufacturing initiatives, the Deloitte and MAPI Smart Manufacturing Ecosystem Study identified four primary ecosystems that support advanced manufacturing ecosystems (see Table 2.1).

Since DoD MIIs vary in the nature, maturity, and intended use of their focused technology, each MII and its stakeholders need to identify the shared challenges and shared opportunities that their ecosystem wishes to develop, grow, and maintain.

Most believe there is a need for a regional manufacturing ecosystems for innovation to thrive.^4,5 Research has also shown that ecosystems also need and benefit from an organized public sector engaged with the private sector—including companies and area business groups—along with education institutions, pursuing a joint strategy. An MII program function is to strengthen regional economic clusters by boosting innovation and collaboration through increased connectivity. A Deloitte 2016 study found that the MIIs were strengthening regional industrial

___________________

³ U.S. Bureau of Census, “Foreign Trade, Trade in Good with Advanced Technology Products, 2019 and 2020,” https://www.census.gov/foreign-trade/balance/c0007.html#2021.

⁴ R. Atkinson, M. Muro, and J. Whiton, 2019, The Case for Growth Centers: How to Spread Tech Innovation Across America, Brookings Institution and Information Technology and Innovation Foundation, December, https://www.brookings.edu/research/growth-centers-how-to-spreadtech-innovation-across-america.

⁵ B. Armstrong, 2019, “Brass Cities: Innovation Policy and Local Economic Transformation,” thesis, Department of Political Science, Massachusetts Institute of Technology, Cambridge, MA.

TABLE 2.1 Four Primary Advanced Manufacturing Ecosystems and the Focus Areas of Shared Objectives

clusters.⁶ The cluster entities derive mutual benefits from geographic proximity and the ability to share knowledge, mutual access to a skilled manufacturing workforce, and the use of shared infrastructure. Management of the process of development and sustainment of an ecosystem for the DoD MIIs is essential to the success of the organizations. In addition to driving technology and leading EWD activities in the appropriate fields, MIIs should be leaders in the development of ecosystems, which will enable these other key objectives to be achieved. Of course, the specific composition of ecosystems within a particular MII’s reach will vary based on the MII’s technology focus and location. But MIIs appear to present a significant regional ecosystem opportunity in their technology fields, and there are opportunities to identify best practice strategies for ecosystem development and sustainment.

The Advanced Manufacturing Partnership, in its Report to the President on Capturing Domestic Competitive Advantage in Advanced Manufacturing, recommended “the establishment of a national network of Manufacturing Innovation Institutes to bridge the gap between basic research performed in universities and national laboratories, and U.S. production enterprises, particularly small and medium sized enterprises (SMEs).”⁷ It noted that the vast majority of U.S. SMEs have limited capital and R&D, so they typically lag behind large firms in developing, acquiring, and utilizing new manufacturing technology and processes. Hence, a founding tenet for establishing MIIs was their ability to reach small and medium-sized manufacturers

___________________

⁶ Deloitte, 2017, Manufacturing USA: A Third-Party Evaluation of Program Design and Progress, January, https://www2.deloitte.com/us/en/pages/manufacturing/articles/manufacturing-usa-program-assessment.html.

⁷ Executive Office of the President, 2012, Report to the President on Capturing Domestic Competitive Advantage in Advanced Manufacturing, President’s Council of Advisors on Science and Technology, https://www.manufacturing.gov/sites/default/files/2018-01/pcast_amp_steering_committee_report_final_july_27_2012.pdf, p. 22.

(SMMs) and to engage them in regional ecosystems. The expectation was that MIIs would “establish distributed manufacturing support centers throughout the region to assist SMEs that want to adopt new technologies.”⁸

However, since that time, what is observed is that although MIIs typically reach larger firms as well as the SMMs developing their technologies, they less frequently reach SMMs with an interest in adopting their technologies. As summarized in the Manufacturing Extension Partnership (MEP) 2019 Annual Report, small manufacturers “tend to explore opportunities before making decisions to commit or implement and that they are interested in demonstration sites and interactive experiences that help them understand technologies. Small manufacturers are also interested in state-of-the-art technologies that can be leveraged in the very near term and in local resources (within less than a two-hour drive) which are particularly helpful for engagement.”⁹ The weakness of SMMs to effectively adopt technologies can jeopardize the efficiency of entire supply chains that depend on the productivity and precision of SMMs that make it up. Indeed, data suggest that U.S. manufacturing productivity increases overall have been at very low levels for over 15 years, and that the productivity gap between small and large firms that continues to grow is a major contributor.¹⁰

The NIST MEP, a public-private partnership dedicated to serving SMMs, has centers in 50 states, as well as Puerto Rico (Figure 2.1).^11,12

Supported by state and federal funding, MEPs are organized to spread manufacturing processes and technologies to SMMs. MEPs tend to focus on meeting SMMs “where they are”—both in terms of technology maturity and physical location. For example, over the last decade, the focus of MEPs has been in areas such as quality assurance and lean manufacturing relevant to their state’s SMM population. As a result, early efforts to align MEP staff into the newly formed MIIs was fraught with many challenges, including inherent mismatches in expectations, roles and responsibilities, business and funding models of the MIIs, MEP centers, and small manufactures.^13,14 For instance, many of the SMMs in the MEP network were looking for commercially ready off-the-shelf technologies; whereas, at that time, the maturity of the technologies being developed by the newly formed DoD MIIs was

___________________

⁸ Ibid., p. 24.

⁹ NIST Manufacturing Extension Partnership (MEP), 2019, Annual Report 2019, https://www.nist.gov/mep/manufacturing-reports/nist-mep-publications.

¹⁰ Massachusetts Institute of Technology, 2021, Advanced Technology, Advanced Training: A New Policy Agenda for U.S. Manufacturing, Initiative for Knowledge and Innovation in Manufacturing, Cambridge, MA, May.

¹¹ See the NIST Manufacturing Extension Partnership (MEP) website at https://www.nist.gov/mep.

¹² NIST MEP, 2019, Annual Report 2019.

¹³ Discussion with Phillip Singerman, formerly of NIST, February 9. 2021.

¹⁴ Discussion with Mary Ann Pacelli, NIST MEP, April 6, 2021.

too early-stage. In addition, for SMMs adopters, the MII membership model was often too costly, largely focusing on technology development programs, without options to focus on near-term equipment training and EWD needs. Finally, physical overlap of location for willing parties may not have existed—an MII’s state-level presence would have needed to match that of a specific MEP center interested in collaborating on that MII’s specific technology focus area. This multitude of requirements just to get started means productive collaborations are unlikely to form.

However, now is a different point in time. With the advancement of Industry 4.0, MEPs increasingly need to shift their SMMs extension efforts from lean manufacturing to adoption of digital and advanced manufacturing technologies. At the same time, MIIs—particularly MIIs working on “closer-in,” more deployable technologies that are being adopted in factories, such as digital, robotics, and additive technologies—are in a position to support these efforts. In doing so, these MIIs can partner with MEPs to extend their reach to more SMMs to encourage their adoption of the new MII technologies. This demand is catalyzing meaningful re-engagement and collaboration between the MEPs and MIIs. For example, Ohio’s MEP Magnet launched the “American Works” project to catalog what MIIs are doing in an effort to identify which programs are aligned to MEP needs and to identify what can be shared with SMMs. The project is purposeful with shared objectives, tying together complimentary expertise and networks of MIIs, MEPs, and universities. It is helping the MEP center get up to speed in EWD for new production technologies and demonstrating how the MII EWD directors can be collaborators to help MEP extend its reach and impact to SMMs.

Because MEPs represent a way to access regional SMMs at scale, increased MII engagement with MEPs appears to present a significant regional ecosystem opportunity to address speed, scale, and impact at a rate greater than either organization can do alone. The committee recognizes that a first attempt at an MII-MEP collaboration showed mixed results. However, the committee believes that the value in the collaboration for ecosystem development warrants that this be revisited and prioritized by DoD. In particular, a purposeful collaboration between MIIs and MEPs with well-established relationships in the commercial defense industrial base, as well as with the many depots, arsenals, ammunition plants, and shipyards in the DoD organic industrial base (OIB) (as discussed in Chapter 3), could be mutually beneficial in terms of technology adoption, strengthening the manufacturing ecosystem, as well as training of a skilled workforce.

Of course, MEPs (with their reach to SMMs) are not the only ecosystem engagement opportunity for MIIs. Working relationships with state and local government agencies; educational institutions, including community colleges and secondary schools; and area industry, small and large, will be key to effective ecosystem involvement. Another way to access regional SMMs and ecosystems at scale may

be the Defense Manufacturing Communities Support Program (DMCSP).¹⁵ The DMCSP designates and supports regional consortia in communities with defense manufacturing to strengthen the national security industrial base. DMCSP in the past 2 years has made awards to upskill regional workforces, familiarize area manufacturers with new technologies, and spur innovation in specialized sectors—all of which fit the goals of MIIs. OSD ManTech has worked with this program, and it has featured MII participation. While there are currently a limited number of DMCSP awardees, if it continues, it could be a potential ecosystem engagement program for continued MII participation.

Additionally, Purdue University and the Indiana Business Research Center appropriately identifies the industry and occupational clusters and related assets present in a user-defined geography (micro perspective), while the second tool created by Harvard University’s Institute for Strategy and Competitiveness identifies the clusters driving the national economy (macro perspective). The Harvard cluster tool also provides a cluster registry where cluster organizations can find and connect with key organizations in their region to help advance cluster initiatives.¹⁶

In the interim report, 11 possible assessment activities, on pages 132–133 in Appendix C, were suggested for the health and growth of regional ecosystems. Given the committee’s further evaluation of this topic and the recognized importance of enabling technology adoption at SMMs to enable robust regional ecosystems, the committee now adds the following potential assessment activities:

1. Assess whether efforts to engage regional MEPs have been effective in reaching SMMs.
2. Assess how far advanced technologies have transitioned down and through the manufacturing supply chain.

FINDINGS

Finding 2.1: Although each MII has its unique technology focus, all MIIs share some key common challenges, such as EWD and ecosystem development. The committee noted the effectiveness of the OSD ManTech team working on EWD is

___________________

¹⁵ See Department of Defense (DoD), “Defense Manufacturing Community Support Program,” Office of Local Defense Community Cooperation, https://oldcc.gov/defense-manufacturing-community-support-program. This program is part of DoD’s Office of Local Defense Community Cooperation (formerly the Office of Economic Adjustment), see DoD, “An Introduction to OLDCC,” https://oldcc.gov/introduction, and Association of Defense Communities, 2021, “Office of Economic Adjustment Officially Renamed,” On Base, January 7, https://defensecommunities.org/2021/01/office-of-economic-adjustment-officially-renamed/.

¹⁶ U.S. Economic Development Administration, “Tools for Economic Development,” Department of Commerce, https://www.eda.gov/resources/tools/.

providing for all MIIs a shared EWD vision, an operational model, and coordination. A similar MII network-level team/function for manufacturing ecosystems will accelerate ecosystem development.

Finding 2.2: Since DoD MIIs vary in the nature, maturity, and intended use of their focused technology, each MII and its stakeholders need to identify the shared challenges, shared opportunities, and technology transition and production risks in the ecosystem that they wish to develop, grow, and maintain.

Finding 2.3: The MEP program sponsored by NIST, which operates in every state, is organized to bring process and technology advances to SMMs. Given the importance of regional ecosystems and supply chains for implementation of advanced manufacturing, a reassessment of how MIIs might engage with MEPs, state/local government agencies, educational institutions, and companies including the SMMs they serve appears important. On the EWD front, there have been successful collaborative efforts organized by the OSD ManTech EWD team and the National MEP EWD lead. To institutionalize and sustain long-term engagement of the MIIs with MEP centers on technology, EWD, and manufacturing ecosystems, OSD ManTech and National MEP Office’s planning, coordination and guidance would be essential. The Defense Manufacturing Community Support Program represents another set of regional consortiums that could enable the MIIs to interact with SMMs at-scale. MIIs, particularly MIIs working on more deployable technologies now starting to be adopted in factories, could build new partnerships with MEPs and DMCSP hubs to reach SMMs to encourage adoption of their mature (technology readiness level 7) advanced manufacturing technologies.

RECOMMENDATIONS

Recommendation 2.1: OSD ManTech should create an Advanced Manufacturing Ecosystems Team. Such a team will (1) help shape a vision and key strategies for the DoD MII ecosystems, (2) identify and share best practices across all DoD MIIs, and (3) coordinate manufacturing ecosystems initiatives across DoD-funded MIIs similar to what the education and workforce development lead does for education and workforce development.

Recommendation 2.2: With respect to manufacturing ecosystem development, OSD ManTech should direct each MII to:

1. Describe the advanced manufacturing ecosystem that they envision based on the needs of DoD science and technology, acquisition, and sustainment, as well as the commercial manufacturing community they serve.

2. Articulate its role and responsibilities and develop a well-thought-out plan for growing and improving the ecosystem.
3. Develop a process by which the MII will monitor the health, resilience, return on investment, and effectiveness of its ecosystem; identify gaps—innovation, human capital, partnership, risks, supply chains, infrastructure, and other factors; and take necessary actions or make actionable recommendations to DoD to close the gap as quickly as possible.

Recommendation 2.3: To ensure effective and broad-based deployment and adoption of the technologies and workforce developed by the MIIs and the strengthening of manufacturing ecosystems, OSD ManTech should assist MIIs in forming strong partnerships with other federally-recognized regional ecosystems, as well as the federal offices that promote these ecosystems, including the state-level Manufacturing Extension Partnerships, the Defense Manufacturing Community Support Program hubs, and their federal offices (the Department of Commerce’s National Institute of Standards and Technology Manufacturing Extension Partnership and DoD’s Office of Local Defense Community Cooperation, respectively).

Recommendation 2.4: To leverage DoD’s investments in MIIs in building and growing U.S. manufacturing ecosystems, OSD ManTech should continue to assist MIIs in forming symbiotic partnerships with state and local government for economic development, education and workforce development, and research and development. The committee recommends that OSD ManTech initiate a couple of pilot projects across all MIIs in key regions and then scale across the balance of the United States after refinement.