Summary
BACKGROUND AND TASK
Since 1959, the National Institute of Standards and Technology (NIST) has annually engaged the National Academies of Sciences, Engineering, and Medicine to assemble panels of experts—from academia, industry, medicine, and other scientific and engineering communities of practice—to assess the quality and effectiveness of the NIST measurements and standards laboratories, of which there are six,1 as well as the adequacy of the laboratories’ resources. These reviews are conducted under contract at the request of NIST. In fiscal year 2023, NIST asked the National Academies to assess the Material Measurement Laboratory (MML). The assessment process included a site visit by the panel, including laboratory tours, individual engagement with MML researchers, and follow-on questions after the site visit. The panel then used its collective experience and expertise to assess MML according to the statement of task and make recommendations. MML is assessed in its entirety every 3 years.
The statement of task has four main components. First, the panel is asked to assess MML’s technical programs, how the quality of research at MML compares to similar programs of research internationally, and whether the quality of the programs at MML are adequate for MML to achieve its objectives. Second, the panel is also asked to assess the portfolio of scientific and technical expertise at MML, whether it is world-class, and how well it supports MML’s technical programs and MML’s ability to meet its objectives. Third, the panel is asked to assess the adequacy of MML’s budget, facilities, equipment, and human resources, and how well these support MML’s technical work and achieving MML’s goals. Finally, the panel is asked to assess the effectiveness of MML’s efforts to disseminate products of its work including how well MML’s work is driven by stakeholder needs, how effective and comprehensive the dissemination and technology transfer mechanisms are, and how well MML is monitoring the stakeholder use, and the impact of, MML’s output.
MATERIAL MEASUREMENT LABORATORY
MML comprises six technical divisions and two offices. The two offices manage programs related to NIST standard reference materials and NIST data products. The technical divisions engage in research and development of the measurement science, standards, technology, and data required to support the nation’s need to design, develop, manufacture, and use materials. These divisions interact extensively with both industry and public institutions to advance the economy and provide tools for the creation of knowledge.
The divisions and offices in MML are the Applied Chemicals and Materials Division, Biosystems and Biomaterials Division, Biomolecular Measurement Division, Chemical Sciences Division, Materials Measurement Science Division, Materials Science and Engineering Division, Office of Data and Informatics, and Office of Reference Materials. The Applied Chemicals and Materials Division is located
___________________
1 The six NIST laboratories are the Communications Technology Laboratory, the Engineering Laboratory, the Information Technology Laboratory, the Material Measurement Laboratory, the NIST Center for Neutron Research, and the Physical Measurement Laboratory.
on the NIST campus in Boulder, Colorado. The remainder of the divisions and offices are located on NIST’s campus in Gaithersburg, Maryland.
CONCLUSIONS AND RECOMMENDATIONS
Assessment and General Conclusions
Across the MML divisions and offices, research staff continue to demonstrate impressive productivity in scientific publications, standards interactions, intellectual property activity, and customer engagement.
The technical quality of the research at MML is excellent. The work being done at MML not only compares favorably with work being done at other national metrology institutes around the world, in some cases it is world leading. For example, there is cutting-edge work being done in standards for additive manufacturing, a new albumin standard reference material that will allow for far more accurate testing for kidney disease, and a new hair peptide analysis technique that is free from concerns about DNA contamination and might be sensitive enough to tell the difference between identical twins. The quality of the technical work is generally adequate for MML to meet its technical objectives. There is room for improvement, however, and this is addressed in each of the chapters where the panel saw the need.
The portfolio of scientific and technical expertise at MML is impressive and is highly recognized in the scientific community worldwide. The expertise resident at MML is world-class in the areas of MML’s mission and objectives. MML staff are active in many standards bodies, often in leadership roles. They produce an impressive output of technical papers and work products. MML staff have won many awards, such as being named fellows of ASM International, ASTM International, and the American Physical Society. They have received the ASTM International Award the Chemical Society of Washington’s Charles Gordon Award, of Merit, and many Department of Commerce and NIST named awards. As with the technical programs, there are places where there is room for improvement; these are addressed in the chapters where they occur.
In contrast with the quality of the technical work and expertise at MML, there are challenges in the areas of budget, facilities, equipment, and human resources. While these are addressed in more detail in the report, generally MML is experiencing budget stress including flat budgets that either barely hold or lose ground to inflation, some of its equipment that is dated and in need of refreshing, and serious challenges regarding MML’s human resources support. The biggest challenge faced by MML’s staff, and across NIST, is the condition of many of MML’s facilities. There are serious deficiencies in places that actively impede MML’s work. These are not issues that MML can address on its own; they require action by NIST headquarters and work is underway at the headquarters level to address MML’s—and all of NIST’s—facilities problems. All of these issues are discussed in the body of the report. Regarding facilities, this report notes MML-specific challenges and ties to the National Academies’ 2023 report Technical Assessment of the Capital Facility Needs of the National Institute of Standards and Technology (NASEM 2023; hereafter the “Capital Facility Needs report”).
Regarding the dissemination of program outputs, MML engages in extensive collaborative work, across MML, NIST, and an impressive number of external stakeholders. In essence, MML continues its essential role of supporting U.S. industry metrology, technology, and standards needs. The strong support of its industrial stakeholders is shown in part by the significant number of cooperative research and development agreements in this review along with an increase in nondisclosure agreements. MML researchers are highly published and influential, they are very active in international standards bodies, holding several leadership positions in those bodies, and they hold many patents. Generally, MML’s work is responsive to stakeholder needs and its dissemination and technology transfer mechanisms are effective. While there is significant evidence of the impact of MML’s work, the panels make some
recommendations about better tracking this impact and thus more effectively demonstrating MML’s value.
The recommendations in report’s chapters are grouped by division or office to provide actionable suggestions that address each division’s unique needs. This Summary highlights overarching, crosscutting themes that emerged over the course of the 2023 assessment.
KEY RECOMMENDATIONS
Adequacy of Facilities
As highlighted in the National Academies’ 2020 MML assessment, An Assessment of the Material Measurement Laboratory at the National Institute of Standards and Technology: Fiscal Year 2020 (NASEM 2021), and more recently in Capital Facility Needs report (NASEM 2023), aging facilities and infrastructure present significant handicaps. Notably, the 2023 report found that NIST research staff spent from 10 to 40 percent of their time working around facility shortcomings, with the typical reported values being 20 percent (NASEM 2023). As direct consequence of these limitations, at NIST generally and specific to this report, MML researchers are left to mitigate these limitations through repeating their work or implementing workarounds with limited research budgets.
The MML laboratories in Boulder, Colorado, and Gaithersburg, Maryland, are negatively affected by poor temperature and humidity control in addition to inadequate air flow in their laboratories, which are not compatible with state-of-the-art measurements and instrumentation. On the Gaithersburg campus in 2022, aging infrastructure led to serious flooding of MML research laboratories, resulting in a loss of millions of dollars of equipment and time. The facilities are generally inadequate for the performance of modern materials chemistry related processes. Much of the infrastructure, both buildings and equipment, has exceeded its expected life, leading to time spent inefficiently compared to the capabilities of state-of-the-art facilities and equipment in academic and industry laboratories and at metrology institutes around the world. Researchers have had to repeat experiments multiple times. They have had their ability to work interrupted by utility failures. Postdocs have ended their postdoc without having been able to do the work needed to author a publication. The impacts are building, and MML broadly is becoming a less and less appealing place to work. MML is already at a compensation disadvantage compared to industry, now it cannot always offer world-class facilities, either. While MML research efforts are world-leading, aging facilities and infrastructure are such that MML is at the cusp of losing its world-leading position and its ability to continue to attract new generations of world-leading researchers. It needs to be noted that MML—or any of NIST’s laboratories—have a very limited ability to address facilities inadequacies. Major facilities work—the construction, renovation, and modernization that is needed to resolve the worst of the problems—is managed and funded by NIST’s Office of Facilities and Property Management (OFPM). As detailed in the report Technical Assessment of the Capital Facility Needs of the National Institute of Standards and Technology, OFPM has a coordinated recovery plan to fix NIST’s facilities. The most important thing for MML to do is to ensure that OFPM is fully aware of its facilities issues and that MML’s priorities are reflected in the coordinated recovery plan.
Key Recommendation 1: The Material Measurement Laboratory (MML) and the National Institute of Standards and Technology Office of Facilities and Property Management should work together to identify and document critical laboratory infrastructure issues. They should collaborate to prioritize items that need to be addressed and create a plan with an associated timeline to address those needs. MML should actively encourage its researchers to document facilities issues and all associated time lost repeating experiments, and the portion of the research budget spent on working around facility issues.
The identified infrastructure issues also represent safety issues: floods can easily lead to sparks that can in turn initiate fires; inadequate exhaust will lead to contaminated laboratory air that can negatively impact research product quality and in the long term can impact the health of research staff. As a whole, aged infrastructure and equipment can have an impact on the overall attitude of the staff toward safety. Safety requires intimate and active management involvement, creating and driving an overall safety culture. Participation in the Occupational Safety and Health Administration’s Voluntary Protection Programs might be a useful way to approach safety at MML.2
Key Recommendation 2: Safety is as much a line management responsibility as an individual one. Leaders should recognize that every task can be done safely, and every task has inherent risks. That attitude needs to be transmitted down the line in order for each individual to take responsibility for not only maintaining but also improving the safety of their environment and the environments of those around them. The Material Measurement Laboratory (MML) should maintain statistics of leading indicators of safety risks and facilities dangers that represent the base of the safety pyramid. These statistics should be socialized within the organization in such a way that a safe and safety-conscious environment is created that results in a safety culture that supports continuous improvement. MML should consider engaging with experienced industrial partners who can bring a fresh perspective on the safety risks as a way of providing an unbiased and fresh pair of eyes to the situation.
Adequacy of Scientific and Technical Expertise
MML conducts research at a very high level. It has formal arrangements with renowned institutions such as Brookhaven National Laboratory and the Institute for Bioscience and Biotechnology Research, a joint institute with the University of Maryland, College Park, and the University of Maryland, Baltimore. MML has primary responsibility within NIST for producing and curating standard reference materials and standard reference data. In addition to maintaining critical standard reference materials and standard reference data, MML has developed newer standards products based on biology. The balance of maintaining legacy standard reference materials versus redirecting limited resources toward producing standard reference materials and standard reference data in new and emerging areas presents both a challenge and an opportunity. MML notes that areas of new opportunities include the bioeconomy and engineering biology, data science and engineering, artificial intelligence, and the circular economy.
The robustness of MML’s program against retirements or unforeseen departures could be improved. In some places there needs to be more systematic succession planning to prevent gaps from occurring. Cross-training to provide backup coverage in case the primary cognizant scientist leaves is a further way to protect against disruption. In other instances, retraining staff to take on emerging issues requiring different disciplinary knowledge was lacking.
Pandemic restrictions limited onboarding and other processes that contribute to a cohesive work culture, which has been recognized by the MML leadership team. The lack of cohesion negatively impacts an organization or culture where career mentoring and professional development for the population of term-limited scientific appointees becomes uneven.
Key Recommendation 3: The Material Measurement Laboratory (MML) should work with National Institute of Standards and Technology human resources to develop effective processes and procedures for hiring and career development of staff. Processes to ensure timely support to MML in hiring and onboarding of postdocs, term, and permanent employees should be developed and implemented.
___________________
2 More information about the Voluntary Protection Programs and how to participate can be found at Occupational Safety and Health Administration, “Voluntary Protection Programs,” https://www.osha.gov/vpp.
Key Recommendation 4: The Material Measurement Laboratory (MML) should require mentorship training for all research staff within the laboratory to ensure quality vertical mentorship and sponsorship in order to increase the impact of the postdoc and early career population both within MML and later in industry, academia, or government positions. MML should increase the number of employees who have formal leadership training to standardize mentoring and onboarding of new personnel. The training should include a formalization of clear mentor and mentee expectations in terms of work, in-person meetings, and supervision. MML should consider a standard message about the processes of transitioning from a postdoc or term appointee to a permanent staff member, and the probability thereof so as to minimize the uncertainty around this process.
Data Management and Data Infrastructure
Data science represents an increasingly important part of MML’s work to meet its mission. Within MML, the Office of Data and Informatics is a dedicated, service-oriented data resource with domain expertise in biological, chemical, and materials sciences, specializing in large and information-rich data sets.
Across MML, data collection, storage, analysis, and movement appear to be handled by multiple, fragmented platforms, and computational data does not appear to be managed in a centralized platform. In addition, not all of the experimental measurement systems have a consolidated and automated data infrastructure, which results in a fragmented data infrastructure that, unfortunately, leads to discrete, inefficient, and stovepiped experimental data infrastructure across MML. These observations lead to opportunities to unify all computational systems, data formats, and data transmission protocols into a single, uniform platform for more efficient and effective data curation, storage, processing, transmission, and security management.
Key Recommendation 5: To the extent possible, the Material Measurement Laboratory should launch an initiative to unify all computational systems, data formats, and data transmission protocols into a single, uniform platform for more efficient and effective data curation, storage, processing, transmission, and security management. This data science initiative could be applied to machine learning deployment and expand capabilities in laboratory automation, high-performance computing, and artificial intelligence. Computing infrastructure should, to the extent possible, be updated to meet current laboratory standards.
REFERENCES
NASEM (National Academies of Sciences, Engineering, and Medicine). 2021. An Assessment of the Material Measurement Laboratory at the National Institute of Standards and Technology: Fiscal Year 2020. Washington, DC: The National Academies Press. https://doi.org/10.17226/26048.
NASEM. 2023. Technical Assessment of the Capital Facility Needs of the National Institute of Standards and Technology. Washington, DC: The National Academies Press. https://doi.org/10.17226/26684.
