B

Study Methodology and Data Collection

PANEL MEETINGS

To answer its charge, the National Academy of Public Administration and National Academies of Sciences, Engineering, and Medicine convened an expert panel of individuals with broad experience in the federal government, the national laboratories, academia, and the private sector, and with expertise in the areas of public administration, budgeting, policy, production, and nuclear security. The panel participated in data gathering, analysis, and drafting of reports and recommendations, as well as provided ongoing guidance to a study team conducting the day-to-day work of the assessment following a structured methodology.

The full panel met 15 times over the course of this study, usually in person but sometimes via conference call or videoconference. Panel meetings were structured so that members could hear presentations by the Administrator and other senior National Nuclear Security Administration (NNSA) and management and operating (M&O) officials, and could discuss relevant material and issues with these decision makers.

In addition, subsets of the full panel met several times as working groups to discuss research questions and data needs around specific topics. These working groups met several times per year via video or conference call.

INTERVIEWS AND OTHER MEETINGS

Information gathering for this report included formal individual and group interviews and meetings with senior-level officials within the NNSA, the greater Department of Energy (DOE), and the Department of Defense (DoD). Some of these interviews and meetings were fairly general, while others focused on specific topics, such as the Administrator’s tenure, changes in financial and contracting processes, or the relationship between NNSA and DoD. Over the course of this study, the panel conducted more

than 140 interviews. In addition, the full panel or a subset of the panel (e.g., the co-chairs) met with the Administrator, the Deputy Administrator, and the Deputy Secretary of Energy.

Panel co-chairs and project staff also spoke frequently with the leadership and staff of NNSA’s Office of Policy. The interviews and Office of Policy calls were structured to inform the panel and staff about initiatives, objectives, plans, accomplishments, and barriers in instituting governance and management reform, and to keep the Office of Policy staff current on the activities of the panel and project staff.

The panel also held longitudinal interviews to check the “pulse” of leaders across the enterprise at specified points in time on progress made and remaining challenges. The laboratory directors and the corresponding field office managers were interviewed in January 2017. All field office managers and M&O directors (sometimes joined by their deputies and other members of the senior leadership team) were interviewed in fall 2018 and again in winter/spring 2020. In winter 2019–2020, the panel interviewed 20 NNSA headquarters leaders of both functional and program offices. Each individual was a senior-level manager within his or her respective office; almost all held Senior Executive Service (SES) status.

Each pulse check interview was conducted using a semistructured interview guide. The comments made by interviewees were organized around predetermined topic areas covered during the interview. Panel staff performed a content analysis of meeting notes to identify patterns or themes, and results included major areas of disagreement when or if they occurred.

The panel also received briefings from and held discussions with external stakeholders and experts, such as Government Accountability Office (GAO) personnel; Norman R. Augustine and Admiral Richard W. Mies, co-chairs of the Augustine-Mies report; Victor H. Reis, former Assistant Secretary for Defense Programs and Senior Advisor to the DOE Office of the Secretary and Undersecretaries; DOE Office of Science officials; and leaders of federally funded research and development centers, such as the Lincoln Laboratory at the Massachusetts Institute of Technology and Pacific Northwest National Laboratory.

DISCUSSION GROUPS

The panel carried out 57 discussion groups to gather thoughts from personnel below senior leadership levels on governance and management challenges and the progress of reforms. A total of 459 individuals participated in the discussions, which were facilitated by panel members and/or staff. Participants were drawn from a mix of organizations, including those with functional and those with program responsibilities from NNSA headquarters and field offices, and from the M&Os. The levels of responsibility and years of experience varied greatly among the participants. The majority of the discussion groups were held during site visits (see below) or at NNSA headquarters, but seven groups were conducted via videoconference. The discussion groups differed from group interviews in that there were five to seven topics with a set of discussion prompts. The goal was for group participants to interact with each other and react to each other’s perceptions and statements.

The discussion topics evolved slightly as the study progressed and NNSA circumstances and initiatives changed, but they were consistent enough for the panel to get a sense of whether or not individuals below the senior management levels were aware of and experiencing change, as well as changes over time.

While the discussion groups provided insights about governance and management within the enterprise, information received through these should not be viewed as conclusive, owing to the small sample size. Nevertheless, an analysis of the discussions, particularly when taken in conjunction with the pulse checks and other interviews, illuminates where progress is being made and where challenges likely remain.

SITE VISITS

The full panel made 14 site visits between July 2017 and March 2020 (see Table B.1). During these visits, the panel met with senior leadership of the field offices and senior leadership and senior researchers of the M&Os. The panel also held discussion groups with midlevel managers and other field office and M&O staff.

In addition, a working group comprised of six panel members conducted site visits in 2019 to evaluate the science and engineering capabilities base at the three NNSA laboratories in four locations: Lawrence Livermore National Laboratory; Sandia National Laboratories, California; Sandia National Laboratories, New Mexico; and Los Alamos National Laboratory. The panel met with senior research leadership, senior researchers, mid-career researchers, and early-career researchers.

Another working group, focusing on pit production, held virtual site visits (owing to the coronavirus) in March 2020 with staff from Los Alamos National Laboratory and the Savannah River Site.

TABLE B.1 Panel Site Visits

DOCUMENT REVIEW

The panel and staff performed extensive research and analysis of documents produced by NNSA and DOE, such as policies and plans; congressional hearings; and previous reports by the Augustine-Mies Commission, Commission to Review the Effectiveness of the National Energy Laboratories (CRENEL), GAO, and others. These documents provided a baseline and helped the panel to track implementation and institutionalization of governance and management reforms, and associated outcomes.