A

Data Collection Methodology

PANEL MEETINGS AND INTERVIEWS

The full panel met three times during the year leading up to this report. Two 2-day meetings were held at the National Academy of Sciences in Washington, D.C., and one 1-day meeting was held at Sandia National Laboratories (SNL) in Albuquerque, New Mexico. Each meeting included panel members, staff, and guests from across the nuclear security enterprise, including the Administrator at one meeting and the Deputy Administrator at another. The meetings were held on May 16, September 5-6, and December 5-6, 2019. Panel meetings were structured so that members could hear presentations by senior National Nuclear Security Administration (NNSA) and management and operating (M&O) officials, and could discuss relevant material and issues with these decision makers.

Additionally, information gathering for this report included formal interviews with senior-level officials within the NNSA and greater Department of Energy (DOE). Some of these were fairly general, while others focused on specific topics such as the Administrator’s tenure and changes in financial and contracting processes.

In addition to these interviews, panel co-chairs and project staff spoke frequently with the leadership of NNSA’s Office of Policy. The interviews and Office of Policy calls were structured to inform the panel and staff about initiatives, objectives, plans, accomplishments, and barriers in instituting governance and management reform.

PANEL DISCUSSION GROUPS AND PULSE CHECK INTERVIEWS

The panel carried out a number of discussion groups and interviews in 2019 to gather thoughts about governance and management from personnel across the nuclear security enterprise. Through these interactions, the panel also gained insight about the degree to which the Administrator’s principles of governance and management have been heard and internalized.

The panel conducted six discussion groups during a site visit to Los Alamos National Laboratory (LANL) in May 2019. Those groups were comprised of NNSA field office and M&O personnel drawn from a mix of organizations that included both those with functional and those with program responsibilities. A total of 37 individuals participated in those discussions.

In addition, the panel held 12 discussion groups between September 25 and October 31, 2019. These discussion groups took advantage of the fact that NNSA was at that time holding focus-group sessions to explore related topics. The members of 12 of those focus groups were able to stay for a separate session with the panel. (NNSA’s “focus groups” and the panel’s “discussion groups” used different methodologies, hence the difference in nomenclature.) Each discussion group consisted of a variety of individuals from across the nuclear security enterprise, including personnel from NNSA headquarters and field offices, and staff from various M&O partners who, collectively, covered a broad range of levels of

responsibility and lengths of tenure in both programmatic and functional offices. The total number of discussion group participants was 116; the groups ranged in size from 7 to 13 participants.

Subsequently, in the winter of 2019-2020, the panel interviewed 20 NNSA headquarters leaders of both functional and program offices. Each individual was a senior-level manager within their respective office. The participants held varying lengths of tenure and levels of responsibility within NNSA. The offices of leaders interviewed included NA-10, NA-20, NA-50, NA-70, NA-80, NA-APM, NA-EA, NA-GC, and NA-MB. Almost all of the participants held Senior Executive Service (SES) status.

Each pulse check interview was conducted using a semi-structured interview guide. The comments made by interviewees were organized around predetermined topic areas covered during the interview. Panel staff performed a content analysis of meeting notes to identify patterns or themes, and results included major areas of disagreement when or if they occurred. These pulse check interviews complement the interviews the panel held with field office and M&O leaders in fall 2018, which it plans to repeat in spring 2020.

The panel also spoke to Los Alamos field office and M&O leaders during the May 2019 site visit.

While the panel’s discussion groups provided insights about governance and management within the enterprise, information received through these should not be viewed as conclusive, owing to the small sample size. Nevertheless, an analysis of the discussions, particularly when taken in conjunction with the pulse checks and other interviews, illuminates where progress is being made and where challenges likely remain. The results of these data collection activities were shared with the Office of Policy and informed the panel’s observations regarding the effectiveness of NNSA’s governance and management and culture change initiatives as described in Chapters 1 and 2.

SITE VISITS

Los Alamos National Laboratory

On May 14-15, 2019, the panel visited Los Alamos National Laboratory (LANL). While there, it interacted with site leaders and held discussion groups with federal and M&O personnel. While at LANL, the panel conducted six 90-minute discussion groups with field office (NA-LA) and LANL participants. The discussion groups contained 6-12 participants, each of whom had 5 or more years of working experience with NNSA and/or LANL. A total of 37 individuals participated in the discussions.

Site Visits to NNSA Laboratories to Assess the Health of Work to Support the Science and Engineering Capabilities of the Enterprise

A working group comprising six panel members conducted site visits from July 31 through August 2, 2019, to evaluate the science and engineering (S&E) capabilities base at the three NNSA laboratories in four locations: Lawrence Livermore National Laboratory (LLNL), Sandia National Laboratories (SNL, California), Sandia National Laboratories (SNL, New Mexico), and Los Alamos National Laboratory (LANL). The panel met with senior research leadership, senior researchers, mid-career researchers, and early-career researchers.

The working group focused its data collection on five categories as they relate to governance and management: mission and science, people, leadership, infrastructure, and bureaucracy.

DOCUMENT REVIEW

The panel’s staff performed secondary data collection through documents provided and produced by NNSA and the greater DOE. Staff and NNSA shared specific documents with panel members through e-

mail correspondence, at panel meetings, and in other meetings and discussion. The documents helped the panel track changes and institutionalization of governance and management principles within NNSA.