2
Phase 1 Report Recommendations
INTRODUCTION
The committee’s Phase 1 report (NASEM, 2021) examined how the Department of Energy’s Office of Environmental Management (DOE-EM) focused on projects—how they are executed, the appropriateness and effectiveness of controls and oversight, and how they are realized through contracts (NASEM, 2021, p. 2). The report presented 16 findings, 4 conclusions, and 9 recommendations1 addressing these issues in four chapters:
- Chapter 4: Project management policies, processes, and procedures
- Chapter 5: Project management metrics
- Chapter 6: Contract structures
- Chapter 7: Contract management metrics
The present chapter revisits the recommendations from the Phase 1 report (referred to below in italic) in light of information gathered during the virtual site visits described in Chapter 1. These recommendations are presented in Annex 2.A and summarized below. Annex 2.B provides definitions for some key terms and concepts used in this chapter and elsewhere in the report.
___________________
1 NASEM (2021, p. 16) describes findings as “facts the committee noted to be of particular importance”; conclusions as the “significance of these facts for project effectiveness and efficiency”; and recommendations, which “translate these [findings and conclusions] into action, assigning a measurable action to a specific actor.”
PROJECT MANAGEMENT POLICIES, PROCESSES, AND PROCEDURES
The committee recommended that DOE Order 413.3B be confirmed, clarified, and expanded to cover all capital asset and DOE-EM projects (Recommendation 4-1). The committee also recommended that the order be clarified to incorporate best practices with respect to dispute prevention and resolution (Recommendation 4-2).
PROJECT MANAGEMENT METRICS
The committee recommended that DOE-EM (1) form a joint task force or less formal cooperative structure with other federal organizations to share and compare best project management practices for using indefinite delivery/ indefinite quantity (IDIQ) contracts (Recommendation 5-1); (2) modify its Earned Value Management System (EVMS) to capture a project’s temporal status more clearly and explicitly (Recommendation 5-2); and (3) explicitly include the percentage of cost overrun or underrun in the project success metrics dashboard, rather than the current “green/yellow/red” metric, to bring more transparency to cost performance (Recommendation 5-3).
CONTRACT STRUCTURES
The committee recommended that DOE-EM (1) establish well-defined, outcomes-based intermediate end states in its 10-year cleanup contracts; (2) develop clear, measurable metrics to assess site-based achievement of those intermediate end states; and (3) report progress on these metrics across the portfolio of end-state contracts on a quarterly basis and use those metrics as key DOE-EM performance measures (Recommendation 6-1). The committee also recommended that DOE-EM structure task orders on a scale appropriate for defining intermediate outcomes, award fewer individual tasks, and apply to these task orders the same management oversight as currently required for major systems projects exceeding $750 million in total cost (Recommendation 6-2).
CONTRACT MANAGEMENT METRICS
The committee recommended that DOE-EM ensure that the contracts it issues for cleanup work create a consolidated set of unambiguous “subjective” criteria for similar types of cleanup activities, and also that DOE-EM use these criteria in the evaluation of all contract performance across its portfolio (Recommendation 7-1).
FINDING ON PHASE 1 REPORT
FINDING: The committee obtained no additional information or insights from its virtual site visits that would lead it to change or amend any findings or recommendations from its Phase 1 report.2
In fact, the information obtained from the committee’s virtual site visits reinforced the findings and recommendations in the Phase 1 report, illustrated in this section with several examples.
At the Savannah River Site, the committee was provided with an overview of the Salt Waste Processing Facility (SWPF) project, which was conducted under Order 413.3B. The committee was told that with a total project cost (TPC) of $2.296 billion and CD-4 (Critical Decision-4, Approve Start of Operations or Project Completion) of August 2020, the project met budget and schedule targets, with project metrics success reported as “green.”3 However, the original budget and schedule targets for the project were $900 million with a CD-4 date of November 2013. In other words, the project cost and schedule were significantly higher than the original baseline.4 The success metric would not have been green under the original baseline. This example reinforces the importance of the Recommendation 5-3 to “explicitly include the percentage of cost overrun or underrun in the project success metrics dashboard relative to the original baseline, rather than the current ‘green/yellow/red’ metric.”5
The X-333 deactivation project was discussed with the committee during its visit to the Portsmouth Site. The project, whose cost was given as $366.6 million to date,6 was conducted as an operational activity and not as a capital asset project subject to Order 413.3B. The committee did not receive a clear description of whether this project was on budget or schedule. The Portsmouth cleanup and Decontamination & Decommissioning project began in 2011. The X-333 project was started in 2016. Both are still ongoing, with preparation for demolition under way. This example illustrates the importance of Recommendation 4-1 to manage these kinds of projects using Order 413.3B.
___________________
2 This finding is unnumbered because it does not explicitly address any of the task statement elements.
3 P. Marks, 2021, “Office of Environmental Management, Salt Waste Processing Facility, 05-D-405,” slide 10, presentation to the committee, February 8, available from the National Academies Public Access Records.
4 The project was re-baselined in 2015 and again in 2021; part of the reason for delay was the long commissioning period, which should have been anticipated for a project of this size and complexity.
5Recommendation 5-3 in the Phase 1 report also specified that everything should relate to the original baseline, not any increased level of funding.
6 M.J. Lilly, III, 2021, “X-333 Process Building Deactivation at the Portsmouth Gaseous Diffusion Plant, DOE Order 413 Appendix D,” slide 22, presentation to the committee, March 19, available from the National Academies Public Access Records Office.
The X-326 demolition project at Portsmouth will be an Order 413.3B project. It had an original baseline cost of $154.4 million. The water treatment (C-Train) was pulled out of this activity and will be conducted separately (the TPC for the X-326 project without C-Train is $127 million). The committee is concerned that the water treatment activity is being conducted separately because it is under $50 million7 and therefore not subject to Order 413.3 requirements.
The Portsmouth Onsite Waste Disposal Facility project, which is an Order 413.3B project, was also discussed during the site visit. The committee judged that this project was proceeding effectively using established EVMS and risk-management practices (see also GAO [2019]).8
As a sidenote, the Portsmouth and Paducah cleanups are similar and are being conducted under one combined office (Portsmouth/Paducah Project Office). DOE-EM has decided to clean up Portsmouth first and then Paducah. The decision to not proceed concurrently will delay final cleanup of Paducah, but it will also allow DOE-EM to apply any lessons learned from the Portsmouth cleanup to Paducah.
The committee observed in its virtual site visits to the Nevada and Hanford sites that operations funds were being used to conduct cleanup efforts that should have been managed as projects. The committee saw little evidence for coordination of efforts, knowledge of program management techniques, or site awareness of how to accelerate cleanups. These observations were in line with those identified in the Phase 1 report and by GAO (2019).
The Government Accountability Office (GAO) has also identified project execution difficulties at DOE-EM sites. For example, GAO (2022) identified execution difficulties with two Order 413.3B projects at the Waste Isolation Pilot Plant in New Mexico (GAO, 2022). One project encountered significant cost and schedule baseline increases, most notably from the lack of a documented risk assessment process. The other project encountered regulatory permitting issues. The 2022 GAO report underscores the systemic project management problems that have plagued DOE-EM over the past two decades.
REFERENCES
DOE (Department of Energy). 2018. Program and Project Management for the Acquisition of Capital Assets. DOE Order 413.3B. Change 5. Washington, DC. https://www.directives.doe.gov/directives-documents/400-series/0413.3-BOrder-B-chg5-minchg/@@images/file.
DOE. 2022. Hanford Lifecycle Scope, Schedule and Cost Report. Washington, DC. https://www.hanford.gov/files.cfm/2022_LCR_DOE-RL-2021-47_12-27.pdf.
___________________
7 M.J. Lilly, III, 2021, Cost of C-Train, Slide 22 in “X-333 Process Building Deactivation at the Portsmouth Gaseous Diffusion Plant, DOE Order 413 Appendix D,” presentation to the committee, March 19, available from the Public Access Records Office.
8 J. Reising, 2021, “On-Site Waste Disposal Facility at the Portsmouth Gaseous Diffusion Plant,” presentation to the committee, March 19, available from the Public Access Records Office.
GAO (Government Accountability Office). 2019. Nuclear Cleanup: Actions Needed to Improve Cleanup Efforts at DOE’s Three Former Gaseous Diffusion Plants. Washington, DC. https://www.gao.gov/products/gao-20-63.
GAO. 2022. Waste Isolation Pilot Plant: Construction Challenges Highlight the Need for DOE to Address Root Causes. Washington, DC. https://www.gao.gov/assets/gao-22-105057.pdf.
NASEM (National Academies of Sciences, Engineering, and Medicine). 2021. Effectiveness and Efficiency of Defense Environmental Cleanup Activities of DOE’s Office of Environmental Management: Report 1. Washington, DC: The National Academies Press. https://doi.org/10.17226/26000.
OMB (Office of Management and Budget). 2016. Preparation, Submission, and Execution of the Budget. Circular No. A-11. Washington, DC: Executive Office of the President. https://obamawhitehouse.archives.gov/omb/circulars_a11_current_year_a11_toc.
ANNEX 2.A: RECOMMENDATIONS FROM PHASE 1 REPORT
This annex provides verbatim recommendations from the committee’s Phase 1 report (NASEM, 2021).
Project Management Policies, Processes, and Procedures
Recommendation 4-1: The committee recommends that the Department of Energy (DOE) confirm, clarify, and expand DOE Order 413.3B to establish its applicability to all capital asset projects (not just construction and major instruments and equipment and certain cleanup projects) and all Office of Environmental Management projects, whether major systems projects or work carried out by a management and operating (M&O) contractor. The committee also makes the following specific recommendations regarding the Order as well:
- Pending the outcome of the National Nuclear Security Administration pilot project, reduce the threshold value for applicability of Order 413.3B from $50 million to $20 million;
- Continue applying the requirements of Order 413.3B to M&O contract work on capital asset projects—the latter including construction projects, major items of equipment and cleanup projects;
- Clarify the definition related to project performance found at Section 3c(4), point 3 to calculate performance on aggregate value and not number of projects; and
- Shift eligibility for project overruns, currently 10 percent per project, to be applied instead based on the aggregate value.
Recommendation 4-2: The Department of Energy should clarify Order 413.3B to incorporate best practices with respect to dispute prevention and resolution, which will be of growing significance as the Office of Environmental Management implements the end-state contracting approach. Sources for such best practices include the Construction Industry Institute.
Recommendation 4-3: The Office of Environmental Management should apply the requirements for project execution plans equivalent to those in Order 413.3B to those projects that are not formally managed under Order 413.3B.
Project Management Metrics
Recommendation 5-1: The committee recommends that as the Office of Environmental Management increases its project management (PM) and Office of Project Management responsibilities using indefinite delivery/indefinite quantity (IDIQ) contracts, it should share and compare best PM practices with others across the U.S. government. To implement this, the Department of Energy’s Office of Environmental Management should form a “Joint Task Force” or less formal cooperative structure with Naval Facilities Engineering Systems Command (NAVFAC) and other base realignment and closure (BRAC) and formerly used defense sites (FUDS) program management organizations.
Recommendation 5-2: The Department of Energy’s Office of Environmental Management (DOE-EM) should implement a modification to its Earned Value Management System that captures the project’s temporal status more clearly and explicitly. Specifically, DOE-EM should immediately require that a revised Schedule Performance Index, SPI(t), which is the ratio of scheduled time of work performed (STWP) and actual time of work performed (ATWP), be reported to accurately track schedule performance.
Recommendation 5-3: The Department of Energy’s Office of Environmental Management should explicitly include the percentage of cost overrun or underrun in the project success metrics dashboard, rather than the current “green/yellow/red” metric, to bring more transparency to cost performance.
Contract Structures
Recommendation 6-1: The Department of Energy’s Office of Environmental Management (DOE-EM) should establish well-defined, outcomes-based intermediate end states in its end-state (10-year cleanup) contracts. Any intermediate outcomes should have clear, measurable metrics to assess site-based (versus task-based) achievement of the defined end states. DOE-EM should report progress on these metrics across the portfolio of end-state contracts on a quarterly basis and such reports should represent a key DOE-EM performance measure.
Recommendation 6-2: The Department of Energy’s Office of Environmental Management (DOE-EM) should structure task orders on a scale that is appropriate for defining intermediate outcomes and award fewer individual tasks. DOE-EM should apply to such task orders the same management oversight as currently required for major systems projects exceeding $750 million in total cost.
Contract Management Metrics
Recommendation 7-1: To increase transparency in contractor performance evaluation, the committee recommends that the Department of Energy’s Office of Environmental Management should ensure that the contracts it issues for cleanup work (1) create a consolidated set of unambiguous “subjective” criteria for similar types of cleanup activities, and (2) use these criteria in the evaluation of all contract performance across its portfolio.
ANNEX 2.2: KEY TERMS AND CONCEPTS DISCUSSED IN THIS REPORT
Completion Contracts: A performance and incentive based contract, typically a cost-plus-incentive-fee, to complete environmental cleanup work at or below the projected life-cycle cost and schedule estimates. A completion contract provides the framework for the final cleanup of the site or a well-defined portion thereof. Completion contracts were successfully used by DOE-EM at Rocky Flats and Fernald.
DOE Order 413.3B: Program and Project Management for the Acquisition of Capital Assets (DOE, 2018) establishes procedures for project development and management, with the attendant controls and oversight; a hierarchy of project approval authority based on size of project; and tracking of project performance via management information systems.
Earned Value Management System (EVMS): A tool used by DOE-EM for measuring contract performance.
End State Contracting Model (ESCM): A contract management model developed by DOE-EM that is a form of IDIQ contract that is awarded to a single contractor for a 10-year order period with the ability to extend an additional 5 years. That contractor then has a sole-source status to work with DOE-EM to establish specific task orders (typically less than 2 years) within the order period. In the ESCM, DOE-EM does not define the end state of the full contract, but rather defines end states for each task order. NOTE: The committee considered “end state” to refer to the contract-level end state at the end of the 10-year period.
Indefinite Delivery/Indefinite Quantity (IDIQ) Contracts: A contract that provides flexibility to acquire goods or services over set periods of time (the order period). The contract must set a minimum amount of goods or services to be delivered over the order period and may set a maximum ceiling. The Federal Acquisition Regulation (FAR) states a preference for IDIQ contracts to be awarded to multiple contractors, enabling individual task orders within the IDIQ to be competed among the qualified IDIQ awardees.
OMB Circular A-11, Preparation, Submission, and Execution of the Budget (OMB, 2016): A circular issued by the Office of Management and Budget (OMB) that provides guidance to federal agencies on preparing and executing budgets.
Portfolio: An organized grouping of programs whose coordination in implementation enables an agency to achieve its missions. As used in this report, Portfolio refers to all DOE-EM programs and projects in their entirety.
Program: Functions or activities that an organizational element is authorized and funded by statute to administer and enforce. As used in this report, “program” refers to the collection of projects at each DOE-EM site, including capital projects as defined by OMB but currently performed as operating activities.
Program Management Improvement Accountability Act (PMIAA): Public Law 114-264, enacted in 2016, to establish standards, policies, and guidelines for program and project management in executive agencies of the federal government.
Project: A temporary effort with defined scope within a program to create products or services.
Project Assessment and Reporting System (PARS): A system used to track project schedule, cost, technical progress, and other metrics on a monthly basis for projects subject to Order 413.3B. Data submitted to PARS by contractors are used in the EVMS.
