Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
O Independent Technical, Risk, and Cost Evaluation The Astro2020 decadal survey evaluated the technical, programmatic and execution risks of large ground and space-based projects in numerous ways. This evaluation was directed by the relevant Program Panel, and included multiple layers of review by the panel, as well as an Independent Technical, Risk and Cost Evaluation (TRACE), performed by a contractor (The Aerospace Corporation). This full level of independent review was undertaken for all National Science Foundation (NSF) Major Research Equipment and Facilities Construction (MREFC)-scale projects selected by the Program Panels to be of primary interest, as well as for some version of all of the NASA flagship concepts (in the case of the Large UV/Optical/IR Surveyor (LUVOIR) and Habitable Exoplanet Observatory (HabEx), multiple implementations were studied, and the EOS-1 program panel selected those of primary interest). For the NASA flagships, in addition to the project cost estimates, NASA performed a Large Mission Concept Independent Assessment Team (LCIT) evaluation, independent of the projects, which, in addition to the project estimates, was considered as input to the panels. The panels included subject-matter experts in technology, management, and instrumentation, and this expertise guided the process, providing key feedback and iterative steps between both the projects and the independent contractor. Information flow is captured in Figure O.1. The Program Panel process is documented in the panel reports and summarized in the main text of the report. The purpose of this appendix is to describe the process through which the Aerospace Corporationâs independent TRACE evaluations were implemented and how they were managed and considered by the survey. The TRACE provided broad guidance to the Program Panels in the form of a forward-looking assessment of potential technical risks and likely cost and schedule boxes associated with a mission concept, but not definitive cost nor schedule estimates. FIGURE O.1 Relationship between projects, program panels, aerospace, and the steering committee. PREPUBLICATION COPY â SUBJECT TO FURTHER EDITORIAL CORRECTION O-1
The decadal independent cost and technical risk assessment process has substantially evolved and improved since its earliest application, as well as its implementation for the New Worlds, New Horizons (NWNH) 2010 astrophysics decadal survey. The current methodology, TRACE, is anchored by successful implementations of its heritage risk analysis methodology, the Cost and Technical Evaluation (CATE) process, for a total of four decadal surveys, including NWNH. It is also anchored in the processes developed in support of the independent estimating of both NASA and Department of Defense projects. The details of the Aerospace Corporation process have been captured in multiple reports and presentations, including The Space Science Decadal Surveys-Lessons Learned and Best Practices1 and CATE Overview, Astro2020 Large Mission Concepts.2 Highlights of the TRACE process included using Aerospace Corporation proprietary models and databases, validated through multiple processes, including comparisons to historical implementations, discussed below. Threats and risks were evaluated using Monte Carlo simulations. Each project was assessed independently, regardless of maturity, and brought to a baseline 70 percent probability of success through application of reserves for both cost and schedule to assure equal programmatic evaluation criteria could be applied. In addition, NASAâs support of the development of high-quality mission concepts significantly assisted the TRACE. The Astrophysics Science and Technology Definition Teams (STDT) in preparation for this decadal survey provided well-developed, compelling, and executable mission concepts with high- fidelity estimates. The Astro2020 decadal survey committee used best practices from NWNH and implemented lessons learned from other completed decadal surveys.3,4 ï· Communicating, reviewing, and coordinating directly with Program Panels to improve the information available to the panels for critical decision-making. ï· Using project-provided descoping or rescoping data to assess mission costs and technical risks. ï· Scheduling to preclude TRACE from becoming the pacing schedule item and allowing for the TRACE results to further inform the steering committee deliberations and decision-making. ï· Using varied budget and funding projections, nominal and optimistic, to frame project execution opportunities and challenges. ï· Factoring costs of current and near-available launch vehicles into mission costs. ï· Using experts, as needed, to augment the Aerospace Corporation independent assessment process, after gaps were identified. ï· Identifying liaisons to serve as go-betweens the panels, the committee, and the Aerospace Corporationâs TRACE team. ï· Early identification of the role of TRACE in panel decisions and deliberations. ï· Presentations of final TRACE results directly to the Program Panels. The overall process is captured in Figure O.2. Science priorities were provided to the Program Panels as input to the evaluation process. An early assessment of project science, scope, technical risks, costs, possible descopes, and associated risks were used as input. The Program Panels used requests for 1 National Academies of Sciences, Engineering, and Medicine (NASEM), 2015, The Space Science Decadal Surveys: Lessons Learned and Best Practices, The National Academies Press, Washington, D.C. 2 Â D. Emmons, 2016, âAerospace CATE Overview: Astro2020 Large Mission Concepts,â teleconference presentation to the LUVOIR Science and Technology Definition Team on September 20, 2016, https://asd.gsfc.nasa.gov/luvoir/events/events/telecons/2016-09-20/emmons.pptx.Â 3 National Research Council, 2013, Lessons Learned in Decadal Planning in Space Science: A Summary of a Workshop, The National Academies Press, Washington, D.C. 4 NASEM, 2015, The Space Science Decadal Surveys: Lessons Learned and Best Practices, The National Academies Press, Washington, D.C. PREPUBLICATION COPY â SUBJECT TO FURTHER EDITORIAL CORRECTION O-2
information (RFI) to gather data on projects using a two-tiered process. Projects of greater maturity were sent RFIs reflecting more detailed data (RFI1). Projects of lesser maturity were sent an alternate, lesser detailed RFI (RFI2). Outcomes were presented to the Program Panels using standardized formats. Results included technical risk ratings, power and mass margins, and other risk data. Cost histograms were presented to compare costs developed using a range of independent cost estimating methods, including classical estimating methods, risk estimating programs (e.g., SEER, MICM, NICM, MOCET, FRISK, etc.5), and the contractorâs proprietary analytics suite. The key analogies and historical projects used in the analyses for the various design elements, subsystems, instruments, construction elements, etc., were also reviewed by the Program Panels to assess appropriate application and allow for programs of varied maturities to be brought to a consistent baseline. TRACE products provided to the Program Panels during the process flow are shown in Figure O.3. FIGURE O.2 Technical Risk and Cost Evaluation (TRACE) process interactions. 5 A subset of classic estimating tools used by TRACE: ï· SEER (Systems Evaluation and Estimation of Resources), a knowledge-based estimation cost evaluation and trade-off tool suite by Galorath, Inc. ï· MICM (Multi-variable Instrument Cost Model), by NASA Goddard Space Flight Center. ï· NICM (NASA Instrument Cost Model), a probabilistic cost estimation tool developed by NASA containing a system model, subsystem model, and a database search engine. ï· MOCET (Mission Operations Cost Estimation Tool), a Phase E estimation tool jointly developed by The Aerospace Corporation and the NASA Science Office for Mission Assessments. ï· PCEC (Project Cost Estimating Capability), a parametric cost model developed and maintained by NASA. ï· FRISK (Formal RISK method), an analytical, rather than Monte Carloâbased, risk analysis method. Typical implementation includes establishing total cost distribution for a design based on WBS (Work Breakdown Structure). PREPUBLICATION COPY â SUBJECT TO FURTHER EDITORIAL CORRECTION O-3
FIGURE O.3 TRACE process and products. Space concepts benefitted from the extensive past program database available to Aerospace Corporation allowing for numerical analytical methods to be effectively and consistently applied across a range of concepts. However, the unique aspects of ground observatories continued to provide challenges to the TRACE. Aerospace made significant efforts to improve the TRACE process for ground programs including adding additional historical analogy data and updating their analyses to include current applicable facilities. Ground TRACE evaluations limitations included the exclusion of operations costs, focusing only on design and construction, and the exclusion of funding threats. All gaps were addressed by the Program Panels, engaging additional unpaid consultants where necessary to augment the database and evaluate program specifics. The TRACE processes forecasted final project costs, timelines, and critical paths, allowing âunknown unknowns,â or âthreats,â to be captured, typically producing more conservative values, e.g., manufacturing development and scale-up, increased schedule to reflect demonstrated development timelines, added costs to capture mass contingencies, increased costs, and time to reflect required launch vehicle capacity and availability, increased costs and schedule to capture historic execution and learning curves associated with specific technologies, etc. Once overall costs and schedules were estimated, probabilistic assessments of cost probability and schedule probability were calculated and provided to the Program Panels using classic S-curves comparing project estimates to 70th percentile probabilities. The role of the TRACE data as input to panel decision making is presented in the individual Program Panel reports. The projected cost growth range for large space and ground mission concepts in real year dollars is shown in Figure O.4. It should be emphasized that the âIncreased projected costâ in the figure reflects the differences between the TRACE total mission cost estimates at 70 percent probability and the project total mission cost estimates. The differences in projected growth assigned to ground versus space concepts were generally attributable to technology maturity gaps and the demonstrated cost and scope growth challenges faced by early phase projects. The impact of early phase programmatic risk on anticipated program cost growth is shown in Figure O.5. (Increased technical risk is associated with higher projected cost growth.) An examination of the aggregate TRACE results, as a percentage of the total project value, were compared to the open technical risks. Every project that received a TRACE evaluation was assessed for technical risk by evaluating both mitigation plans and historical risks. Figures of merit (FOM) were used to normalize the data and highlight relationships. FOM formulae are provided in Box O.1. In addition to the general trend PREPUBLICATION COPY â SUBJECT TO FURTHER EDITORIAL CORRECTION O-4
of higher technical risks driving forecasted cost growth, a distinct difference was apparent between the inherent technical risks of the space mission concepts and the ground concepts. FIGURE O.4 TRACE projected cost growth (70 percent probability level). FIGURE O.5 Correlation of TRACE âlook forwardâ to technical risk. PREPUBLICATION COPY â SUBJECT TO FURTHER EDITORIAL CORRECTION O-5
BOX O.1 Figures of Merit Calculations To assess the extent to which the calculated program execution threats (TRACE) were correlated to open technical risks across all programs, figures of merit (FOM) were developed. TRACE values were normalized as a percentage of project values: TRproject = ((TRcost Ã 100)/ Pcost) â 100, where ï· TRproject = TRACE percent increase over total project submitted cost ï· TRcost = TRACE estimate for complete project in real year dollars ï· Pcost = Project estimate for complete project in real year dollars As an example, if a project cost totaled $1,000 million in real year dollars and the TRACE 70 percent probability value was $1,100 million in real year dollars, TRproject = 10. Because the TRACE process captures and monetizes risks, all values of TRproject were positive. Technical risks were normalized using a FOM capturing the number of large open technical risk categories, the technology readiness level (TRL) of the open technology, and the time (in years) needed to mature the technology as submitted by the program: TechFOM = (Rhigh Ã (TRL3 Ã TRL3time + TRL4 Ã TRL4time + TRL5 Ã TRL5time + TRL6) + (Rmedium-high Ã (TRL3 Ã TRL3time + TRL4 Ã TRL4time + TRL5 Ã TRL5time + TRL6) + (Rmedium Ã (TRL3 Ã TRL3time + TRL4 Ã TRL4time + TRL5 Ã TRL5time + TRL6) + (Rmedium-low Ã (TRL3 Ã TRL3time + TRL4 Ã TRL4time + TRL5 Ã TRL5time + TRL6) + (Rlow Ã (TRL3 Ã TRL3time + TRL4 Ã TRL4time + TRL5 Ã TRL5time + TRL6), where ï· TechFOM = Technology Risk FOM ï· Rhigh, Rmedium-high, Rmedium, Rmedium-low, and Rlow = Technology risk rating assigned by Aerospace = Number of major program risks in this risk category Ã Risk multiple, where â High risk multiple = 5 â Medium-high risk multiple = 4 â Medium risk multiple = 3 â Medium-low risk multiple = 2 â Low risk multiple = 1 ï· TRL3, TRL4, TRL5, and TRL6 = number of technology categories identified by the project at these TRL levels Ã TRL risk multiplier, where â TRL 3 multiplier = 9 â TRL 4 multiplier = 3 â TRL 5 multiplier = 2 â TRL 6 multiplier = 1 ï· TRL3time, TRL4time, and TRL5time = total length of time, in years as defined by the projects for a technology to mature to the next level. As an example, if a project defined two major development areas, one currently at TRL 6 and rated a medium risk by Aerospace Corporation, and the other at TRL 3 and rated a medium-low risk by Aerospace Corporation, but requiring 2 years each to reach TRL 4, TRL 5, and TRL6, the project TechFOM = 61. PREPUBLICATION COPY â SUBJECT TO FURTHER EDITORIAL CORRECTION O-6