5
Lessons Learned and Recommendations
NASA’s charge to the study committee is shown in the statement of task (see Appendix A). In this concluding chapter, the committee organizes its lessons learned, findings, and recommendations in a crosscut that clarifies their relationships to the elements of the task statement.
MEASURES OF SUCCESS FOR EV-I AND EV-M ENDEAVORS
As noted in Chapter 1 of this report, NASA initiated the Earth Venture (EV) program in part to provide frequent flight opportunities for high-quality, high-value, and focused Earth science investigations. The EV program has indeed offered increased opportunity for new ideas to be tested. Calls for proposals have until recently come out on a regular cadence, which provides potential principal investigators (PIs) some certainty in upcoming opportunities. The frequency of the calls for new proposals to develop new instrumentation and use them for science investigations has been aided by the imposition of fixed cost caps on the projects, and the recent break in the mission cadence is associated with a cost overrun on one mission (see the section “Earth Venture Foundational Principles: Implementation, Enforcement, and NonConformity” below). An additional objective of the EV program was to “provide opportunities to expand the pool of well-qualified PIs and project managers (PMs) for implementation of future NASA missions.” Here, the committee felt that EV PIs and PMs were indeed new in the sense that they had not previously occupied PI and PM roles in Earth science missions, but that the pool was not particularly diverse and could be further diversified.
Potential Changes to the Proposal, Selection, and Implementation Process (Chapter 4)
FINDING 4.2: The mission development experiences needed to put together full EV proposals currently limits the pool of PIs and program management entities for assembling competitive proposals; indirectly, it also limits the diversity of selectees.
RECOMMENDATION 4.5: NASA should ensure that Earth Venture announcements of opportunity include examples of contract deliverables with descriptions for various classes of instruments deployed in flight projects in order to provide the proposal teams with a better idea of reporting requirements that will facilitate budgeting and better inform contract negotiations.
RECOMMENDATION 4.6: To support diversification among potential principal investigators that may not have strong existing ties to NASA’s Earth Science Division or to NASA’s Centers, NASA’s Science Mission Directorate should call for “mission concept planning proposals” in its annual solicitations for Research Opportunities in Space and Earth Sciences.
EXPERIENCES OF PRINCIPAL INVESTIGATORS, PROJECT MANAGERS, AND INSTITUTIONS
The committee contacted PIs, PMs, and other key members of their project management teams to obtain their views and, indirectly, the views of their institutions. NASA also interviewed staff from the Centers and the Science Mission Directorate, particularly the Earth Science Division. Overall, the committee found the experiences of all parties are quite positive. The committee, however, was not able to identify and thus interview proposers who were not selected. While not a specific recommendation, the EV program is encouraged nonetheless to consider gathering statistics from all proposers and ask questions such as those found in Chapter 2 in order to improve the process in the future.
The consistent responses received by the committee could roughly be divided into comments related to PIs being hesitant to highlight the full potential of proposed new measurements lest they be penalized for concepts that may be perceived as speculative, and comments related to the onerousness of the review process during implementation.
The committee also found that the selection process itself, while structured after previous processes, may be favoring more incremental concepts over ideas that are less tested, and proposes some ideas for remedying this.
The committee found credence in the characterization of the mission reviews as particularly onerous. While EV missions have higher risk tolerance than NASA’s flagship missions, the review process itself may not be that different from that for larger missions.
Earth Venture Proposal, Selection, and Implementation Process (Chapter 2)
FINDING 2.1: The overall EV solicitation, evaluation, and selection process follows previously established guidelines. In general, the process is clear and methodical, the science and technical management review process is streamlined, and there is an appropriate assessment of the strengths and weaknesses of a proposal. Selected missions have tended to be “Category I” missions, meaning excellent or very good science and implementation strategies with low risk as assessed by the review of technical, management, and cost feasibility.
FINDING 2.2: By using the same selection process as previous missions in the Earth System Science Pathfinder Program, which has minimal feedback between science and technical risk portions of the assessment, the EV selection process appears to favor lower risk missions independent of potential advances that were not explicitly part of the baseline science mission.
FINDING 2.4: Many of the selected missions did not win on their first proposal. Instead, most of the missions benefited from the feedback they received and were able to win in subsequent solicitations. Having a regular and predictable solicitation cadence was critical to keeping teams together.
Potential Changes to the Proposal, Selection, and Implementation Process (Chapter 4)
RECOMMENDATION 4.2: NASA should keep the Earth Venture selection process as a one-step process.
FINDING 4.1: Classifying all EV missions as Class “D” and tailoring the project management regime to be consistent with the results from the proposal’s technical, management, and cost review can identify more specific risks to be addressed during a risk-based safety and mission assurance process.
RECOMMENDATION 4.3: NASA project management should implement risk-based safety and mission assurance principles and procedures by using NASA procedural document (NPR) NASA Space Flight Program and Project Management Requirements (NPR 7120.5F), Section 3.5 (Principles Related to Tailoring Requirements), to tailor management and review requirements to the particulars of the Earth Venture project and thereby reduce cost and management burden.
RECOMMENDATION 4.4: NASA should conduct an in-house analysis of the project management and review practices used in Earth Venture missions with the aim of streamlining processes and reducing budgetary and schedule pressures on these small, budget-constrained projects.
Evaluation of Enhanced Science and Applications (Chapter 4)
FINDING 4.3: Although new technologies or observation strategies often can lead to new discoveries and enhanced science, PIs appear unwilling to highlight any uncertain aspects of their proposal as it could be criticized as “undemonstrated” or dilute the primary science focus as there is typically not enough space within proposal page limits to cover multiple topics.
RECOMMENDATION 4.7: To facilitate selection of Earth Venture (EV) missions that are considered high risk but also have the potential to deliver an additional important science and/or a high-value applications product, NASA should request that EV teams include in their submission a supplemental document that highlights what a mission might accomplish beyond the stated baseline objectives. Given the difficulties in rating applications that have not been demonstrated, or the enhanced science that may be enabled by new types of observations, the Earth Science Division should evaluate the supplemental information and provide its assessment to the associate administrator of the Science Mission Directorate at the time of mission selection. The associate administrator would have the option of working with the appropriate program to fund the enhanced science or applications.
EARTH VENTURE FOUNDATIONAL PRINCIPLES: IMPLEMENTATION, ENFORCEMENT, AND NON-CONFORMITY
While the EV program has adhered to most of the foundational principles, the GeoCarb mission has been an exception in terms of adhering to the cost cap. This has had implications on future mission calls and deviation from the regular and predictable cadence that is essential in maintaining proposal teams if these are not immediately successful.
Changing Program Emphasis for Earth Venture Missions (Chapter 3)
FINDING 3.1: NASA’s interactions with the National Oceanic and Atmospheric Administration (NOAA) in the EVM-3 mission appear to be beneficial to both agencies. Following NASA’s selection of future EV missions, it may also be advantageous to allow mission enhancement opportunities relevant to other agencies.
RECOMMENDATION 3.1: To encourage consideration of a wider set of ideas benefiting Earth system science, NASA’s Earth Science Division should, in future Earth Venture solicitations, emphasize that science priorities of potential interest encompass the full range of science priorities in the 2017 National Academies of Sciences, Engineering, and Medicine decadal survey Thriving on Our Changing Planet: A Decadal Strategy for Earth Observation from Space.
FINDING 3.2: The greatly reduced budget for EV Instrument 6 (EVI-6) appears to be a direct consequence of GeoCarb’s significant cost overruns. This has a significant negative impact on the range and extent of the science that can be proposed. The science return of future EV missions, the benefits of a regular and predictable cadence for EV selection, as well as the ability of potential principal investigators to keep their teams together if attempting a second (or further) proposal, are threatened when previously selected missions are allowed to grow substantially beyond their planned cost cap.
RECOMMENDATION 3.2: NASA’s Earth Science Division (ESD) should not deviate from the foundational principles of the Earth Venture (EV) program. In particular, the ESD should establish and implement an effective process to strictly enforcing the cost caps established for EV missions.
POTENTIAL TRADES AMONG CADENCE, COST (INCLUDING COST CAPS), AND RISK IN IMPLEMENTING FUTURE EV MISSIONS
Preserving the cadence of EV mission solicitations is essential to maintain interest by the broad community.
Changing Satellite Platforms and Launch Opportunities (Chapter 3)
RECOMMENDATION 3.3: In future Earth Venture (EV) announcements of opportunity (AOs), NASA should consider discontinuing the distinction between EV Mission (EV-M) and EV Instrument proposals. NASA would then solicit proposals that provide the full mission architecture as is currently done with EV-M. The AO should list any specific hosting or launch opportunity that NASA offers to provide. EV teams would have the option to incorporate these opportunities in their proposals, accounting for their cost to ensure a level competition against proposals that do not take advantage of such NASA-provided accommodation(s).
Potential Changes to the Proposal, Selection, and Implementation Process (Chapter 4)
RECOMMENDATION 4.1: NASA should maintain a cadence of approximately one Earth Venture solicitation every 18 months to allow institutions to maintain proposal teams and ensure broad community engagement.
IMPLICATIONS OF THE CHANGING LAUNCH VEHICLE AND HOSTED PAYLOAD MARKETS
Missions going to GEO have had various degrees of difficulty in accommodation, while low Earth orbit (LEO) missions have either been accommodated on the International Space Station (ISS) or provided their own satellite bus (CubeSats). The ISS now appears to have relatively few upcoming opportunities. In addition, some of the CubeSat missions (e.g., TROPICS, PREFIRE) have had fairly unique orbit requirements that necessitated dedicated launch vehicles—albeit small ones. This blurs the distinction between EV-M and EV-I missions and leads to the recommendation that the distinction be dropped.
Earth Venture Proposal, Selection, and Implementation Process (Chapter 2)
FINDING 2.5: Delays in launching three EV-I missions and programmatic and budget issues with EVM-2 (GeoCarb) are the largest factors limiting the EV program’s science return to date. Multiyear schedule slips and projects growing substantially beyond their cost cap are inconsistent with the rationale for the EV program.
Changing Satellite Platforms and Launch Opportunities (Chapter 3)
FINDING 3.3: TROPICS and PREFIRE are examples of EV instrument selections that blur the lines between EV-I and EV-M. Both provide the instrument and the bus (CubeSats), and both need dedicated launchers because of their specific orbit requirements. In addition, the diminished prospects for hosting opportunities to geostationary orbits, and the anticipated loss of the ISS as a host platform after 2031, suggest this blurring of project elements is likely to persist.
ADDITIONAL LESSONS LEARNED FOR FUTURE EV-I AND EV-M
In addition to the finding and recommendations detailed above, the following findings apply to the interaction between the EV program and various other agencies and programs.
Earth Venture Proposal, Selection, and Implementation Process (Chapter 2)
FINDING 2.3: A Category III ranking occurs when proposals with high science ranking, but mission or technology risks raise concerns during the review of technical, management, and cost feasibility. Some Category III proposals have been successfully transferred to NASA’s ESTO, where solutions to specific issues were successfully demonstrated at much lower costs. The EV program has benefited from this as the TEMPEST instrument will be a component of the EVM-3 mission.
Changing Program Emphasis for Earth Venture Missions (Chapter 3)
FINDING 3.1: NASA’s interactions with NOAA in the EVM-3 mission appear to be beneficial to both agencies. Following NASA’s selection of future EV missions, it may also be advantageous to allow mission enhancement opportunities relevant to other agencies.
Evaluation of Enhanced Science and Applications (Chapter 4)
FINDING 4.4: Although science-driven missions may focus on both foundational science questions as well as more applied questions, it appears impossible to evaluate applications quantitatively when all proposals are not required to answer both sets of questions.
This page intentionally left blank.
