4
Meeting the EV-I and EV-M Broader Objectives
OBJECTIVES THAT APPEAR TO GUIDE EARTH VENTURE MISSIONS
As noted in the introduction to this report, NASA initiated Earth Venture (EV)-class in part to provide frequent flight opportunities for high-quality, high-value, and focused Earth science investigations. Reflecting on the EV program shortly after its initiation, a former head of NASA’s Earth Science Division (ESD) noted that creation of the new flight element was also driven by the perception that NASA’s base of new ideas and technologies was increasingly dependent on just a few NASA centers and contractors.1 According to this official, there was concern that fewer universities were able to engage with NASA, increasing the potential that innovative approaches to technology would not be incorporated into the mission portfolio. In addition, the interval between flight opportunities was becoming too long to sustain a reasonable pace of innovation, science advances, and community leadership. Thus, the community was concerned that NASA might lose its ability to attract new researchers and maintain its culture of innovation and cutting-edge technologies. Below, the committee comments on progress in achieving several of the specific goals of the EV program.
The EV program has offered increased opportunity for new ideas to be tested. As discussed in Chapter 2, calls for proposals have come out on a regular cadence, which provides potential principal investigators (PIs) some certainty in upcoming opportunities. The frequency of the calls for new proposals to develop new instrumentation and use them for science investigations has been aided by the imposition of fixed cost caps on the projects.
Although the EV program has greatly increased the pace of competition for new instrument and mission proposals, launches of several EV Instrument (EV-I) missions have been delayed, largely due to accommodation and launch vehicle issues on the NASA side (see Chapter 2). Launch provider and technical issues have also delayed EV Mission 2 (EVM-2). The cadence of execution of EV missions has not been regular, with only three launches from seven missions selected prior to 2016, one mission transferred to NASA
___________________
1 Michael Freilich, private communications, 2015.
Goddard Space Flight Center oversight, two instruments in storage, and one EV-I mission delayed but now ready for launch later in 2022.
The recommended cost caps for EV missions fall in the range of $100 million to $200 million, and these caps have been adhered to with some consistency except for GeoCarb, which was allowed to significantly exceed its cost constraints. As discussed in Chapter 3, this appears to have led to a substantially reduced allowance for EVI-6 proposals, which are now limited to Class D CubeSat implementations with a cost cap of $37 million. The committee is concerned that this lower cap may be approaching a level that will limit the potential to make significant science advancements.
POTENTIAL CHANGES TO THE PROPOSAL, SELECTION, AND IMPLEMENTATION PROCESS
Proposal Cadence
Over the past 10 years, NASA has, on average, issued an EV-I or EV-M solicitation every 17 months. This pattern should continue going forward, though the previous approximate 18-month cadence for EV-I calls will be replaced with alternating EV-I and EV Continuity (EV-C) calls every 18 months. The committee sees important value in maintaining an approximate 18-month cadence for EV announcements of opportunity (AOs). In addition to providing frequent and reliable mission proposal opportunities for the Earth science community, a regular cadence is critical to maintaining project teams if they are not selected in their first EV proposal. Feedback received by the committee suggests that debriefings provided by NASA to non-selected EV proposals provided important insight to help improve proposals for more competitive future bids.
RECOMMENDATION 4.1: NASA should maintain a cadence of approximately one Earth Venture solicitation every 18 months to allow institutions to maintain proposal teams and ensure broad community engagement.
Proposal Process: One Step or Two Steps?
Significant resources are needed to develop a competitive EV proposal. To reduce these resource requirements and as a way to facilitate the expansion of the pool of proposers, the committee analyzed the merits of a one-step versus two-step proposal process.
In committee interviews, PIs were asked if the current one-stage selection process should be expanded to a two-step process where the first step is a shorter, science-focused, proposal of intent. NASA would then choose two or three proposals for the second step, which would be a more complete proposal similar to the current one-step proposal. This would enable PIs to address critical deficiencies as well as receive feedback on potential competitiveness before submitting a full proposal during the second step. It would also enable the proposing institution to make more effective use of its limited proposal development resources and reduce the burden on both the PIs and the reviewers.
PIs, however, expressed a number of concerns with a two-step process, including its duration, duplication of effort, increased burdens, and increased costs. The postponement to Step 2 of a detailed discussion of mission implementation was also viewed negatively. The committee believes that these concerns are especially significant for the EV-M solicitations, where a two-step process with the current 4-year cycle could result in approximately 10 years from initiation to selection if a proposal is selected in the second proposal cycle.
RECOMMENDATION 4.2: NASA should keep the Earth Venture selection process as a one-step process.
Is the Process Correctly Identifying and Managing Mission Risk During the Selection and Early Implementation Process?
The NASA AOs for EV missions identify the documentation that guides the management procedures for all flight missions. These guidelines apply to all classes of missions with a wide variety of complexity and budgets. Although EV missions have very constrained budgets and schedules, all of them (even relatively small Class D missions) are required to comply with review requirements and deliverables that are not very different from those of larger missions. Indeed, a common complaint from EV PIs is the onerous number, scope, and cost of the documentation and review process. Moreover, the AO and the NASA procedural document (NPR) NASA Space Flight Program and Project Management Requirements (NPR-7120.5F)2 (and documents referenced within) do not present a clear idea about the number and scope of contract data requirements (CDRs) that would be expected during mission implementation. This complicates the budgeting for mission management during the preparation of proposals, and during negotiations it indirectly favors NASA internal management, which is familiar with the CDRs. Many CDRs may also complicate contract negotiations, and their cost reduces the resources available to the project. For example, TEMPO had 79 CDRs when the contract was issued. GLIMR had 108 and, given the expected cost to the vendor (Raytheon), the University of New Hampshire assumed responsibilities for formal submission of most of the CDRs. Negotiations took so much time that there was a late start to the project, which increased its cost because of inflation.
A typical implementation of NPR-7120.5F is overly complex for typical EV missions and not commensurate with their budgets. As an alternative, risk-based safety and mission assurance (SMA) principles and procedures could be implemented for Class C and D missions. Risk-based SMA has been defined as, “The process of applying limited resources to maximize the chance for safety and mission success by focusing on mitigating specific risks that are applicable to the project vs. simply enforcing a set of requirements because they have always worked.”3 Indeed, NPR-7120.5F Section 3.5 (Principles Related to Tailoring Requirements) presents a clear road map for tailoring requirements to the particulars of a flight project.
RECOMMENDATION 4.3: NASA project management should implement risk-based safety and mission assurance principles and procedures by using NASA procedural document (NPR) NASA Space Flight Program and Project Management Requirements (NPR 7120.5F), Section 3.5 (Principles Related to Tailoring Requirements), to tailor management and review requirements to the particulars of the Earth Venture project and thereby reduce cost and management burden.
FINDING 4.1: Classifying all EV missions as Class “D” and tailoring the project management regime to be consistent with the results from the proposal’s technical, management, and cost review can identify more specific risks to be addressed during a risk-based safety and mission assurance process.
RECOMMENDATION 4.4: NASA should conduct an in-house analysis of the project management and review practices used in Earth Venture missions with the aim of streamlining processes and reducing budgetary and schedule pressures on these small, budget-constrained projects.
RECOMMENDATION 4.5: NASA should ensure that Earth Venture announcements of opportunity include examples of contract deliverables with descriptions for various classes of instruments
___________________
2 NASA, NPR-7120.5F, https://nodis3.gsfc.nasa.gov/displayDir.cfm?t=NPR&c=7120&s=5F.
3 J. Leitner, Chief Safety and Mission Assurance Engineer, NASA GSFC, “Risk-Based SMA,” February 13-20, 2020, https://ntrs.nasa.gov/api/citations/20200000819/downloads/20200000819.pdf.
deployed in flight projects in order to provide the proposal teams with a better idea of reporting requirements that will facilitate budgeting and better inform contract negotiations.
Is the Pool of Principal Investigator and Project Managers Expanding?
One of the intentions of the EV program is to provide opportunities to expand the pool of well-qualified PIs and project managers (PMs) for implementation of future NASA missions. Whether the program is succeeding in this regard appears to center on the definition of what constitutes a new PI. The committee found that a majority of selected PIs have had significant roles in prior NASA missions, or they had led significant panels on behalf of NASA. While only 30 percent of the PIs are working at a NASA center, all EV missions/instruments science or instrument teams have participation at one of the NASA centers (with highest involvement coming from NASA GSFC and Jet Propulsion Laboratory) or have NASA centers in the management teams. Most PIs are well-established scientists, with an average of 27 years in a professional career at time of selection.
Diversity in terms of gender and race needs particular attention. For example, only one of the EV-M and EV-I missions has had a female PI. This lack of diversity, however, is not limited to the EV program, and the root of the problem is much deeper and broader than the EV program. Across all of SMD, during the period from 2001 to 2017, only 30 of 301 PI-led mission proposals had female PIs, and one division (the Planetary Science Division) accounted for 26 of the 30 female PI applicants.4,5
Actions by NASA to increase diversity in the EVs is warranted; the committee notes that focused efforts to examine and address diversity, equity, and inclusion among SMD PIs are ongoing.6 The effectiveness of NASA actions will be greatly enhanced in the longer term by societal progress in addressing wide-ranging factors that impede diversity in science, technology, engineering, and mathematics (STEM) in education and professional careers.
FINDING 4.2: The mission development experiences needed to put together full EV proposals currently limits the pool of PIs and program management entities for assembling competitive proposals; indirectly, it also limits the diversity of selectees.
One approach to improve engagement of new PIs would be to use a ROSES call for “mission concept planning proposals” that would support PIs over a 2-year period to develop and refine their EV concepts as well as to forge scientific, technical, and management partnerships that would be engaged in a full EV proposal. Absent new funding, resources devoted to this effort would need to be balanced against the value of other ROSES calls and other ESD activities. Selection criteria for mission concept planning proposals could be modeled after other comparable efforts, such as NASA’s New Investigator Program. Other federal agencies have also used similar processes successfully to nurture the community’s capabilities to propose and manage complex awards. For example, the National Science Foundation uses planning grants for its Industry University Cooperative Research Centers and its Engineering Research Centers. These centers issue grants that span 5 to 10 years and involve multiple institutions and a wide range of stakeholders.
___________________
4 M. New, 2018, “Diversity and Inclusion,” NASA Science Mission Directorate, https://science.nasa.gov/science-red/s3fs-public/atoms/files/New_Diversity&Inclusion_APAC_April2018.pdf.
5 Data on the race of the PIs were not available.
6 In 2022, the National Academies of Sciences, Engineering, and Medicine will publish a study report with recommendations to increase diversity, inclusion, equity, and accessibility in the leadership of space mission proposals submitted to SMD competed space mission programs: Advancing Diversity, Equity, Inclusion, and Accessibility in the Leadership of Competed Space Missions.
RECOMMENDATION 4.6: To support diversification among potential PIs that may not have strong existing ties to NASA’s Earth Science Division or to NASA’s Centers, NASA’s Science Mission Directorate should call for “mission concept planning proposals” in its annual solicitations for Research Opportunities in Space and Earth Sciences.
Broadening the scope of science objectives of interest to the EV program (see Recommendation 3.1) could also expand the pool of potential PIs by drawing in PIs with expertise areas outside the scope of prior AOs.
EVALUATION OF ENHANCED SCIENCE AND APPLICATIONS
To be selectable, EV proposals must offer the potential for a significant science return for missions that can be executed within rigid cost and schedule constraints. Thus, the proposers must convince NASA reviewers that all of the risks (science, technical, cost, and schedule) are both understood and manageable. In interviews with the committee, mission PIs have noted that these requirements led them to include less in their EV proposals than they thought their missions, if fully successful, could accomplish.
Although one of the goals of the EV program is to drive innovation, having only threshold and baseline science objectives, without allowing for more speculative discussions, may disadvantage truly new observations. Selection committees, however, cannot be asked to quantitatively evaluate possible new scenarios that are not part of the proposal itself.
CYGNSS, GEDI, and ECOSTRESS are examples of EV missions with the potential to demonstrate significant science and/or applications beyond what could be judiciously described and justified within a single proposal. CYGNSS was proposed as a mission for ocean surface winds, but the GPS reflections capability was also known to have potential uses over land for terrestrial hydrology and ecosystems science and applications research. The ECOSTRESS team proposed to use its infrared imaging instrument to study water use by vegetation and associated plant/ecosystem health via temperature-based estimates of evapotranspiration. However, the mission technology has applications, among others, to wildfires, geology, and sources of urban heat. In addition to these examples, the upcoming EMIT mission will use its hyperspectral visible imaging spectrometer to characterize terrestrial source regions of atmospheric dust, and the same technology can be applied to a wealth of other science and applications studies (e.g., vegetation, ecosystems, other geological topics).
FINDING 4.3: Although new technologies or observation strategies often can lead to new discoveries and enhanced science, PIs appear unwilling to highlight any uncertain aspects of their proposal as it could be criticized as “undemonstrated” or dilute the primary science focus as there is typically not enough space within proposal page limits to cover multiple topics.
For many years, the value of potential applications of Earth observing science data were not considered in NASA’s consideration of EV proposals. When queried about success criteria, several PIs emphasized that the definitive measure of the success of their mission was the uptake of their data for user applications. As GEDI PI Ralph Dubayah stated, “GEDI’s success may ultimately be measured not by the number of publications, but rather by how widely and how often the data sets are used.”
In 2016, NASA’s then-director of ESD, Michael Freilich, released the ESD Directive on Project Applications Program, which integrated applications into all Earth science program activities. Accordingly, the recent EVM-3 and EVI-6 AOs required that proposals, “articulate, to the extent possible, potential innovative and practical applications of the mission data and a plan to engage those users that would use mission
data to inform their decisions and actions.” Proposers referred to the 2016 directive for further guidance. The AOs made it clear that proposals without significant applications needed to describe why applications are not appropriate in their proposed mission. All Earth science missions that are still functional at the end of their primary operational phase may undergo a senior review process at NASA Headquarters to assess whether an extended operational phase should be implemented to make additional progress in advancing applications or to achieve additional science objectives.
FINDING 4.4: Although science-driven missions may focus on both foundational science questions as well as more applied questions, it appears impossible to evaluate applications quantitatively when all proposals are not required to answer both sets of questions.
RECOMMENDATION 4.7: To facilitate selection of Earth Venture (EV) missions that are considered high risk but also have the potential to deliver an additional important science and/or a high-value applications product, NASA should request that EV teams include in their submission a supplemental document that highlights what a mission might accomplish beyond the stated baseline objectives. Given the difficulties in rating applications that have not been demonstrated, or the enhanced science that may be enabled by new types of observations, the Earth Science Division should evaluate the supplemental information and provide its assessment to the associate administrator of the Science Mission Directorate at the time of mission selection. The associate administrator would have the option of working with the appropriate program to fund the enhanced science or applications.
