2

Understanding the Mission Formulation and Proposal Process

In its 2020 Science Strategy, the National Aeronautics and Space Administration (NASA) Science Mission Directorate (SMD) states: “SMD seeks to discover the secrets of the universe, to search for life, and to protect and improve life on Earth” (NASA 2020b). NASA SMD’s flight missions are essential to addressing critical scientific needs stated in the NASA’s strategic plans, and exist as competed principal investigator (PI)-led missions and very large, “flagship missions” that are led through NASA centers. Specifically, PI-led missions exist in a larger context of a “balanced portfolio” that encompasses a broad range of scientific investigations that includes flight missions, research and analysis, technology development, and applied/fundamental science. These projects together support NASA’s strategic plans, and the relative balance across these efforts is informed by the National Academies of Sciences, Engineering, and Medicine through its decadal surveys and is responsive to Administration priorities and direction from Congress.

PI-led missions have been an increasingly important element of the balanced portfolio over the past several decades in the four science divisions in NASA SMD that conduct spaceflight missions: Astrophysics, Earth Science, Heliophysics, and Planetary Science. The fifth division, Biological and Physical Sciences, does not generate satellite missions. Competed opportunities for NASA-funded investigations vary in scope, complexity, and cost from ~tens of millions of dollars for instruments or Small Innovative Missions for Planetary Exploration (SIMPLEx) investigations to over $1 billion for New Frontiers or Astrophysics Probes investigations (see Table 2.1 in the next section). The processes whereby NASA advertises competed opportunities—announcements of opportunity (AOs)—and makes selections based on proposals developed by aspiring PIs have evolved significantly and quite effectively since the 1970s. In its 2006 report Principal-Investigator-Led Missions in Space Science, the National Research Council reported that the space science community sees a lot of scientific value in PI-led missions as a complement to NASA flagship missions (NRC 2006).

Investments in a flight investigation are large, requiring contractual vehicles¹ (or other appropriate vehicles for NASA centers or other government entities) that provide enforcement and oversight mechanisms to ensure that the investment is well managed and can deliver on its promise. The PI on each selected proposal is assigned responsibility and authority for the success of the investigation, through a contract between NASA and the PI’s institution. The requirements are defined in detail in a NASA Supplement to the Federal Acquisition Regulations (FAR) and are presented in summary form in this report. The 2006 report on PI-led missions broadly examined

___________________

¹ A contractual vehicle is the method the government uses to buy goods and services.

PI-led missions, from the process of proposals to selection to the execution of missions (NRC 2006). The recommendations in that report addressed the factors that contributed to the success and challenges of PI-led missions and provided some recommendations related to the selection process, concept studies, selection criteria, team development, funding profiles, International Traffic in Arms Regulations (ITAR) compliance, and termination reviews. The scope of this report is focused on diversity and inclusion in the leadership of competed space missions and will not discuss the execution of missions.

The mission proposal process leading up to selection can be divided into three stages: (1) the development of an idea into a mission concept (concept development); (2) proposal development and submission; and (3) in the case of two-step mission proposals, preparation of a concept study report (CSR) for concepts that are selected through the Step 1 process. It is important to note that proposal preparation for competed missions begins far in advance of an AO released by NASA, typically 1 or more years before the release. Once a proposal has been developed and submitted, proposal evaluation and review of CSRs are almost entirely under NASA’s direct control. The overall process, at a high level, is captured in Figure 2.1. NASA follows strict procedures specified in federal regulations, time tables, and rules that are generally transparent and well documented. In contrast, concept development, which has been referred to as “the competition before the competition,” is idiosyncratic, organic, opaque, and often personality-driven.

This chapter discusses the current proposal process for competed space missions as understood by this committee through its examination of disparate sources of evidence and testimony from members of the space science community. In the sections to follow, the committee begins with a look at concept development, team formation, and proposal development, and then the NASA proposal evaluation process in more detail.

FORMING A TEAM AND DEVELOPING A PROPOSAL

Concept and Technology Development

Concept development, which begins years before a draft AO or other solicitation is released by NASA, is effectively a competition before the competition that starts when a proposal is submitted. In the concept development phase, an aspiring PI formulates and matures a concept for a mission and assembles a team of key co-investigators (Co-Is)—critical scientific partners at other institutions—and supporting institutions that bring a range of expertise to the development and execution of the mission. Teams are formed to develop early-stage ideas into more mature concepts, as well as to build mission designs and prepare complete proposals from mission concept studies. The team formation is intertwined with the concept development, where often a scientist or engineer, or a small group of scientists or engineers, conceive of a new measurement or mission. As the concept becomes more mature, people with needed skills or expertise are brought onto the team. The process of going from a new concept to a mission design often takes many months or years. Many ideas are proposed numerous times before they are selected, so the team may evolve and develop during the iterations. A typical team composition during the early phases of concept development may include science experts, mission designer, project systems engineer (PSE), project manager (PM), instrument lead, and ground data system engineer. During concept development, a team may engage in technology development, and this may be supported by NASA instrument or concept development grants, or it may be supported by institutional investments. In this stage, the team also begins generating proposal material in anticipation of a future AO, and the PI and others may publicize the mission concept at conferences and in published papers and/or make presentations to appropriate NASA and other committees. After the release of an AO, the teams refine and submit the proposal, but the concept development stage can easily stretch over 1 year and often extends over many years and across multiple AOs.

Since the conception of a mission idea, formation of a team, and maturation of a concept to be proposed precede the NASA release of an AO, several frameworks have been conceptualized to map the proposal process from idea to fruition. One example is the concept maturity level (CML) framework (Wessen et al. 2013). The CMLs span from level 1, a new idea sketched on a napkin, to level 8, an actual project (see Figure 2.2), and brief descriptions of CMLs 1 through 6 are as follows:

• CML 1 Sketch on a Napkin—The science questions have been well articulated, needed science observations identified, rough sketch of mission concept of operations and objectives, and an assessment of technology readiness level.
• CML 2 Initial Feasibility—The idea is expanded and questioned on the basis of feasibility, from a science, technical, technology readiness, and programmatic viewpoint.
• CML 3 Trade Space—Exploration done of the science objectives and architectural trades to explore the relationship between science return, cost, and risk.
• CML 4 Point Design—A specific design and cost that returns the desired science has been selected within the trade space. Subsystems trades have been performed.
• CML 5 Baseline Concept—Implementation approach has been defined including partners, contracting mode, technology maturation, integration and test approach, cost and schedule. Appropriate for Step 1 proposals.
• CML 6 Integrated Concept—Expanded details on the technical, management, cost and other elements of the mission concept have been defined and documented. A NASA Step 2 CSR is at this level of maturity.
• CML 7 and 8—Correspond to selected or assigned missions in early implementation, preparing for design reviews.

The effort, time, and financial resources to take an idea from CML 1 to CML 6 are significant (see Annex 2.A for more complete details and typical resource requirements to achieve each level). Concepts typically need to mature to at least CML 5, “Baseline Concept,” to be further developed into a mission proposal. The concept maturation process can be non-linear, where trade spaces are revisited after a point design is explored, or technology

challenges require revisiting the trade space. A great degree of overlap exists between the CMLs and the NASA mission proposal process (see Figure 2.2), the latter beginning after the AO is released and continuing beyond proposal submission. Figure 2.2 also shows the relationship of CMLs to the NASA project phases that encompass the lifetime of a mission. The standard NASA project phases are listed in Box 2.1, although the scope of this report is primarily related the Pre-Phase A through Phase B of the NASA framework.²

The steps preceding proposal development include feasibility studies, trade space studies, and full conceptual design (or point design). These steps culminate in several specific mission concept items such as a notional concept of operations plan, an instrument technology readiness level (TRL) discussion, and science traceability including high-level science goals traced down to instrument requirements and then capability. These steps may also include other items like securing a spacecraft provider; plans for subsystem testing, budget and scheduling; and establishing a management structure. Taken together, the progress from CML 1 to CML 5 requires extensive resources—team formation, funding for feasibility and trade space studies, engineering and scientific analysis, and often team reviews as a baseline concept is refined. Thus, concept development alone requires an institutional investment and significant effort by a science team, a core group of engineers, a PSE and PM, and financial and budget analysts. It is clear that a significant amount of work is required prior to the release of an AO by NASA, and this can be thought of as another form of a competition. It is also important to note that while the CMLs depicts a real-world process of how concepts become missions, they are not meant to be an exact description of the steps required to be taken. They are also not a formal NASA scheme like project phases and TRLs, which are discussed later.

Another aspect of development that occurs well before proposals are submitted is technology development. Often new measurement capabilities are enabled by new and advanced technologies in instrumentation. NASA expects the technology included in mission proposals to be sufficiently mature, or at high TRL. NASA’s defined TRLs range from 1 to 9, where TRL 1 is technology in an early development stage, and TRL 9 is flight proven through successful mission operations (NASA 2012). In advance of including new technology in proposals, teams may be investing and developing the new technology to move them to the appropriate maturity levels. This is an area where experience and familiarity with the technology development, resources, and pathways may be a challenge to new PIs. There are many NASA proposal opportunities specifically focused on technology development, some with specific TRL development objectives. Funding levels range from $200,000 to $300,000 per year to >$1 million per year.

Becoming a PI

Becoming a PI or Co-I on a winning proposal is typically a long training and development process for the investigators as well as the proposing team as a whole. The PI’s responsibilities range from concept formation and maturation, leadership during proposal development, and if selected, responsibility for the overall execution of the mission. The resources required for this pursuit in terms of dollars and time are substantial, therefore creating a challenging environment for investigators, particularly new ones with many expectations as listed below:

1. Defining, building, and leading the team well prior to any AO;
2. Being aware of AOs, their release and cadence, and recognition that the process starts well in advance of the release of the AO;
3. Developing a compelling scientific investigation concept that is responsive to priority questions in National Academies decadal surveys and other NASA strategic needs;
4. Securing committed collaborations among investigators to refine and pursue the concept;
5. Defining an instrument and/or suite optimized for the pursuit of the investigations;
6. Identifying and securing a partner institution experienced in putting together winning proposals, knowledgeable of policies and requirements, and that is willing and able to invest institutional bid and proposal (B&P) resources in the proposal;

___________________

² See “NASA Program/Project Life Cycle,” Section 3.0 in NASA (2019).

7. Dedicating the time and expertise required to develop, review, and refine a proposal that is compliant, compelling, and timely;
8. Taking ownership and accountability for the investment in the proposal and for the technical and programmatic success of the investigation if selected; and
9. Demonstrating strong perseverance and a commitment to winning.

Mission teams are formed to provide the scientific and technical skills and knowledge to write a winning proposal and implement a successful mission. Individuals in these roles are sourced from NASA centers, industry, nonprofits, universities, and university-affiliated research centers—often a web of partnerships is required to fill all roles effectively. Many PIs or teams submit a number of proposals before winning (Zurbuchen 2019). As teams can form around the socialization of ideas often years before an AO, this could translate to potentially more than a decade spent championing an idea, proposing and re-proposing, before a mission proposal is awarded. This lengthy process limits participation of earlier career scientists and contributes to shaping the pool of potential PIs.

Finding: The expectations of the principal investigator of a NASA competed mission are significant and wide ranging, from science expertise to assembling a team, shepherding proposal development, and accepting ownership and accountability for the technical and programmatic success of not just the proposal, but the implementation of the mission.

Team Formation and Proposal Development

Team formation as part of concept and proposal development involves individuals (PI and Co-Is) and institutions, typically the PI’s home institution as host, a managing institution (often but not always a NASA center), an industrial spacecraft provider, and possibly one or more instrument providers. These institutions must commit to support the proposal effort itself as well as the subsequent implementation and science phases (NASA Phases A-E, concept development through operations, Box 2.1).³ Large investments are required for competed space mission proposals of any size. These initial investments are used in proposal development, idea maturation, and trade space analysis. According to the Laboratory for Atmospheric and Space Physics (LASP) at the University of Colorado Boulder, and others an estimated $2 million to $3 million is typical for the development of a Discovery or New Frontiers mission proposal.⁴ These commitments are not easily secured, both because of the up-front financial investment and the uncertainty of success inherent in the competition. Thus, decision makers at each institution find themselves in a situation where they must choose which PIs to support, if any; hence the “competition before the competition.” This level of initial investment, before any return is even possible, represents a sizable barrier to potential PIs with differing impacts and ability to secure the resources.

Finding: The process of maturing a mission concept into a complete and feasible mission design and complete proposal requires large investments, often years before proposal submission, of time, money, and expertise. Access to the significant resources required for this process are not uniformly available to aspiring PIs.

Additionally, potential PIs must also be familiar with the organizations they may want to partner with and often have to navigate and adjust to a range of organizational cultures. There are no universal organizational processes for team formation and concept development; consequently, there are clear differences across institutions regarding access to mission design laboratories, flight dynamics resources, hardware and spacecraft bus design knowledge, proposal support, and scientific expertise that can disproportionately affect how teams are formed and who leads or even joins those teams. Respondents in the commissioned qualitative study of the experiences of mission proposers, cited networking as a primary means of proposal team formation, where factors such as institutional affiliation and personal reputation impacted how teams were assembled, as they shaped differential access to information,

___________________

³ More on NASA mission phases can be found at NASA (2019).

⁴ See more on mission costs in Table 2.1.

timelines, and processes. These potential inequities are not easily addressed, but they could be partially mitigated by improved transparency throughout the process. Procedural transparency in both the criteria for inclusion in the team, as well as in evaluation of potential team members, would allow for less reliance on networks and other social and knowledge capital, and better align decision-making about inclusion in the team with pre-specified rationale rather than post hoc justifications. Such transparency could include stated criteria of need and expertise requirements that are shared with potential team members prior to selection; making the assessment rubrics publicly available; and communicating directly with those ultimately not chosen about how to strengthen their chances in the future (Bielby 2000; Reskin 2003; Stainback et al. 2010). Chapter 5 discusses additional aspects of bias and the challenge of access to professional networks in more detail.

The greatest barriers can be at a single institution: who decides which individuals lead a particular mission concept? And whose ideas are financially supported? Through the testimony of several individuals in the space science community, the committee gained insight into the large variations that occurs surrounding these questions from institution to institution.

The committee heard from a range of organizations that serve as instrument builders, spacecraft providers, and research partners on NASA mission proposals. The types of organizations included for-profit businesses, universities, University Affiliated Research Centers, and nonprofit organizations that have led space missions. These representatives acknowledged that a small core group of “gatekeepers” are responsible for determining which proposal ideas move forward and are afforded access to the pool of B&P resources, contacts, support, and mentorship. In some cases, it is one individual who makes that decision for the institution. Other institutions described a process of internal gates wherein proposal ideas are evaluated by a proposal development committee composed of engineers, scientists, and management, and are assessed for likelihood of success before making institutional support decisions. The opportunity to use a committee approach like this exists at all major spaceflight proposing institutions, and selection processes need to continue to evolve based on the most recent evidence-based research on implicit bias and team dynamics. An even greater opportunity exists for these “internal” competitions to be open to “external” PIs and collaborators. For scientists who aspire to have a leadership role in a mission, activities such as involvement in mission data collection, participating in science team meetings, serving in review panels, and seeking mentorship from active mission leaders would help them learn firsthand about the mission process.

As discussed earlier, proposals may be submitted to a number of AOs before they are selected. The feedback from the review process, as well as the opportunity to mature technology can result in stronger proposals in later cycles. Critical feedback from experts outside of the team during the proposal development process is also a useful technique for gathering critiques and identifying potential proposal weaknesses before submission. Such outside expertise is more readily available through larger organizations and NASA centers, and lack of access to these experts can negatively impact proposal development and realizing a diverse pool of potential PIs. As previously mentioned, the institutional decision makers at some institutions and organizations become “zeroth-order gatekeepers” for missions and the PIs. Most of this power is concentrated in a relatively small number of institutions. Nearly two-thirds of the ~400 Step 1 proposals submitted over 20 years to SMD were managed by five institutions: Jet Propulsion Laboratory (JPL) (28%), Goddard Space Flight Center (GSFC) (22%), Johns Hopkins University Applied Physics Laboratory (JHU/APL), University of California, Berkeley (4.5%), and NASA Ames Research Center (ARC) (4.5%). Furthermore, of 93 proposals where a spacecraft provider could be determined, 78% of proposals included partnerships with only 6 organizations: Lockheed Martin (18%), Northrop Grumman (17%), Ball Aerospace (15%), JHU/APL (12%), Millennium Space Systems (10%), and Southwest Research Institute (SWRI) (6%) (Zurbuchen 2019).

Institutions often favor internal PIs such that the institutional investment remains “inhouse.” Thus, PIs at smaller institutions with fewer resources may find it difficult to secure partners at larger centers for proposal development support. Figure 2.3 shows a preponderance of in-house PIs at the top mission management organizations, indicated by the proportions corresponding to “Self.” While it is understandable that private and university-affiliated institutions would predominately or exclusively support in-house PIs, it is notable that 40-70% of the proposals supported by NASA centers had in-house PIs.

Figure 2.4 shows the submitting organizations for each of the divisions in SMD by discipline area between 1996 and 2019. Universities are the largest submitting organizations across all four disciplines, followed by NASA

centers in all disciplines except Heliophysics. For Heliophysics, nonprofit organizations are the second-largest type of submitting organization. Thus, aspiring PIs may need to connect with leading universities, NASA centers, or nonprofit organizations for proposal development and submission.

Finding: Proposals to NASA mission calls involve collaboration of NASA institutions, spacecraft providers, and potentially instrument providers, research laboratories, and universities. Institutions invest hundreds of thousands to millions of dollars in developing concepts and assembling proposals. Decision makers at the institutions effectively control the investments and the opportunities to become a PI.

Conclusion 2-1: The process of team formation and concept development requires significant resources and is, in part, informal, idiosyncratic, organic, opaque, and often personality-driven. These informal and organic processes, influenced by who knows whom in the community, directly impacts the diversity of the PI candidate pool.

Conclusion 2-2: The variation of non-NASA/internal resources available for preparation and production of mission concepts, full proposals, and site visits may result in an unfair advantage of some institutions over others.

Finding: NASA centers supported ~28% of mission proposals. Of these, 40-70% had in-house (e.g., NASA) PIs.

Conclusion 2-3: NASA is a leading force in the overall demographic makeup of mission PIs. The process of PI selection for NASA center-led missions, however, could be much more transparent and committed to supporting traditionally underrepresented PIs. A restructuring of this type would provide a roadmap for other institutions to follow in their own PI selection processes.

NASA PROCESS: PROPOSAL AND SELECTION MECHANISMS

There are five science divisions in NASA SMD: Heliophysics, Planetary Science, Astrophysics, Earth Science, and Biological and Physical Sciences. The competed opportunities and subject of this report focus on the first four divisions. Within these four NASA divisions, there are a number of programs for competed missions, with each program focused on a particular size/cost of mission. These are summarized in Table 2.1, which also includes the non-competed mission series for context: small to medium-class mission AOs (Explorers, Venture, Probes, SIMPLEx) are released approximately every 2 years, New Frontiers opportunities normally on a 3- to 4-year cadence, and flagships normally on a 4- or 5-year cadence.

The process for the release of AOs, response through proposal submission, and review of proposals has been improved and made more uniform across NASA’s four divisions since 2009. As seen in Figure 2.5, the timeframe from the AO to the selection and initiation of a mission may take on the order of 1 year for one-step proposals or 2 years for two-step proposals (see Figure 2.6). These steps are almost entirely under NASA’s direct control and strictly proceed according to procedures specified in federal regulations (the NASA FAR). Specifically, Section 1872 of the FAR governs the acquisition of investigations, with detailed specifications for the AO (see 1872.3), evaluation of proposals (see 1872.4; processes defined in 1872.401), and recommendation, selection, and debriefing (FAR section 1872.5). The mission directorate associate administrator (MDAA) retains the authority and responsibility for final approval of the AO and its release as well as for proposal evaluation and resulting selection(s).

TABLE 2.1 Programs for Competed and Non-Competed Missions Across NASA SMD Science Divisions

^a The SMD divisions also have suborbital programs (rockets and balloons).

^b See Appendix A for definitions of acronyms.

SOURCE: Adapted and updated from S. Lipsy, “PI-Led Mission Proposal Process,” presentation to the committee, February 8, 2021.

NASA Proposal Process Overview

The overall flow of the NASA proposal process starting with the AO using a one-step process is shown in Figure 2.5. For some larger missions, a two-step process is utilized, as illustrated in Figure 2.6. The one-step process and the first step of a two-step process are identical. Proposals are evaluated against a set of criteria around science and technical factors that are discussed in more detail in later sections. For a one-step proposal, the evaluation reviews flow through the categorization subcommittee step below where one committee integrates the evaluations into one of four categories ranging from recommended for acceptance to recommended for rejection (details in later section) and the steering committee then recommends selections. The decision-making authority, the SMD associate administrator (AA), then finalizes the selection of missions to go into formulation. For two-step proposals, the evaluation feeds into Categorization and Selection for the Concept and Technology Development Phase (Phase A). All missions selected then complete a CSR on the same schedule and a review step (technical, management, and cost [TMC] feasibility required, science evaluation if necessary) is used to down-select to a final mission selection for formulation. In the sections that follow, the main steps of this process will be discussed in more detail.

NASA AOs and Proposal Submission

One unique aspect of the NASA AOs is the FAR Supplement that governs the processes. NASA is one of the few federal agencies with a supplement of this nature because of the unique nature of its work with the science and technology communities and the sizeable investments it makes to obtain science data derived from investigations to, from, and in space. An important element in the FAR is Part 1872: Acquisition of Investigations, under which AOs for competed opportunities fall. The regulations spell out in rather specific detail the process for solicitation, review, evaluation, and selection in addition to roles, accountability, and authority.

As described by the FAR, the AO system used for acquisition of science, technology, or exploration flight investigations is different and separate from the agency procedures for acquisitions for prior known requirements. A general distinction between normal acquisitions for goods or services and AO investigation acquisitions is that AO proposers define not only how they will accomplish investigation objectives but also the objectives themselves within the framework of broader articulated NASA goals. Unlike a proposal responding to an RFP, an AO proposal effectively defines a statement of work in addition to the approach (and cost) for how the proposer plans to carry it out. Typical examples where the use of AOs is most applicable are solicitation of complete small- and intermediate-size investigations (larger and more costly than suborbital-class research but smaller and less expensive than the largest and most expensive strategic flagship missions), of instrument investigations on larger missions whose overall management is assigned to a NASA center, or of spaceflight instrument investigations contributed to a non-NASA mission.

The Science Office for Mission Assessments (SOMA) at NASA Langley Research Center (LaRC) supports SMD at NASA Headquarters in the acquisition of Earth and space science missions and instruments through the development of AO solicitations and the TMC evaluations of proposals received in response to the AO solicitations

and Phase A concept studies.⁵ This office serves as a resource center with a library of AOs, lessons learned documents, and the schedule of upcoming AOs.

NASA has created a centralized portal for information about proposal opportunities and the submission of proposals called the NASA Solicitation and Proposal Integrated Review and Evaluation System, or NSPIRES. Proposals of all kinds are submitted through the NSPIRES system, and all submissions are electronic. In the case of mission proposals, a typical proposal length is 200-300 pages, with 20-30 pages focused on science. Given the complexity of the evaluation criteria (to be discussed below), a proposal must be extremely well crafted to respond to all of the criteria. Preparation of such a large document, including proper formatting of figures and cross-referencing of sections and information is a significant effort. In addition, NASA flight missions must comply with NASA Space Flight Program and Project Management Requirements, known as NASA Procedural Requirement (NPR) 7120.5 (NASA 2021), which defines the NASA life cycles for spaceflight programs and projects, as well as program and project management roles and responsibilities.

Finding: The NASA AO process is highly structured and bound by FAR and the NASA FAR Supplement, which fully articulate it; however, in practice, the process is complex and can be very difficult to navigate and understand.

NASA Evaluation and Selection Processes

As described in the FAR and illustrated in Figures 2.5 and 2.6, once proposals are submitted, the NASA process begins with evaluation. The evaluation results feed into categorization of the proposals, and then the Steering Committee validates the categorization. Recommendations for selection are made based on the categorizations, discussed further below, but are influenced by the division programmatic considerations. The division prepares a briefing for the selection official, who may, and typically does, consult other agency officials, but has the sole authority for the selection.

The scientific and technical evaluation criteria, at the highest level, are quite streamlined. The three high-level criteria are: (Criteria A) scientific merit of the proposed investigation (4 factors); (Criteria B) scientific implementation merit and feasibility of the proposed investigation (5 factors); and (Criteria C) TMC feasibility of the proposed mission implementation (5 factors). These are described in detail in the FAR in Section 1872.402, as follows Criteria and Factors (see Annex 2.B, Tables 2.B.1-2.B.3 for descriptions of the factors). These are the evaluation factors applied in one-step proposals and in the first step of two-step proposals. The weight of the evaluation factors must be stated in the AO if they are unequal. The typical weighing is science merit (Form A) 40%, science implementation (Form B) 30%, and TMC (Form C) 30%. Annex 2.B, Tables 2.B.1-2.B.3 reveal the complexity of the evaluation factors, which contributes to the large level of effort and high costs of producing a competitive proposal.

The evaluation criteria do not specifically include diversity, which is a missed opportunity, as the evaluation criteria drive the proposal development strategy and the focus of efforts and resources. Factor B-5 (probability of science team success) does include the probability of the investigation team success, including the experience, expertise, and organizational structure of the science team. In September 2021, NASA released a request for information (RFI) with the title “Adding Inclusion, Diversity, Equity, and Accessibility (IDEA) Requirements to NASA Announcements of Opportunity.” One aspect of the RFI was to solicit feedback on the proposal to add this evaluation criteria “Factor B.6. Inclusion, diversity, equity, and accessibility (IDEA). The team’s IDEA plans for forming a diverse team, and plans for creating and maintaining an inclusive and equitable environment will be assessed. This factor includes the alignment of the proposal with NASA’s core value of inclusion and the likelihood of successfully achieving the objectives of the “Code of Conduct” in service of mission success (NASA 2021). The final status of this proposed criteria is unknown at the time of the writing of this report, but if adopted, this is a positive step in including an aspect of diversity, equity, inclusion, and accessibility (DEIA) in the evaluation of proposals.

___________________

⁵ Standard template of the AO can be found in NASA (2018).

Finding: Team diversity is not included in the criteria used to evaluate NASA mission proposals, but currently may be included with “other factors” for consideration by the selection official in final selections. A NASA RFI was released in September 2021 seeking comment from the public on the proposed addition of an evaluation criteria focused on the DEIA plans of the proposing teams.

The evaluation of Criteria A and Criteria B are performed by panel(s) of community experts who are free from personal and organizational conflicts of interest. Criteria C are evaluated by a TMC peer review panel. For each Factor in each criterion, the panels develop assessment findings or state that there are none. The panel chair prepares a report form on findings that captures the major points, and these are expressed in short narrative for identified major strengths, minor strengths, major weaknesses, and minor weaknesses. A summary evaluation is then developed, ranging from Excellent to Poor for Forms A and B, and from Low Risk to High Risk for Form C. The rubric for these summary evaluations is included in Annex 2.C where Table 2.C.1 contains details for the science and implementation evaluation, and Table A.5 tabulates the TMC summary evaluation details.

The results of the proposal evaluation Form A, Form B, and Form C are then used by the Categorization Committee and Steering Committee in the process described in FAR Section 1872.404. These committees are composed wholly of civil servants and Intergovernmental Personnel Act appointees. The Categorization Committee will consider the evaluation results and categorize the proposals as defined in the AO, in levels from Category I (recommended for acceptance) to Category IV (recommended for rejection). The details of all four categories are presented in Annex 2.D, Table 2.D.1. The categorization process integrates the information and scoring from the evaluation against Criteria A, B, and C, and the written assessments. As stated in the FAR, the objective of the categorization process is to derive a portfolio of consolidated merit-risk characterizations that will be subjected during the recommendation and selection processes to additional programmatic criteria. In this way, all of the science and technical evaluation information is considered by one body and categorized by one committee. The inclusion of programmatic criteria is important for NASA to achieve its broader goals, but may appear to proposers as a lack of transparency.

The Steering Committee then reviews the results of the proposal evaluations and categorizations, and conducts an independent check on the quality, balance, and integrity of the evaluation process to this point. This committee requires less detailed technical expertise and can provide a fresh perspective from qualified and experienced individuals with broad or different backgrounds. Details of the steering committee responsibilities and expectations are in FAR Section 1872.405.

In the case of one-step proposals, the results of the proposal evaluation will be presented to the SMD AA, who will make the final selections. The overriding consideration for selection will be to maximize scientific return and minimize implementation risk while advancing NASA’s science goals and objectives within the available budget for this program. In addition, the SMD AA may take into account a wide range of programmatic factors in deciding whether or not to select any proposals and in selecting among top-rated proposals. Programmatic factors can be administration priorities, recent discoveries, balancing portfolio, etc. In this way, NASA selection officials may also act as gatekeepers.

In the case of two-step proposals, multiple missions are selected and funded by NASA at this step to begin the Concept and Technology Development Phase (Phase A). The product of the Phase A study is the CSR, which corresponds to a CML of 6. The CSR allows the TMC to see additional detail, a higher fidelity cost estimate, and schedule consistent with expectations for Phase A products. The science objectives and science requirements are re-validated or amended if needed. If the science, exploration, technology objectives, or mission requirements change, Form A (the proposal’s intrinsic merit) will be re-evaluated. In all cases, Form B and Form C are reassessed by a Concept Study Review Panel. The requirements for the CSR are delineated in a 50-page document (NASA 2010), and example slideshow presentations that outline the CSR evaluation plan are publicly available.⁶ The overall flow of evaluation steps is shown in Figure 2.7, and the presentation of the CSR is conducted in a full day in-person meeting or in a virtual format, where 10 hours of presentation occurs over 2 days. Site visit

___________________

⁶ See, for example, NASA (2021a).

instructions are provided to teams with details of the actions, timing of events, and tools to be used for site visits. The PIs brief NASA Headquarters in the final stages of the CSR evaluation.

NASA provides some funding for Step 2 proposals with the expectation that all Phase A and products requirements will be developed and ready for in-depth review, consistent with the CSR requirements including independent evaluation by a group of experts during a site visit. Typically, at this stage, many institutions, including NASA centers and some universities, provide additional funds to supplement the NASA Phase A funding. The results of the evaluation of the CSR and site visit are used to select a mission for continuation (down-select) and recommendation to the selection official. The process is rigorous and, in many cases, taxing, and the NASA-provided funding can be inadequate to meet expectations for the CSR and the preparations and execution of the site visit. Moreover, limitations in resources beyond the NASA provided funds can specifically put smaller colleges, universities, companies, and nonprofits at a disadvantage as well as new PIs who are not aware of this additional need.

Conclusion 2-4: Preparation for both the CSR and the associated site visit for two-step proposals are intense and demanding activities, involving 1 to 2 days of the site visit and a subsequent presentation from the PI to the SMD AA. Some institutions provide additional funds to supplement the NASA Phase A funding. Thus, the variability of resources available and the critical role of a single presentation (PI to SMD AA) merit reconsideration of the resource allocation, review content, and purpose.

Competed mission opportunities vary in scope, complexity, and cost boundaries. What is required to put together a winning proposal (i.e., compelling science investigation, technically feasible, and programmatically credible) varies accordingly. It is a challenging job with great expectations, particularly for the PI who by definition has the most at stake in the process in terms of time investment and expectations for accountability for the scientific/technical and programmatic success or failure of the mission.

Feedback to the PI Teams

PI teams expect and deserve feedback, particularly non-selected investigators. Constructive feedback is useful, even when there are critical and major weaknesses. The highest value component of the feedback process may be the face to face debrief during which evaluation process leaders do their best to explain weaknesses that may have been most discriminating in the final selection process by senior officials. Anecdotal evidence suggests that feedback to non-selected PI teams on mission proposals is uneven, particularly for Step 2 proposers who are not selected, because decisions are sometimes driven by other factors and are more programmatic in nature. While there may be other factors that influence the final selection, conveying such information would be helpful especially for mission concepts that fall in the selectable categories. The feedback will shape reinvestment strategies to bid again armed with helpful tools and feedback. The feedback is an essential element and a big part of the learning process, particularly for new investigators. Going forward in the highly competitive mission AO competitions, attention to the quality and constructive utility of the personal and written feedback is almost as important as debriefing the selected teams. The attention will foster an encouraging environment and assist to better develop a pipeline of high quality mission proposals.

The process for evaluating and reviewing mission proposals and the information presented and made available by NASA have become more uniform across divisions, with improvements in transparency and clarity. Nevertheless, it is a complex and long process with detailed, extensive evaluation criteria that can be daunting for those going through the process for the first time.

Conclusion 2-5: The steps of the NASA AO process are governed by detailed regulations, but scrutiny for sources of bias throughout the implementation of the process is warranted, and the process would benefit from including DEIA as criteria for evaluation.

PI AND TEAM LEADERSHIP DEVELOPMENT: CHALLENGES, OPPORTUNITIES, AND POTENTIAL BARRIERS

As has been described, all aspiring PIs must overcome numerous challenges stretching over a period of several years or more. These impediments are likely to be exacerbated for historically underrepresented populations, including women, or for those from research institutions that lack a record of NASA missions. Challenges discussed earlier in this chapter include:

• Gaining mission leadership experience
• Benefitting from mentorship
• Winning commitment from one’s home institution
• Forming a suitably broad and deep team,
• Establishing leadership
• Climbing the CML and TRL levels
• Securing institutional commitments and B&P resources (competition before the competition)
• Leading preparation of a compelling proposal
• Leading preparation of a compelling CSR
• Conducting an appropriate site visit
• Briefing the SMD AA

In the final section of this chapter, additional challenges are discussed, including but not limited to team dynamics, access to information and training, and proposal review.

Team Dynamics

A wide range of expertise is brought together for a mission proposal. The PI has the ultimate responsibility carrying out and reporting the results of the mission (NASA 2021d), but there are many other key roles. For example, the project manager is responsible for the formulation and implementation of the mission in accordance to NPR 7120.5, and the engineering technical authority or system engineer manages the engineering activities, including systems engineering, design, development, sustaining engineering, and operations. A deputy PI may be on the team, sharing responsibilities with the PI. An instrument lead may be included to provide engineering leadership for the instrument. For PI-led competed projects, there may be a project scientist, who is part of the PI team and works closely with the PI. The project scientist is typically delegated the responsibility to monitor the scientific output of the project and ensure that the project achieves each of its science requirements. For directed projects, there is no mission PI. The project scientist in that case is responsible for a more significant fraction of the project-level management than in a competed project. The project scientist works closely with the project manager and is directly responsible for all science related tasks. Other key roles are the operations manager, who oversees the planning of flight operations and instrument commanding. A ground data systems lead manages the data processing system. The PI is critical for the mission; however, the overall success and team dynamics are driven many contributors that require technical expertise and leadership skills.

There is no defined process by which teams form in the early development space between CML 1 and 2 or 3. To this end, the committee relied heavily on the scholarship on team science, interdisciplinarity, and team dynamics to make determinations about how the undefined nature of team formation for missions impacts diversity in the proposal pool. In their presentations to the committee, scholars who specifically study NASA spacecraft teams as well as team effectiveness in science described team formation as “organic,” meaning that it is tied to who you know and who you are mentored by. For example, in working with NASA robotic spacecraft teams, Vertesi (2021) found that teams form across NASA institutions and universities in a “federation of cultures,” and with precious, few missions and interplanetary timelines, teams work together for multiple decades on the same project, and lifetime appointments in the same positions do not allow for organizational or intergenerational mobility. Network effects and relationships also matter, as PI-led missions reduce costs up to 30% by relying on existing ties (NRC 2006). Other scholars reported how “high-performing teams” tend to stay together from project to project, which may discourage new researchers. Also, minorities or other marginalized groups may be subject to tokenism (Kanter 1977b) or backlash (Rudman and Glick 2001), leading to continued low representation on teams (Correll 2004; Rathbun et al. 2018; Porter et al. 2020; Rivera-Valentin et al. 2020). An opportunity exists to integrate emerging researchers as mentees and participants across the mission teams, including the higher leadership positions. In practice, this may be through formal mentorship, training of existing team members to prepare them for advancements, and strategies to expand participation of emerging researchers in science team meetings. Such activities present enormous opportunities for emerging, early career researchers to rise through the ranks, gaining invaluable experience in running a team and mission along the way.

The scholarship on the science of team science highlights that team culture, team dynamics, collaboration readiness, decision-making strategies, intersectional identities, cultural biases toward women and underrepresented groups in leadership positions, and inter-organizational cultural differences are all important factors that influence who is on the team and how well the team will perform. For instance, working across institutions was found to present challenges to teams as inter-institutional collaboration may involve variable chains of commands and intercultural communication challenges akin to international teams (Ting-Toomey and Oetzel 2001; Lebaron and Pillay 2006; Hall et al. 2008; Vertesi 2020). In a study by Rozovsky et al. (2012) on how teams work, it was found that psychological safety, which allows for “moderate risk-taking, speaking your mind, creativity, and sticking your neck out without fear of having it cut off” was the most critical factor for the success of high-performing teams. Additionally, using an intersectionality framework allows for a better understanding of how social issues restrict access and power, and without this framework one defaults to listening to only the most privileged groups (Cole 2021). Chapter 5 of this report also discusses team dynamics and insights of the impacts on PI-led missions. Increased awareness of the evidence-based considerations for science team effectiveness by all who participate in NASA-funded missions can influence future team formation and team performance. Efforts to form teams and

bring together the needed skills and knowledge would benefit from considering current research on key topics, such as how the most excellent team science gets done, what are the impacts of cultures on diverse teams, and how to use consensus-based decision-making. NASA can encourage these concepts to be integrated at the start of team formation by providing best practices documents and training sessions that educate PIs and team members on such strategies as the Concordance Method of decision-making (Schutz 1994).

The expectation of what makes a “good PI” may be outdated and misdirected. For example, a National Academies study from 2015 states “leadership is not a quality that an individual either has or lacks, and there is not a single leadership style that is effective in all contexts (NRC 2015). Yet, there is a myth that persists in the fields that PIs must have certain personality traits such as dominance, paternalism, sole decision-maker, and more. It is also communicated to potential PIs that “your life will be taken over, including evenings and holidays” (Zurbuchen 2019). However, that is not a prerequisite. While the role of a PI is significant and a large undertaking, leading a space mission is a team endeavor. The leadership of a large technical team and execution of a complex technical mission is demanding, but the leadership of such teams can be executed effectively in many different ways, as described in a recent National Academies report. Discussions of the expectations of PIs by NASA do not currently reflect the wide range of leadership styles.

Finding: A frequently heard message is that being a PI is all-consuming and nights and weekends will need to be sacrificed to take on this role. Other data show that different PIs have different styles of management and delegation, and there are PIs who effectively employ delegation, time management, and boundary setting.

Conclusion 2-6: NASA can do more to socialize the concept of diverse leadership styles and highlight that leadership success is more important than leadership style.

Information Access and Training

Each AO has attracted a reasonable number of PI-led proposals that achieve the highest ranking and are eventually selected and flown (see section “NASA Process: Proposal and Selection Mechanisms” of this chapter for more discussion of selection process). These proposals have originated in NASA centers, university affiliated research centers, and in a select set of universities and public or private research centers that either have the capability of carrying out such large-scale space projects or can collaborate with these institutions. Within the past few years, NASA has recognized that such an organic approach limits the population of potential PIs and can limit access to the pool of skills and ideas that would come from broader participation. Findings from the commissioned qualitative study (see more in Chapter 5; also see Appendix C) reinforce the perspective that early career mission experience, mentoring, and access to networks of colleagues across a range of institutions are seen as very important for developing PIs, yet can be difficult to access. The commissioned qualitative study also revealed that respondents felt that the dedication and qualifications of the proposed PI play an important role in mission success; however, all of the skills needed are not learned through formal education routes like undergraduate and graduate training. Some respondents also believed that NASA training opportunities are not always widely publicized and are more accessible to those who are well-connected. These topics are also further discussed in Chapter 4 and Chapter 5.

The PI Launchpad initiative is one mechanism that was motivated by an aspiring PI who was frustrated with the process of proposal development. Created in collaboration with NASA and the Heising-Simons Foundation in 2019, this still evolving training is currently a 3-day workshop targeted at researchers and engineers who would like to submit a NASA space mission proposal in the next few years but do not know where to start. The workshop is designed to introduce the proposal process and some of the key steps, such as creating a science case, developing requirements, writing a science traceability matrix, building a team, and getting support at your home institution.⁷ Overall, this initiative exists to better prepare and aid the scientific community with the proposal preparation and submittal process.

___________________

⁷ A complete list of areas covered by the PI Launchpad Initiative can be found in the workbook NASA (2019).

In addition to the PI Launchpad, other resources for aspiring mission PIs are offered by NASA.⁸ These include the following: a recording and slides from a colloquium presentation by Thomas Zurbuchen, AA of NASA SMD, titled “Writing Successful Proposals: Observations from NASA” (Zurbuchen 2019); recordings and slides from presentations on writing Research Opportunities in Space and Earth Sciences (ROSES) proposals; proposal writing workshops at the American Geophysical Union and other conferences; planning resources for AOs, such as the standard template used; and mission management resources such as NASA’s procedural requirements. Additionally, a general guidebook for proposers is routinely updated by NASA (2021b). Aside from this resource, few professional development offerings by NASA exist for prospective mission PIs, thus requiring a reliance on other information from the field to navigate the proposal process. A good example of this is a host of resources put together by a PI of an Earth Science Mission proposal (Gentemann 2021).

As discussed earlier in this chapter, a significant commitment of time and money is needed to develop concepts into winning proposals. In addition, the process can be intimidating to new proposers, particularly those at the early career stage, and those that historically have not had the benefit of a support system that values diversity and inclusion. This suggests that unless there is significant support, awareness of the challenges, and sustained mentorship in place it can be hard to start and remain committed to the process while navigating the barriers that exist along the way, particularly for women and underrepresented investigators. Investigators with a broad network of support either within their own or a partner institution or through other means of exposure and interactions are obviously at an advantage. A key question is whether the training opportunities to be exposed to the process and support systems are available to others in an equitable way. This is a complex issue that requires a multi-pronged approach and investment in capital if realizing DEIA in the PI proposer pool is a goal.

A variety of issues are at play, many of which may not be within NASA’s control. For example, are the intersectional identities of proposers taken into account or are they defined by only one demographic to which they belong? Also, what is the current state of the space sciences pipeline and are national level STEM-related programs and priorities, such as NSF ADVANCE (Organizational Change for Gender Equity in STEM Academic Professions) and NASA Science Activation, having the intended effect and in a timescale that allows participants to see themselves growing up the ladder into positions of leadership? Looking beyond the pipeline, it is important to understand the venues and opportunities whereby meaningful contributions (i.e., not just token participation) with consideration for the full expression of DEIA (e.g., a PI, deputy PI, investigation team lead) is enhanced and nurtured. Figure 2.8 shows the distribution of science team members across NASA mission proposals from 2006 to 2020. There are more Co-Is than any other role (81%), and PIs are only 5% of the pool. Interestingly, there are less than 1% of deputy PI roles, suggesting very few PIs are planning on working with a deputy. Training and opportunities to gain mission leadership experience exists in these other roles, especially the Co-I role, as well as the deputy PI role.

Finding: Aspiring PIs are expected to have some experience on flight missions or instrument development, and will need networking and management skills. This type of experience and skill development is not typically acquired during undergraduate or graduate degree work. There are some opportunities in existing missions, but they are extremely limited. Many aspiring PIs reported frustration with the limited opportunities for training and gaining relevant experience.

Conclusion 2-7: Additional opportunities for training and mission experience are important for expanding the group of trained professionals that become potential mission PIs, and could range from formal mission roles on mission teams; to mentorship from current and previous mission leaders; to resources and strategies for increasing the involvement of Earth and space science community members in mission science team meetings.

___________________

⁸ NASA, “New Principal Investigator (PI) Resources,” https://science.nasa.gov/researchers/new-pi-resources.

Proposal Review

Proposal review is a process where bias can impact the results. NASA seeks to remove barriers to participation and reduce the impact of bias in the process, and one approach to achieving this objective is to ensure that the review of proposals is performed in an equitable and fair manner. Recently, NASA has explored Dual Anonymous Peer Review (DAPR) processes in its ROSES calls following the successful implementation at the Space Telescope Science Institute (Strolger and Natarajan 2019). Other similar studies conducted at the National Radio Astronomy Observatory, the Atacama Large Millimeter Array, and the European Southern Observatory have also indicated biases affecting proposals (Lonsdale et al. 2016; Patat 2016).

NASA has begun pilot implementations of DAPR in ROSES-2020 and early data appear consistent with improvements, both in terms of the overall quality of the review process, as well as in the demographics of awardees. For instance, in the Astrophysics Data Analysis program, prior to dual-anonymous review, women constituted 26% of the applicant pool, but only finished in the top two places in the panels’ rankings 16% of the time. Following the switch to dual-anonymous peer review, women constituted 31% of the applicant pool and finished in the top two places 32% of the time.⁹ Moreover, the success rate of early-career investigators even eclipsed that of the more senior investigators, further enriching the talent pool. The evidence to date continues to underscore the value of removing any possible sources of bias, while recognizing that it is not a substitute for other steps to improve the diversity of the pool of applicants for all NASA competed opportunities. The DAPR process and principles can be a vehicle to also improve the diversity of competed mission investigator teams and deserve attention and methodical experimentation by NASA throughout the proposal evaluation and selection process.

Finding: NASA SMD has begun to test and implement changes to peer review to broaden applicant pools and reduce potential biases in evaluations and awards. The observed changes after instituting DAPR were a broadened applicant pool and selections with gender ratios closer to submission gender ratios.

___________________

⁹ NASA, “Overview of the Dual-Anonymous Peer Review Process,” https://science.nasa.gov/researchers/dual-anonymous-peer-review.

ANNEX 2.A

Concept Maturity Levels

The committee adapted the following concept maturity levels (CMLs) from Wessen et al. (2013) to provide more complete details and typical resource requirements to achieve each level.

• CML 1 Sketch on a Napkin—The science questions have been well articulated, the type of science observations needed for addressing these questions have been proposed, and a rudimentary sketch of the mission concept and high-level objectives have been created. The essence of what makes the idea unique and meaningful have been captured. The conceiver of the idea may not be the principal investigator (PI) (e.g., an orbital dynamicist may identify an interesting trajectory that may enable unique science, and approach a scientist who has a potential to lead the development as the PI). CML 1 idea is shared with a person/entity with capabilities/resources to test initial feasibility to advance to CML 2.
• CML 2 Initial Feasibility—The idea is expanded and questioned on the basis of feasibility, from a science, technical, and programmatic viewpoint. Lower-level objectives have been specified, key performance parameters quantified and basic calculations have been performed. These calculations, to first-order, determine the viability of the concept. Initial feasibility study is usually a low-level effort that may or may not be funded. Whether to invest time, effort, and energy into this activity is often a personal or career judgment on risk, benefit, ambition, and interest. When funding is available, it is funded internally (e.g., Internal Research and Development [IRAD] programs). Technology maturation needs may be identified from this work.
• CML 3 Trade Space—Exploration has been done around the science objectives and architectural trades between the spacecraft system, ground system and mission design to explore impacts on and understand the relationship between science return, cost, and risk. Proposing team makes contact with a mission design center to perform a trade study (CML 3). Variations exist in the manner of contact—mission proposal team may contact a design center, or vice versa. In other cases, a mission concept may organically arise between the proposing team and the design center. Design center performs a design trade study to achieve CML 3. A trade study is a high-level effort, typically involving a small science team (~5 members) and 10-20 engineers and takes place over a 2-3 month period. Technology maturity assessments are likely to be an aspect of the trade study. Proposing team may be affiliated with the institutions that has a design center capability, in which case the process is fully internal to the institution. When the proposing team/PI are not affiliated with the design center’s institution, the effort of the PI/Proposing Team may not be funded, or a contract may be involved between the design center and the proposing team. In some cases, the proposing team pays the design center, while in others the design center may fund the proposing team.
• CML 4 Point Design—A specific design and cost that returns the desired science has been selected within the trade space and defined down to the level of major subsystems with acceptable margins and reserves. Subsystems trades have been performed, and technology readiness level assessment is further refined. Design center builds a point design to advance the concept to CML 4. A point design process is a high-level effort by the science (5-10 members) and engineering teams (~20 members, usually the same members as the trade study team). Sometimes, CML 3 and CML 4 efforts are decided together as the continuity in the personnel and effort is beneficial.
• CML 5 Baseline Concept—Implementation approach has been defined including partners, contracting mode, integration and test approach, cost and schedule. This maturity level represents the level needed to write a NASA Step 1 proposal (for competed projects) or hold a Mission Concept Review (for assigned projects). Design center builds a baseline concept to be included in a proposal. Some institutions require Blue/Red team reviews at the design center to down-select concepts within the institution to avoid internal competition and/or to determine whether the concept should be proposed at all.

• CML 6 Integrated Concept—Expanded details on the technical, management, cost and other elements of the mission concept have been defined and documented. A NASA Step 2 CSR is at this level of maturity. There is no corresponding milestone for assigned projects.
• CML 7 Preliminary Implementation Baseline—Preliminary system and subsystem level requirements and analyses, demonstrated (and acceptable) margins and reserves, prototyping & technology demonstrations, risk assessments and mitigation plans have been completed. This is the maturity level needed for competed missions to hold their Preliminary Mission System Review and for assigned projects to hold their Mission Definition Review.
• CML 8 PDR (Integrated Baseline)—Design and planning commensurate for a Preliminary Design Review.
• CML 9 CDR—Design and planning commensurate for a Critical Design Review.

ANNEX 2.B

NASA Proposal Evaluation Factors

Tables 2.B.1 to 2.B.3 provide descriptions of NASA proposal evaluation factors for scientific merit, scientific implementation merit and feasibility, and the technical, management, and cost (TMC) feasibility of the proposed mission implementation, reproduced from NASA Standard Announcement of Opportunity Template (NASA 2018).

TABLE 2.B.1 Evaluation Factors for Scientific Merit

SOURCE: NASA (2018).

TABLE 2.B.2 Evaluation Factors for Scientific Implementation Merit and Feasibility

SOURCE: NASA (2018).

TABLE 2.B.3 Evaluation Factors for the Technical, Management, and Cost (TMC) Feasibility of the Proposed Mission Implementation

SOURCE: NASA (2018).

The AO includes additional information about how these criteria are applied to the Baseline and Threshold Science Mission, how feasibility is used in selection, and the role of student collaboration proposals. The details from the Earth Venture Mission 3 AO are included here as an example (NASA 2020a).

Scientific merit will be evaluated for the Baseline Science Mission and the Threshold Science Mission; Operational Enhancement Opportunity beyond the Baseline. Factors A-1 through A-3 are evaluated for the Baseline Science Mission assuming it is implemented as proposed and achieves technical success. Factor A-4 is similarly evaluated for the Threshold Science Mission. This evaluation will result in narrative text, including specific major and minor strengths and weaknesses, as well as an appropriate adjectival rating for the scientific merit of the investigation.

The panel evaluating the “TMC Feasibility of the Proposed Mission Implementation” may provide comments to NASA regarding the feasibility of the proposed access to space. While these comments will not be considered in the evaluation, they may be considered during selection. The panel evaluating the “TMC Feasibility of the Proposed Mission Implementation” will also provide comments to NASA regarding the extent to which the proposed investigation provides career development opportunities to train the next generation of engineering and management leaders. While these comments will not be considered in the evaluation, they may be considered during selection.

Student collaboration proposals, if any, will be evaluated only for the impact they have on the TMC Feasibility of the Proposed Investigation Implementation to the extent that they are not separable. Student collaboration proposals are not penalized for any inherent higher cost, schedule, or technical risk, as long as the student collaboration is shown to be clearly separable from the implementation of the Baseline Investigation.

ANNEX 2.C

Summary Evaluation Scores for Science Merit and Technical, Management, and Cost Evaluations

Tables 2.C.1 and 2.C.2 provide descriptions of NASA summary ratings for the evaluation of proposals for scientific merit and scientific implementation merit and the technical, management, and cost feasibility of the proposed mission implementation, reproduced from NASA Standard Announcement of Opportunity Template (NASA 2018).

TABLE 2.C.1 Evaluation Rubric for Scientific Merit and Implementation

SOURCE: NASA (2018).

TABLE 2.C.2 Evaluation Rubric for Technical, Management, and Cost (TMC) Feasibility

SOURCE: NASA (2018).

ANNEX 2.D

Categorization

The categorization process continues to focus on the scientific and technical merits, implementation feasibility, management approach, cost realism, and comprehensive risk assessment of each proposal individually. The objective of the categorization process is to derive a portfolio of consolidated merit-risk characterizations that will be subjected during the recommendation and selection processes to additional programmatic criteria (see Table 2.D.1).

TABLE 2.D.1 Categories Used in Proposal Selection

SOURCE: NASA (2018).