The Space Science Decadal Surveys: Lessons Learned and Best Practices (2015)

   Best Practice

• Those drafting new statements of task do well to review the statements of task for the previous survey in the field as well as the most recent round from other disciplines for historical and comparative value.

Lessons Learned

• There is no “one-size-fits-all” approach to a decadal survey. Each discipline has cultural heritage and scientific goals that cannot be directly mapped to any other group.
• The presently used process for conducting decadal surveys is able to accommodate these differences in the four disciplines and achieve the goal of community consensus on science goals and the activities that are required to achieve them.

   Best Practice

• Because the disciplines are so different, clear articulation of their unique aspects is useful to readers of the survey reports who might expect “uniformity” in approach across the surveys.

Lessons Learned

• The tendency to over-define mission concepts in pursuit of more accurate cost evaluation can stifle creative approaches to addressing survey goals.
• Because each of the disciplines define and develop missions differently, there can be no uniform approach in dealing with mission concepts from prior decadal surveys, mission concepts that are under competition, or missions in Phase A study. Nonetheless, it is highly desirable for a survey committee to decide early on how to deal with such situations and to communicate this to the panels.

   Best Practice

• A solicitation process to gather community input via white papers together with invited presentations by community groups and field leaders broadens survey participation and ensures there is opportunity to consider important activities for the coming decade. This approach recognizes emerging opportunities and innovative approaches to program implementation that may not be readily apparent to the members of the survey committee and panels.
• The practice within decadal surveys of not defining specific NASA mission concepts for lower cost and competed missions, yet recommending that such missions address priority decadal survey goals and objectives, allows flexibility to leverage innovative implementation approaches.
• A two-step CATE process that allows more concepts to remain in consideration in the early stages of the survey includes a faster, cruder “cost box” analysis for a longer list of candidate concepts. This would be followed by a detailed CATE for candidates for the final program that require more detailed assessment due to their cost, complexity, risk, or importance to the community.
• Decadal surveys can present their implementation strategies as reference missions—that is, a credible hardware configuration that can achieve the science goals and is sufficiently defined for robust cost evaluation—instead of blueprints for detailed implementation.
• It is desirable that the survey committee determine, as early in the process as possible, how robust a mission concept needs to be to provide sufficient cost certainty. An example is an ambitious mission where the survey committee needs to know—with reasonable confidence—that a mission team will be able to propose a credible design that meets science requirements and fits within the cost cap for the mission class.

Lessons Learned

• Although science goals may not be explicitly ranked, science priorities drive the rankings of missions, facilities, programs, and other activities.
• Experience indicates that survey prioritization puts a high value on programmatic balance across missions and facilities, and is also attentive to the need for long-term continuity in certain observational data.
• Technical readiness and affordability can often influence science prioritization of missions and facilities. High-priority science may be deferred if needed technology is immature and there is a significant risk of cost growth that would affect both scope and schedule.
• The potential for international collaboration, interagency cooperation, and inclusion of the private sector impacts science and mission prioritization across all disciplines.

   Best Practice

• Therefore, the best practice for decadal surveys is to choose and describe reference missions (see below, “Suggested Changes in the Prioritization Process”) that are judged capable of carrying out the science, but to encourage agencies to follow, first-and-foremost, the science objectives of the prioritized missions.
• Establishing a community-wide consensus is arguably the most important goal of a decadal survey, and community “buy-in” to the decadal survey’s process is crucial. Community trust in the decadal survey process depends on a clear understanding of the prioritization methodology used by the survey committee and its supporting panels.

Lessons Learned

• It is important that decadal surveys explicitly note which proposed missions are reference missions—i.e., subject to further development—versus those intended as explicit implementation recommendations based on mature and well-refined concepts.
• The community has many means, especially between decadal surveys, to address the evolution of science in the discipline. This forms the basis of a two-phase process without separating the decadal survey process itself in two.

   Best Practice

• Agencies, committees of the Academies, community workshops and meetings, and white papers can contribute to pre-survey science priority identification as preparation for, and a valuable contribution to, the next survey. These activities can also spur early development, evaluation, and maturation of concepts for new missions for potential priorities well in advance of the survey itself.

Lessons Learned

• CATE involves assessment of a single point design to assess cost and technical risk. It is most useful as a reasonableness check on what is being recommended. Details used to support the CATE analysis are not necessarily indicative of how a mission will ultimately be implemented.
• The combination of budgetary constraints (wedges) and large, complex mission concepts can lead to difficult choices, reinforcing the need for a thorough understanding of mission costs and risks as well as the establishment of clear decision rules.
• The CATE process benefited from and indeed depended on, survey committee and panel members serving as liaisons between Aerospace Corporation’s CATE team and panels. Continuity of some individuals across multiple surveys also served a useful purpose as each survey had different experiences, and “lessons learned” were fed forward. Panels provided an important cross-check critique and key feedback to the CATE process.

   Best Practice

• The survey committee can choose, and subsequently identify in its report, the role of CATE in their survey. The CATE could provide, for example, a best-possible cost estimate for a point design or an independent, rough estimate for comparative purposes.
• To prevent the CATE analysis from unnecessarily “driving” the decadal survey process, survey committees can consider implementation of a two-step CATE in which rough technical readiness and risk assessment feedback (accurate to a factor of two or three) would be provided for most, if not all, concepts early in the survey process. The more detailed and comprehensive CATE analysis (as used in recent surveys) would be reserved for those concepts that the committee identifies as worthy of further study.
• Decadal survey committees are advised to determine a fair and consistent way to evaluate all international partnerships, which would be communicated to the panels early in the decadal process. The technical evaluation can be comprehensive and inclusive of the international portions and risks. However, assessment for affordability may need to be, in a pragmatic sense, for the U.S. portion only.
• When composing the survey committee, it is worth considering identification of one or more liaisons who will serve as go-betweens the panels, the committee, and the Aerospace Corporation’s CATE team.

   Best Practice

• Decadal surveys may review the recommended program from previous surveys and choose to endorse certain activities in their own recommendations. Such reviews are best if they focus on those missions and facilities that play a critical role in the proposed science of the survey report.
• It is desirable that the statement of task explicitly address the extent to which existing programs or projects are to be reviewed and recommendations from prior decadal surveys are to be revisited.

Lessons Learned

• Although extended mission phases are not generally budgeted for future missions in the decadal survey process, they may present significant hidden costs that may influence decadal survey implementation cadence.
• High-profile missions are special cases within each of the disciplinary areas, presenting great opportunities for major advances in understanding, but also carrying significant risk for maintaining a balanced portfolio of activities—should unanticipated cost growth occur.
• Mission creep within high-profile missions and large facilities and a general unwillingness to de-scope or cancel large missions or facilities during development can result in large, negative impacts on other programs at the division and directorate level.

   Best Practice

• It is important for the decadal survey to estimate and clearly describe and illustrate in their report potential liens on the following decade from their recommended program.
• It is highly desirable that the decadal survey report includes clear discussions on how the decadal survey committee determined the optimal balance of programs and activities for the coming decade.
• It is incumbent on a decadal survey report to clearly delineate the respective roles of NASA, NSF, NOAA, USGS, and/or other federal agencies in implementing the science program when the capabilities and interests of multiple agencies are involved.
• When recommending high-profile missions, survey committees are advised to explicitly state which aspects of the mission are essential to retaining the mission’s consensus priority and which can be further considered during design development to enable cost control.
• Clear decision rules for high-profile missions and large facilities that include both de-scope and cancellation options can provide some level of protection against unconstrained cost growth and possible collateral damage to other programs.
• While high-profile missions are likely to retain their high ranking from one decadal survey to the next, evolution in mission concepts and changing science priorities may occur over time. As such, it is desirable that the survey committee and panels carefully evaluate all candidate mission concepts on their merits, rather than be unduly influenced by advocacy and inertia.
• Strong preferences by the agencies on how to deal with high-profile missions and interagency and/or international participation in missions and facilities need to be spelled out in the statement of task.
• In developing the recommended decadal program, survey committees and panels are advised to include explicit consideration of various forms of programmatic balance. This might include, for example, the balance across the subdisciplines, between mission and non-mission activities, between novel and continued observations, across mission and facility cost, and between program elements (e.g., R&A, technology, infrastructure, missions) and activities (e.g., education, engagement, and workforce development).

• It is desirable that discussion of technology development in the decadal report be included in both the survey recommendations section and in the panel reports, so that technology requirements for the coming decade (and beyond) can be adequately captured, while identifying subdiscipline-specific requirements.

Lessons Learned

• While the survey report is the primary result of the decadal process, engagement of stakeholders and the broader community in the survey recommendations is also critical to the success of decadal surveys. Communication by the decadal survey committee leadership and members with science community groups and at science and society meetings promotes broad community buy-in.
• A single, unified list is expected by many stakeholders—and when one is not provided, the closest thing to it will be used—more often, misused. It is very important that the survey report carefully describes the recommended program in its entirety, with proper emphasis on lists of prioritized missions and facilities.
• Individual chapters representing the work of the panels provide important information and the back-story to the final recommendations of the decadal survey in terms of both science priorities and implementation strategies. As panels represent subdisciplines within the decadal survey, all their priorities do not necessarily align with the survey committee, but are used as critical input into the discussions of the survey committee.
• As the best and most detailed record of community input, a decadal survey’s panel reports are a fundamental part of the survey’s work product. It is essential that they be made public along with the committee report. Publishing the survey committee report and the panel reports together, as has often been done, has the important advantage of providing traceability within one document of the decadal survey process of science and program prioritization.

   Best Practice

• Community acceptance and buy-in on decadal survey recommendations requires careful documentation and communication by the survey committee of their decision-making process for developing science goals and objectives and tracing these into a recommended program of activities for the decade.
• When drafting the decadal survey report, it is best that authoring committees remain mindful of the wide audience for the report, including the international discipline community, federal agencies, Congress, the federal executive branch, and the public, to ensure clear and effective communication of the community’s consensus science priorities as expressed in the decadal survey report. Professional societies, such as the American Astronomical Society and the American Geophysical Union, can be very effective in disseminating the survey program to a wide audience.
• When drafting a decadal survey it is important to clarify the intended use of the cost appraisal for each mission or facility: Is it for a configuration that is intended to serve as (1) a “proof of a concept” that merely establishes the scale of the project; (2) a cost estimated for a mature, well studied concept; (3) as a cost cap; or (4) something else entirely.
• To the extent possible, it is desirable that survey committees craft text that describes the priority activities—why they are the priorities and under what circumstances those priorities might change. So-called “decision rules” that are clearly identified in the survey report can help with this process. Collections of all decision rules in a single section, with traceability back to the body of the report, will facilitate clear communication of the decadal survey’s intent.
• It is important for survey report to clarify the extent to which implementation details are prescriptive or notional, in order to ensure that agencies understand the committee’s intent when developing their implementations strategies.

• In some cases, it is desirable for survey committees to document not only the missions and facilities that are part of the recommended program, but also those that were considered but not prioritized, as well as the rationale behind decisions.
• Development by each panel of prioritized science goals and implementation strategies ensures that its community’s science priorities are fully understood and considered for incorporation into the recommended program for the coming decade.
• A clear articulation of the roles played by the survey committee and panels in absorbing, analyzing, and prioritizing the community’s science goals, within and across the discipline is essential for securing community support for the finished survey—a crucial element of a decadal survey’s credibility to stakeholders.
• To make clear the utility of panel reports and to reduce ambiguity as to their use, decadal committees can choose to publish the panel reports in the same volume as the survey report, adding clear labeling that the panel reports are for reference only.

Lessons Learned

• Decision rules provide deeper insight into the survey scientific priorities and reflect the wisdom and consensus of the scientific community.

   Best Practice

• Decision rules ordinarily are best when strategic in nature rather than tactical. The objective is to provide insight into how science priorities evolve or change under specific circumstances without over-constraining implementation. Long-term advice that advances the scientific goals of the community is useful, whereas short-term rules quickly become obsolete or are better determined by administrators, policy makers, and community members familiar with the immediate situation.
• The best decision rules are clear, unambiguous, and easy to implement without being overly prescriptive. Decision rules are best designed to support the achievement of survey priorities, without overly specifying the means by which they are achieved. Clear if-then rules are more useful, whereas decision rules that require interpretation may often be less helpful. For example, references to general concepts (like balance) are difficult to understand, implement, and evaluate without specific definition and guidance.

• Decision rules need to be clearly identified and labeled as such in the survey report. In order to facilitate clear communication, all of the decision rules from the entire survey should be collected in one section, with each being traceable back to the body of the report.
• Decision rules can be evaluated as part of the midterm assessment process. This may include reviewing existing decision rules, assessing their status, and considering whether any specific decision rule requires reevaluation due to emerging circumstances.

Lessons Learned

• As long as the standing committee restricts its work to the current program, there is no meaningful conflict that would preclude continuation of the SSB standing committees during the execution of a decadal survey.
• The current advisory structure does not provide an effective mechanism for provision of short-term tactical guidance from the scientific community (i.e., tactical guidance for accomplishing strategic visions).
• The AAAC can play an important and unique role in stewardship of a decadal survey through its focus on interagency cooperation.
• Providing scores or grades in a midterm assessment report can result in unintended consequences when used by a wide audience. When grades are used, it is best if narrative text clearly indicates the desired response to a good or bad grade.
• Midterm assessments offer an opportunity to initiate the process of concept development for the next decadal survey. While primarily serving to provide a report on progress to date, the assessments can also act as forward-looking documents in the preparation for future decadal planning.

   Best Practice

• SSB standing committees can continue their work throughout the period when a new decadal survey is in progress in order to provide an uninterrupted channel of communication between these committees and NASA and other agencies, with respect to strategic issues that concern the current program.
• NASA division directors and program officers for other interested agencies (e.g., NSF, the National Oceanic and Atmospheric Administration, the U.S. Geological Survey) can work with the SSB’s standing committees to commission letter reports, meetings of experts, or workshops when specific advice is needed on a more rapid turnaround basis.
• Midterm assessment reports are most useful when they engage and inform the broad community by providing a progress report on implementing the decadal program, together with sufficient context to understand the rationale behind the program’s current implementation strategy.

   Best Practice

• By identifying the essential features and potential challenges international collaborative missions and projects early on, decadal surveys can recommend processes and procedures for avoiding breakdowns, thus limiting the impact in science and cost on both sides.
• Decadal studies can use a combination of existing scientific conferences, meetings, and symposia, as well as more targeted dialogues between survey committees and their closest analogs in the scientific advisory apparatus of other countries, to ensure that lines of communication are open.
• Individual, non-U.S. scientists can be invited to participate in a decadal survey. Participants need to be selected for their scientific backgrounds and expertise, and not as institutional representatives, and be cognizant of a broad range of international activities. International representatives that are experienced and senior enough can provide information that will open avenues for collaboration and strengthen channels of communication back to their home space agencies and national space societies and organizations.
• Decadal reports can include specific descriptions of the types of international collaboration that the decadal survey committee finds desirable (e.g., cost-sharing, development of instrumentation, coordination of individual missions, or mission architecture).
• Decadal reports can explicitly identify any significant programmatic uncertainties and/or craft decision rules that might be required when considering international collaborations. This may be particularly important when international collaborations are a significant component of the survey’s recommended program—in terms of budget or scientific strategy.

   Best Practice

• Achieving the science goals of a decadal survey and successfully implementing survey recommendations requires that the science program be acknowledged as an interagency, multi-agency activity, one that typically extends beyond the purview of a NASA SMD division.
• Participation by all relevant agencies is optimized when decadal reports include specific descriptions of the types of interagency collaboration that the decadal survey committee finds desirable.