2

The Biosignature Standards of Evidence Workshop

CONTEXT FOR THE WORKSHOP AND THIS REPORT

The National Aeronautics and Space Administration (NASA) is among the world’s leading organizations supporting astrobiology research, with a history stretching back to its first planetary exploration missions. Its commitment to astrobiology has evolved over time from missions such as Viking¹ in 1976 to the founding of the NASA Astrobiology Institute² in 1998 to the modern-day Research Coordination Networks (RCNs). Today, NASA and NASA-funded organizations are among the world leaders in the research and development of life detection methodologies, technologies, theories, and missions.

As astrobiologists increase their theoretical and experimental understanding of astrobiology, the capabilities of life detection technologies, and the cadence of missions utilizing these advances, the likelihood of a positive detection of extraterrestrial life grows every year. It is therefore timely to consider how NASA and the scientific community would react to and manage such a discovery. The detection of life beyond Earth would be a signal event in humanity’s history. Scientists and organizations such as NASA will be crucial to disseminating and contextualizing any such discovery, both before and throughout its confirmation. It is therefore crucial that the scientific community be prepared for this responsibility.

A call for a framework for reporting the detection of extraterrestrial life highlighted several motivating factors that merit focused preparation.³ Past claims of evidence for extraterrestrial life have engendered debate both among specialists and within the public sector alike. There is, as this call argued, also a possibility of sensationalization by the media. These factors could potentially have detrimental effects on public confidence and understanding of a life detection, such as negatively impacting broad acceptance of a positive detection or proliferating acceptance of a false-positive detection. This would be an unfortunate outcome for decades of work to ensure the scientific rigor of this discovery. However, with foresight can come preparedness.

To ensure that it is prepared for these eventualities, NASA has been sponsoring and supporting dialogues within the science community. The NASA Office of the Chief Scientist called for a workshop to “encourage the science and technology communities to have a dialogue about standards for communicating findings that may be interpreted as signs of life.”⁴ Several scientists from the astrobiological community answered the call and convened a 23-member Workshop Science Organizing Committee (SOC) to coordinate and execute the workshop.

___________________

¹ NASA Science Mars Exploration Programs, “Viking 1 & 2,” https://mars.nasa.gov/mars-exploration/missions/viking-1-2.

² National Aeronautics and Space Administration, “The NASA Astrobiology Institute Concludes Its 20-Year Tenure,” https://astrobiology.nasa.gov/nai.

³ J. Green, T. Hoehler, M. Neveu, S. Domagal-Goldman, D. Scalice, and M. Voytek, 2021, “Call for a Framework for Reporting Evidence for Life Beyond Earth,” Nature 598:575-579, https://doi.org/10.1038/s41586-021-03804-9.

⁴ “Community Report from the Biosignatures Standards of Evidence Workshop” (henceforth Community Workshop Report), p. 10.

The workshop itself, the “Biosignatures Standards of Evidence Workshop,” was jointly hosted by the Network for Life Detection (NfoLD) and the Nexus for Exoplanet System Science (NExSS), two NASA RCNs that are key stakeholders in the science of life detection. To accommodate the ongoing COVID-19 pandemic, the workshop was held virtually on July 19-22, 2021, in a “flipped” format with asynchronous sessions to accommodate time zone differences between attendees. Workshop attendees were encouraged to read preparatory materials and watch instructional videos and presentations beforehand. The workshop itself was organized into various working groups focused primarily on the biosignature assessment and reporting protocols.

In selecting workshop attendees, 355 applications were received for 80 participant slots, with ultimately 82 being selected by the SOC for balance in discipline and career stage. With the SOC and invited guests, a total of 125 participants attended the workshop including international participants. The final reported demographics for the workshop are presented in Figure 2.1.⁵

After the workshop, writing of the workshop report began in small working groups encompassing approximately 50% of the discussion participants. The initial draft of the Community Workshop Report, titled “White Paper Report from the Biosignatures Standards of Evidence Community Workshop,” was completed on October 18, 2021. This version was provided to the Committee on Astrobiology and Planetary Sciences (CAPS) for comment while the white paper authors solicited public comment for edits and endorsements of the report. The comment period ended on November 18, 2021, just as CAPS began its information-gathering efforts and deliberations.

CAPS initially reviewed the October 18, 2021, version of the report, which served as the basis for much of its information gathering. CAPS had discussions with both the leaders of the workshop, Dr. Heather Graham and Dr. Victoria Meadows, and with several participants for their perspective on the recommendations of the Community Workshop Report. The white paper authors edited the workshop report incorporating comments, releasing to CAPS a second draft of the report now titled “Community Report from the Biosignature Standards of Evidence Workshop” on February 18, 2022, which has served as the basis of this review. The summary below reflects CAPS’s understanding of the key points of the document, two of which are defined in Box 2.1.

___________________

⁵ Community Workshop Report, Table 2.

The Assessment Framework

The Community Workshop Report proposes a generalized assessment framework, composed of five separate questions, by which the scientific community can assess and accurately communicate progress and credibility in a life detection claim. These questions are not meant to be a linear progression or a strict order that life detection claims must meet, but are meant to be a generalized tool to facilitate communication between various disciplines within astrobiology and with the general public. The goal was that these questions would be adaptable regardless of the methodology and would recontextualize the search for life as a progressive endeavor rather than a definitive single moment.

The five questions that compose this generalized framework are:

1. Have you detected an authentic signal?
2. Have you adequately identified the signal?
3. Are there abiotic sources for your detection?
4. Is it likely that life would produce this expression in this environment?
5. Are there independent lines of evidence to support a biological (or non-biological) explanation?

As a demonstration of this assessment framework, the Community Workshop Report includes four worked examples of the utilization of this framework across a variety of detection disciplines within astrobiology: Early Earth, In-Situ, Remote Sensing, and a completely agnostic detection. This structure outlines a sequence of questions that an observation may go through in order to demonstrate increasing confidence that the observation is indicative of life beyond Earth. Examples given in the report (of stromatolitic features in the geologic record, organic detection on a Martian sample, and O₂ detection on exoplanets) illustrate how such a framework could be applied to determine the scientific confidence in a claimed biosignature detection.

Given the potential impact on society of the discovery of life beyond Earth, a framework such as this, that encourages the scientific community to demonstrate confidence in life detection measures and observations, to the extent that is possible, would be inherently valuable. While the examples are not fully comprehensive and there is debate about the answers to some of the questions for each example, it is impressive that the assessment framework proposed could be used across the breadth of findings, thus potentially facilitating communication of the level of confidence between various subfields. Moreover, the SOC for the Standards of Evidence Workshop did an excellent job in creating an expandable

assessment structure to allow for wide participation by a diverse audience, and for this they should be commended (see Figure 2.1).

Further considerations of the communications implications of this framework are discussed in Chapter 3.

Verification and Reporting Protocol

The Community Workshop Report also proposes a protocol for assessing, verifying, and communicating a potential life detection. This proposed reporting protocol contains concretely actionable suggestions. These recommendations include: considering the proposed reporting protocol in the mission life cycle considering statistical likelihoods and uncertainties; independent verification of a result prior to publication in addition to the standard peer-review process; communication of such results to the media.

In considering the uncertainties and statistical inferences made from these experimental claims, the Community Workshop Report has several recommendations. The first is that researchers endeavoring to detect a biosignature should collaborate with statisticians in order to ensure the robustness of their claims. This is based on the understanding that statistical inferences and inductive reasoning will likely be the core of a biosignature detection, and thus be uncertain; a statistics specialist would be able to prevent faulty statistical reasoning prior to making a life detection claim. The Community Workshop Report suggests that a Bayesian statistical methodology can be advantageous in biosignature assessment to make more robust statistical inferences. However, they also suggest that other schools of analysis should reach the same conclusion when independently applied. In any case, the assumptions behind the analysis should be clearly and explicitly stated for further evaluation. The Community Workshop Report argues that any uncertainties in the measurement be reported through confidence or credibility intervals in order to report discoveries on a spectrum of confidence.

A second recommendation in the Community Workshop Report focused on new mechanisms to disseminate a potential biosignature detection accurately. Existing collaboration networks such as NExSS and NfoLD were suggested as forums by which information on a potential discovery can be coordinated and shared for verification in addition to, or prior to, the publication review process. They also argue that modifying existing mechanisms of reporting protocols could prove effective and easy to follow.

One proposed system was a “two-factor” publication process that would simultaneously support both the discovery and the verification, highlighted by Figure 5.1 in the report (reproduced in Figure 2.2 here). With additional pools of funding to support biosignature validation, an independent corroboration of the results could produce simultaneous releases of publications for both the discovery and its validation. The other system discussed was a stepwise system where each step of the discovery and verification process would refer to a confidence level scale similar to the Assessment Framework or the Confidence of Life Detection (CoLD) scale,⁶ and where confidence intervals would be provided at each step.

___________________

⁶ J. Green, T. Hoehler, M. Neveu, S. Domagal-Goldman, D. Scalice, and M. Voytek, 2021, “Call for a Framework for Reporting Evidence for Life Beyond Earth,” Nature 598:575-579, https://doi.org/10.1038/s41586-021-03804-9.

In either case, the Community Workshop Report posited that significant obstacles exist within current reporting protocols and the modern research climate, including the nature of discovery embargoes and other similar agreements, and the fear of someone publishing a result unrightfully and gaining scientific priority. As such, the workshop attendees argued that community-led change and a change in incentives are needed to ensure that the verification process is scientifically robust and could also put a restraint on those seeking to use controversial results to bolster reputations.

Part of this incentivization would be in recognition for verification work. This could, as the Community Workshop Report suggests, take the form of mutual agreements that verification teams could be listed as co-authors on the discovery paper. This could also involve widespread consideration of how the astrobiology field considers productivity and job performance to incentivize following of the verification protocol. It could also take the form of distributing the reward and recognition of a groundbreaking discovery. The authors argue that this diffusion of responsibility would promote collaborative research.

Another part of this incentivization structure would be funding. The Community Workshop Report advocates for a responsive verification funding structure that could ensure a robust verification process without delaying progress significantly, reducing the time between initial discovery and the

verification analysis. One suggestion was a discretionary life detection verification fund that could respond to these needs, possibly including international options for a cohesive global network.

The third and final piece of this incentivization would be in encouraging more venues for communicating results to the scientific community and to the media and general public. The former would require efforts centralized within academic journals, including proposed mechanisms such as publishing reviewer reports, interim update papers on progress on the assessment framework, and null results and negative data. The latter would include training in communications and building collaborations between institutions and science journalists.

The Community Workshop Report discussed suggestions proposed at the workshop to encourage and support collaborative verification. These included:

• Dedicated sessions at scientific meetings to discuss and establish verification support mechanisms.
• Establishing a committee, ideally with international participation, to coordinate information exchanges and support the implementation of this protocol. The authors suggest that this committee could support and coordinate community verification efforts.
• Periodically assessing the evidence of life at particular targets by a group constituted for this task, perhaps by the aforementioned international committee meant to coordinate information exchanges.
• Creating databases of consensus standards in other fields.

The Community Workshop Report concludes by discussing its findings and future work that can further the discussion.