4

Relationship to the Scientific Process

The Community Report from the Biosignatures Standards of Evidence Workshop (henceforth Community Workshop Report) advocates for a new reporting protocol that would facilitate the communication and the verification of a potential biosignature. The reporting protocol has two separate but interdependent components. First is an assessment framework to serve as a common language for communications and to guide astrobiology research along a prescribed path of verifying a detection claim. The second is a verification and reporting protocol by which a potential biosignature claim is verified and then reported. The Community Workshop Report continues by recommending various mechanisms—from augmented funding opportunities to changes in the publication structure—to implement these proposed solutions at a community level.

The goal of these proposals is to address possible problems with the most objective interpretation of a potential life detection claim. Past claims have demonstrated that these issues exist, and the Community Workshop Report points to several examples including the Labeled Release (LR) Experiment results from the National Aeronautics and Space Administration (NASA) Viking¹ missions, the Allan Hills meteorite ALH84001² claim of Martian life, and most recently the community debate regarding the claimed detection of phosphine on Venus.³ The Community Workshop Report sets aside the healthy scientific debate that these discoveries generated within the astrobiology community and argues that these examples illustrate a need for systematically increasing the robustness of claims and improving communication both within and outside the field.

The committee concurs with the Community Workshop Report that these are laudable goals. Vetting a potential life detection claim and communicating any potential claim more precisely—which could be facilitated by a robust and refined confidence scale—are problems with the existing process that need to be addressed. However, excitement and scientific debate about findings is healthy and a normal part of the scientific process. If adopted, the proposed framework may have substantial impacts, both positive and negative, to the scientific process and the scientific community depending on the extent to which the framework is adopted (e.g., voluntarily or mandated for federal funding). A discussion of the negative ramifications is lacking in the Community Workshop Report, as such may not have been part of the workshop’s scope. In the spirit of this report’s charge, this chapter will specifically discuss the potential implications that these proposals may have on the scientific process, which we mean to include the scientific method including the consideration of multiple working hypotheses by both individual research groups and the broader community of scientists within a research area.

___________________

¹ G.V. Levin and P.A. Straat, 1976, “Viking Labeled Release Biology Experiment: Interim Results,” Science 194(4271):1322-1329, https://doi.org/10.1126/science.194.4271.1322.

² D.S. McKay, E.K. Gibson, Jr., K.L. Thomas-Keprta, H. Vali, C.S. Romanek, S.J. Clemett, X.D.F. Chillier, C.R. Maechling, and R.N. Zare, 1996, “Search for Past Life on Mars: Possible Relic Biogenic Activity in Martian Meteorite ALH84001,” Science 273(5277):924-930, https://doi.org/10.1126/science.273.5277.924.

³ J.S. Greaves, A. Richards, W. Bains, P.B. Rimmer, H. Sagawa, D.L. Clements, S. Seager, J.J. Petkowski, C. Sousa-Silva, S. Ranjan, and E. Drabek-Maunder, 2021, “Phosphine Gas in the Cloud Decks of Venus,” Nature Astronomy 5(7):655-664, https://doi.org/10.1038/s41550-020-1174-4.

POTENTIAL IMPACTS OF THE PROPOSED FRAMEWORK TO INNOVATION

Innovation is at the heart of science. Therefore, preserving the scientific community’s ability to freely develop new ideas and techniques must be the top priority of any proposed change to the scientific process. This extends to every step in the scientific process.

Innovation is promoted by encouraging the free exploration of ideas and requires flexibility in the scientific process. The commonly taught scientific method⁴ of observation, research, hypothesis, experiment, analysis, and report are, in practice, not always done sequentially or independently. Rather all steps are engaged interdependently and sometimes almost simultaneously. Data-driven and data-limited fields, like planetary science and select biological sciences, more commonly employ an approach of “multiple working hypotheses.”⁵ Furthermore, the ensuing scientific debate that follows the reporting of results is a fundamental aspect of both scientific correction and innovation. As such, science requires flexibility in how researchers approach problems. Proposals that would reduce flexibility in the scientific process should be subject to strict scrutiny.

One potential impact that the assessment framework and reporting protocol as proposed by the Community Workshop Report may have is that confining the search for life and biosignature analysis to a defined structure could inadvertently stifle innovation and debate, especially at the science-development (i.e., proposal) level. The potential interpretation of the five questions of the protocol as a required step-by-step sequence of the assessment of a possible biosignature detection forfeits the advantages afforded by a more flexible structure. If applied to the proposal review process, overly structured frameworks could limit instrumentation and detection techniques by compelling developers to abide by a prescriptive checklist, even when some scientific details may be investigated through peer-review after the initiation of a scientific study. Novel detection techniques or methodologies could lose out in proposal competitions should they not neatly follow the proposed structure. This preference toward well-established techniques or methods could lead to missed opportunities for significant advances or, in the worst case, a missed detection. Other impacts may be less immediate but similarly problematic. Such an assessment framework and reporting protocol could make it difficult to introduce biosignatures that are not universally agreed upon. In fact, as justification for the need for the proposed framework and protocol, the report cites the two-decade long discussion of one set of potential Archean microfossils.^6,7 In doing so, the workshop report implies, perhaps incorrectly, that this specific case is closed, a position that would limit further study of the structures. There is presently active research on other biosignatures that similarly would not fit in the framework as presented. These include, for example, the candidate Precambrian fossil worm burrows that were used to suggest the existence of multicellular animal life more than a billion years ago,⁸ as well as the often biotic microtubules in basaltic glass from both modern and ancient

___________________

⁴ Editors of Encyclopaedia Britannica, 2021, “Scientific Method,” Encyclopedia Britannica, October 15, https://www.britannica.com/science/scientific-method.

⁵ L.P. Elliott and B.W. Brook, 2007, “Revisiting Chamberlin: Multiple Working Hypotheses for the 21st Century,” BioScience 57(7):608-614, https://doi.org/10.1641/B570708.

⁶ J.W. Schopf, 1993, “Microfossils of the Early Archean Apex Chert: New Evidence of the Antiquity of Life,” Science 260:640-646, https://www.science.org/doi/10.1126/science.260.5108.640.

⁷ M.D. Brasier, O.R. Green, A.P. Jephcoat, A.K. Kleppe, M.J. Van Kranendonk, J.F. Lindsay, A. Steele, and N.V. Grassineau, 2002, “Questioning the Evidence for Earth’s Oldest Fossils,” Nature 416:76-81, https://doi.org/10.1038/416076a.

⁸ A. Seilacher, P.K. Bose, and F. Pflueger, 1998, “Triploblastic Animals More Than 1 Billion Years Ago: Trace Fossil Evidence from India,” Science 282(5386):80-83, https://doi.org/10.1126/science.282.5386.80.

seafloor locations.^9,10 A confining framework could also curtail claims based on a biosignature introduced in parallel with or simultaneously with the detection itself, for instance magnetite in ALH84001.¹¹ The committee notes that subsequent and still ongoing research and debate on these topics have had a positive impact on the discipline of Astrobiology, and continues to stimulate innovation, as the scientific process is meant to work.

The report implies there is a critical need for consensus to avoid long-term discussions. In contrast, the committee assessment is that open, frank, and sometimes protracted scientific debate is critical to scientific progress. A highly structured framework could limit research involving any nascent or debatable biosignature, regardless of whether or not the phenomenon is authentically biogenic.

Overall, in considering the Community Workshop Report, the committee found that the worked examples were demonstrative of the breadth of the application of this potential framework, and, therefore, the inclusion of these examples was helpful for starting to envision how the community might use the framework or a verification process and what the implications, intended or unintended, would be. However, the committee found that the examples were neither comprehensive nor complete, and thus in the committee’s view these may underrepresent significant relevant discussions within the astrobiology community on these topics.

The Community Workshop Report argues (with reasonable grounds) that the first detection of an extraterrestrial biosignature will likely be ambiguous and require significant follow-on work. An approach that would encourage more innovation and more flexible thinking could be that all five of the proposed assessment framework questions are useful metrics that could be approached almost simultaneously. Arguably, this would, to some degree, defeat the purpose of the confidence levels presented in the Community Workshop Report. It could also be argued that approaching all five of these proposed questions in any order is just the execution of the standard scientific process rather than a new assessment framework for this purpose.

In order to avoid these issues and maintain flexibility in the pursuit of a biosignature detection, the committee finds that a more generalized and less structured assessment framework would be more beneficial, useful as a communications tool rather than a prescriptive guide. Any proposed framework should be kept as general as possible, so as not to curtail the introduction of new candidate biosignatures or limit the development of novel biosignature detection techniques.

Proposed Application of the Framework to Mission and Proposal Development

The Community Workshop Report advocates for the use of the proposed framework throughout the lifecycle of a mission. This begins in pre-Phase A, where mission designers would consider which of the proposed assessment framework questions the mission would address. Also proposed is the creation of a biosignature verification plan, which would outline potential parallel lines of investigation that can be used to assess a potential life detection claim.

The proposed framework would also, as the Community Workshop Report argues, be useful in determining technologies and prioritize development funding for mission instrumentation. The framework would be used in deliberation of top-level mission architecture decisions, which would then logically determine instrumentation requirements. As the Community Workshop Report claims, “Such

___________________

⁹ H. Furnes, H. Staudigel, I.H. Thorseth, T. Torsvik, K. Muehlenbachs, and O. Tumyr, 2001, “Bioalteration of Basaltic Glass in the Oceanic Crust,” Geochemistry, Geophysics, Geosystems 2(8):1049, https://doi.org/10.1029/2000GC000150.

¹⁰ N. McLoughlin, M.D. Brasier, D. Wacey, O.R. Green, and R.S. Perry, 2007, “On Biogenicity Criteria for Endolithic Microborings on Early Earth and Beyond,” Astrobiology 7(1):10-26, https://doi.org/10.1089/ast.2006.0122.

¹¹ D.S. McKay, E.K. Gibson, Jr., K.L. Thomas-Keprta, H. Vali, C.S. Romanek, S.J. Clemett, X.D.F. Chillier, C.R. Maechling, and R.N. Zare, 1996, “Search for Past Life on Mars: Possible Relic Biogenic Activity in Martian Meteorite ALH84001,” Science 273(5277):924-930, https://doi.org/10.1126/science.273.5277.924.

requirements are used as benchmarks for performance for the mission and its instrumentation throughout the design, fabrication, assembly, integration, and testing phases.”¹²

The proposed framework would then be used in mission operation planning to improve the confidence in a potential biosignature detection. While this is highly dependent on the specifics of the methodology of the life detection claim and its associated instruments, applying the questions of the assessment framework would encourage missions to pursue independently verifiable but interdependently consistent lines of investigation. This would continue into extended mission operations and follow-on mission design and planning.

The Community Workshop Report also suggests that using this assessment framework would be beneficial in evaluating a proposal for potential funding.¹³ For the reasons previously mentioned, an assessment framework if directly used as evaluation criteria during an award process could limit emerging approaches by having their development abide by a prescriptive checklist. The role of an assessment framework should be to facilitate communication, not to be a tool by which to evaluate proposals.

The Community Workshop Report appears to be attempting to avoid potential fallout from sensational claims that may not prove viable or may be even damaging to the field, funding agencies, and other concerned players. Mitigating such fallout from unsubstantiated and/or sensational claims is understandable, but such mitigation should not occur at the proposal stage before the science has even begun which may limit the scientific process.

This is not the only instance where the proposals contained within the Community Workshop Report have the potential to create gatekeepers in the scientific process. We discuss those ideas further in the following section.

POTENTIAL IMPACTS OF THE PROPOSED IDEAL VERIFICATION PROTOCOLS

The Community Workshop Report discusses an ideal community-based biosignature verification and reporting protocol, which, it advocates, the community would voluntarily abide by. The ultimate goal for such a reporting protocol would be that the scientists and the general public understand the context and confidence in which a potential biosignature claim is made. As stated earlier, the committee agrees that this is an issue that the community needs to address. However, there are unintended consequences to the specific verification and reporting protocol as proposed by the Community Workshop Report.

Parallel Validation Tracks to Publications and Funding Sources

The Community Workshop Report asserts “Life detection will … not be an instantaneous process, and it is unlikely to be unambiguous—yet it is a high-stakes scientific achievement that will garner an enormous amount of public interest”¹⁴ and promulgates the idea that life detection requires a special level of verification treatment. In response to this need, the document proposes adoption of a “Scientific Process + Validation Track” (see Figure 2.2) in which a submitted life detection paper should

___________________

¹² Community Workshop Report, Section 4.3.

¹³Ibid.

¹⁴ Community Workshop Report, Executive Summary.

invoke a parallel funded verification effort that leads to publications in tandem. It should be noted that, as presented, this proposed process is distinct from the current accepted scientific process of an initial peer-reviewed publication, which is followed by subsequent follow-on research and publications by that group or other independent groups seeking to expand on, confirm, or refute the initial findings.

The idea of a “Scientific Process + Validation Track” could be perceived as a logical and measured response given the maxim “extraordinary claims require extraordinary evidence.”¹⁵ The committee agrees wholeheartedly with the Community Workshop Report that life detection is not likely to be instantaneous, not likely to be unambiguous, and will garner substantial public attention. These circumstances may even merit a particular level of caution within the scientific community in dealing with a potential life detection claim. However, the committee does not believe that such a special parallel validation/verification process is warranted.

The committee differs with the Community Workshop Report’s assumption that life detection research is especially high stakes or that any single publication is of such a high stake that a procedural departure from standard scientific practice is justified. Indeed, life detection would be a profound discovery, among the most profound in history. Nevertheless, the committee considers that “high stakes” refers to a discrete situation with significant risk that leads to either permanent gain or loss of significant advantage or resources, depending on the outcome. While individual authors may in the short term lose or gain prestige and perhaps funding, there are no lives at stake nor significant national resources. The risk of concern appears to primarily be reputational or with regards to the public trust, which does not justify such a departure from standard scientific processes but rather highlights a need for better science communication (see Chapter 3).

The Community Workshop Report states:

The proposed process and stated desired outcome imprecisely conflates two aspects of scientific discovery: the development of a consensus body of knowledge in a new area (typically a multi-year to multi-decade process) and the public communication of scientific progress associated with specific publication. It is important to recognize—and communicate to the public—that an initial publication is often the start of a scientific process rather than the end.

In considering the creation of funding lines for vetting studies and collaborations, the Community Workshop Report compares proposed life detection verification studies to “short term rapid response funds managed by agencies such as the National Science Foundation (NSF) and the International Ocean Discovery Program (IODP) that can direct funds towards immediate needs (e.g., in the event of a geohazard).” However, the analogy does not fully hold. These NSF and IODP funding lines support research into events that have “a severe urgency with regard to availability of, or access to, data, facilities or specialized equipment, including quick-response research on natural or anthropogenic disasters and similar unanticipated events.”¹⁷ Additionally, funding calls, evaluations, and responses rarely are speedy in spite of best intentions. Such efforts to speed scientific discovery may be misguided, unless there is truly a time-sensitive or possibly ephemeral phenomenon related to life detection (e.g., a

___________________

¹⁵ C. Sagan, A. Druyan, and S. Soter, 1980, “Cosmos: A Personal Voyage,” Episode 12 in Encyclopaedia Galactica, Public Broadcasting Service, December 14.

¹⁶ Community Workshop Report, p. 62.

¹⁷ National Science Foundation, “NSF Organizations: National Science Board,” Chapter II, Section E, Subsection 1 in Proposal and Award Policies and Procedures Guide, NSF 20-1, OMB Control Number 3145-0058, https://www.nsf.gov/pubs/policydocs/pappg20_1/pappg_2.jsp#IIE1.

technosignature). Furthermore, rapid response funding to non-urgent topics tends to limit rather than expand the community level of participation in the response.

Simultaneity and rapidity do not always generate the best science. Indeed, in some cases, further development of instrumentation, methods, or conceptual understanding is required. Universal agreement about all aspects of a given publication is often elusive, given that much publication happens at a time of discovery when understanding may yet be immature. Indeed, diversity of thought (multiple working hypotheses) and a dialectic between acceptance and skepticism is a hallmark of properly performed science.

Even a claim later shown to be incorrect can be a contribution that pushes the field in new directions. Though used as an example to rationalize the framework, the claim of life in ALH84001¹⁸ is a perfect example of this. That discovery paper laid out five lines of evidence for the life claim, asserting that collectively, abiotic origins for all five were unlikely. It stimulated a prodigious line of subsequent research into microfossils and microfossil-like structures, oxide and sulfide as potential biomarkers, and organic carbon generation in mineralizing systems have opened new pathways for understanding geologic processes and rock-hosted habitats.¹⁹ In the end, further work demonstrated that the life explanation was, while not disproven, also not uniquely biological. This life detection claim was an important spur for new directions of inquiry and stands as an example of the scientific process working.

The Community Workshop Report outlines several aspects of an incentivization plan that calls for changes in the academic and research climate, including new ways of sharing authorships for both discoveries and verification activities, alternative approaches for evaluating productivity within the field, a responsive verification funding structure, and potential changes to the publication process in coordination with publishers. The authors suggest that because traditional productivity assessments in academic and research settings are focused on quantity and impact of publications, such a climate might discourage scientists from participating in the proposed verification process, which suggests that authorship could be shared between discovery and verification teams, thus dampening the impact of a discovery publication on its own. Furthermore, there is potentially a growing trend within academic institutions, and federal funding agencies, to place more value on those scientific publications that garner public media attention. The committee hopes that changes within the scientific culture will eventually be borne out within academic and research institutions and does not think that following the processes described in the Report will enhance the work to change academic culture.

The Community Workshop Report also recommends that publication disparities between discovery papers and those that provide follow-up research, verification results, and null hypothesis testing contribute to a culture in which verification efforts are not as highly valued. The authors suggest collaboration between the scientific community and publishers to accommodate publication of the caveats and nuances of life detection claims, elevating follow-on publications to the level of the initial publication, and the addition of new publication types that allow for interim progress publications and null hypothesis results. The committee agrees that such publications are valuable to the scientific endeavor in general, as is evidenced by the Community Workshop Report’s extensive citation of such publications related to the worked examples (e.g., ALH84001, LR experiment). The committee does not find that there are specific barriers within the realm of scientific journal publishers that prohibit these types of publications. The committee, however, does not agree that “incentives for reporting life detection to the scientific community must be centralized at the journal level.”

___________________

¹⁸ D.S. McKay, E.K. Gibson, Jr., K.L. Thomas-Keprta, H. Vali, C.S. Romanek, S.J. Clemett, X.D.F. Chillier, C.R. Maechling, and R.N. Zare, 1996, “Search for Past Life on Mars: Possible Relic Biogenic Activity in Martian Meteorite ALH84001,” Science 273(5277):924-930, https://doi.org/10.1126/science.273.5277.924.

¹⁹ C.Q. Choi, 2016, “Mars Life? 20 Years Later, Debate Over Meteorite Continues,” Space, https://www.space.com/33690-allen-hills-mars-meteorite-alien-life-20-years.html.

Proposed Independent Committees and Reviewers to Assess Evidence of Life

The Community Workshop Report details further potential actionable steps to implement an ideal biosignature verification and reporting protocol. Among these proposals are “forming an independent international committee to help coordinate information exchange and verification efforts” and “scientists could also partner with journals to provide specialized subject matter experts for peer review.”²⁰ These are all meant, as the Community Workshop Report describes, to promote increased transparency and produce robust science. The committee is highly skeptical these proposed actions will yield the desired results.

First, volunteering to abide by such a process is disadvantageous for both early acceptors and for those who decline to participate. Those who volunteer to abide by these protocols will have their work delayed, potentially impacting community standing or career prospects such as in cases of tenure and promotion review. Those that do not volunteer could see their work viewed as less worthy of consideration or scientific attention. Furthermore, the proposal that funding, as discussed in the previous section, be contingent on the utilization of this framework calls into question whether this acceptance is truly voluntary.

Moreover, the committee’s reservations regarding the proposed international committee and vetted reviewer list stem from a standpoint of equity, inclusivity, and the prevention of bias. Despite efforts to increase overall diversity, equity, and inclusivity, problems continue to persist and the space science community is still actively addressing these issues. As an example, NASA and some NSF observatories have implemented dual-anonymous review procedures for its proposal merit review processes in order to prevent bias and promote equity in proposal review.²¹

The notion of creating an international committee to coordinate verification activities or assess the totality of evidence for life on specific targets is worrying in that it has the capacity to concentrate power in the hands of a few intellectual gatekeepers. The same concerns are held for the proposal for a slate of “specialized subject-matter experts for peer review.” This seems in contrast to the ideal of the free exchange of ideas and an inclusive and equitable scientific environment as well as the fundamental tenets of the peer-review process. It raises questions of how such a committee or reviewers could be composed in a way that ensures diversity of thought and of demographics and of how it could be free of biases where, since life detection claims are at the moment quite rare events, anonymity cannot be guaranteed. This has the potential to negatively impact underrepresented, less well-established, and early-career scientists, as seen in non-dual, anonymous review procedures.²²

These steps are proposed with the goal of combatting disinformation. Too much early information control is not an outcome that should be sought for similar reasons as above in that it creates a situation where there exists a group with the capacity to decide what information to control and what should be released. The committee does consider efforts to improve public communication of nuance in discovery claims as necessary and laudable, as discussed in Chapter 3. However, this is problematic and, if implemented by a federal agency like NASA, could result in legal challenges. Though disinformation is a problem, this is better confronted openly by training scientists to be better communicators about the limitations of their own results and to better refute poor science in the public arena, not controlling when scientists should talk to the media.

This is not to say that the committee disagrees with the notion of periodic assessment by the community. The committee is supportive of the longer-term verification and validation activities recommended (e.g., by review efforts and periodic community discussions to evaluate progress in the science and methodology of extraterrestrial life detection). Reflective assessment on the state of a field, explicit articulation of unknowns, rigorous review of papers with key findings, and development of

___________________

²⁰ Community Workshop Report, p. 62.

²¹ National Academies of Sciences, Engineering, and Medicine, 2021, Pathways to Discovery in Astronomy and Astrophysics for the 2020s, Washington, DC: The National Academies Press, https://doi.org/10.17226/26141.

²² A. Witze, 2019, “NASA Changes How It Divvies Up Telescope Time to Reduce Gender Bias,” Nature 571:156, https://doi.org/10.1038/d41586-019-02064-y.

priorities for needed work are valuable activities. The Community Workshop Report’s recommendation for “Periodic review and assessment of the totality of evidence for life on specific targets” could act as a mechanism for scholarly review from diverse perspectives, preventing such gatekeeping and allowing the community to build support and agreement for a scientific result naturally.

In that the committee finds the core assumption unmerited, we believe that the Community Workshop Report’s conclusion that a new external coordinating entity is needed is similarly unjustified. As stated previously, the committee concurs that life detection would be a profound history-changing discovery with far-reaching impacts and that it merits a certain degree of caution. However, in the committee’s view, it is difficult to envision a single publication that is of such a high stake that it merits the risk of creating further gatekeeping in the scientific process.

CONCLUDING REMARKS

The committee does consider the assessment framework an excellent first step and a foundation on which future community discussions can build, this Community Workshop Report included. However, by the committee’s interpretation, the Community Workshop Report seems to imply that the current scientific method with regards to biosignature detection and validation is inadequate to some extent and, therefore, requires oversight. The committee disagrees with this implication and the consequent recommendations from the Community Workshop Report. The scientific community must recognize that all assessment frameworks within its field are inherently changeable and may be discarded if the community does not broadly accept its implementation. Therefore, it is incumbent on the field as a whole to continue to develop the idea of this assessment framework, as a way to both adapt it to the ever evolving scientific landscape and to continue to build community awareness and conversation. Such an effort might be executed on a timescale and via a process similar to the Planetary Science and Astrobiology Decadal Survey or Astrobiology Strategy development efforts.

Appendixes

This page intentionally left blank.