Emerging Approaches for Using, Interpreting, and Integrating ML/AI for Earth System Science
In the context of the active use of ML/AI for Earth system science, many conceptual, technical, educational, and cultural challenges remain, as well as emerging approaches and opportunities to address these challenges. One key theme raised throughout the following sessions was the importance of multidisciplinary collaboration in a concerted way, and the need to integrate Earth system and ML/AI practices and approaches to better support the scientific and technical needs of Earth system science applications. Humans are also a crucial driver of uncertainty in the future of the Earth system, and participants discussed how to learn from and integrate knowledge of social and human engineered systems into ML/AI applications.
EMERGING APPROACHES FOR USING AND INTERPRETING ML/AI
This session considered unique challenges in Earth system science that emerging ML/AI approaches could help to address. Specific approaches discussed included the integration of physics, expert knowledge, multiple modalities of data, and ML/AI techniques; explainable and interpretable AI; and data assimilation (see Box 1 for definitions of key terms).
Challenges for Using ML in Earth System Science Applications
Pierre Gentine, Columbia University, focused his remarks on identifying four science challenges for building and using ML at scale within Earth system science, with a focus on climate modeling. The first challenge is that standard ML algorithms do not respect physical laws (e.g., Rasp et al., 2018), inhibiting their use for applications to climate and other Earth science studies. One solution would be to embed physical laws (e.g., mass and energy conservation) within ML algorithms (Beucler et al., 2021a). A second challenge identified by Gentine is that ML algorithms have been trained on existing data and have a difficult time handling situations the algorithms have not encountered before, such as distributional shifts evident in modeling climate change or extreme events. One way to address this challenge would be to develop algorithms that support out-of-sample distributions, which would require strong collaborations between ML experts and domain scientists to understand the inputs and outputs of a model (Beucler et al., 2021b).
Another challenge for the community that Gentine identified is how to utilize ML tools to make more accurate climate projections. Typically in a modeling system, the sources of errors are the initial conditions, parameter interference, and model structure. For weather prediction, the data assimilation community uses observations to nudge the (incorrect) model state toward the observations when the latter are available, but this methodology is not sufficient for climate predictions and projections for which only historical observations are available, and the model is not completely accurate. Gentine suggested that the community work on developing new ML algorithms to assimilate data in order to correct structure, to better represent physical and biological processes, and to produce accurate projections (Figure 2).
Explainable and Interpretable AI for Earth System Science
Elizabeth Barnes, Colorado State University, focused her remarks on explainable and interpretable AI for Earth system science. Earth scientists already have a large set of tools to sift through data, find relationships, and make predictions, and Barnes sees ML as one additional tool to add to the existing suite. For Earth system science in particular, ML methods can be used to enhance existing research, make computation faster or cheaper, or enable knowledge discovery.
ML algorithms are often put into a “black box,” and advances in explainable and interpretable AI can help to open the “black box,” specifically for the geosciences (e.g., Ebert-Uphoff and Hilburn, 2020; McGovern et al., 2019; Toms et al., 2020). Many different explainable AI (XAI) methods exist, and Barnes showed Layerwise Relevant Propagation as one example, in which a probability value predicted by a neural network gets propagated back through the neural network to produce a heat map of the most relevant regions of the input for each prediction (Figure 3). Barnes has found this XAI method to be particularly useful because these heat maps are largely consistent with how climate scientists ask their research questions.
Beyond better understanding the Earth system, XAI can help identify problematic methods in ML algorithms, such as when neural networks trained to do image recognition get the right answer for the wrong reasons—for example, learning to recognize a copyright symbol on a series of training images of horses instead of identifying the horse itself (Lapuschkin et al., 2019). XAI can also be utilized to gauge general trust in an algorithm, choose a general approach, and enable knowledge discovery. For example, Barnes has used XAI to understand how to predict temporary slowdowns in global mean surface temperature (Labe and Barnes, 2021).
XAI methods are simplifications of an AI model and are by definition not precisely faithful to what the original model computes. Thus, the success of XAI approaches is typically assessed by human judgment, which can be difficult for Earth system applications for which the right answer is often unknown. Echoing Dueben’s remarks about benchmarking ML methods,
Barnes emphasized the importance of benchmarking XAI approaches for Earth science applications in order to understand each method’s successes and failures (e.g., Mamalakis et al., 2021, 2022). Overall, she stated that the best approach would be to use multiple AI methods to provide a more holistic understanding of a scientific problem.
Barnes noted that while XAI is useful, knowing where in an algorithm the decisions were made does not reveal how decisions were made (Rudin, 2019), and this is where interpretable AI comes in. Interpretable AI approaches are built to explicitly incorporate the decision-making process into their structure. Barnes suggested that scientists should work toward building AI models that mimic scientific human reasoning to improve intrinsic interpretability. If interpretable ML is built to be understood from start to finish, it has the potential to be easily understood by a wider audience.
Building on the ideas Gentine raised about bringing physical constraints into ML algorithms, Barnes suggested that physical knowledge should be brought into the entire AI design framework. Echoing Dueben’s comments, Barnes agreed that the Earth system science community should move away from off-the-shelf approaches, and in order to make substantial progress, Earth system scientists should be a part of the development of AI tools to fit their unique needs. For example, because the Earth system is complex and chaotic, making it difficult to predict, she noted the need for methods that extract signals when they are present instead of waste energy learning unpredictable samples (Albers and Newman, 2019, 2021; Mayer and Barnes, 2021). While many complex AI methods exist, sometimes simple approaches can be the best—for example, for uncertainty quantification (e.g., Barnes et al., 2021).
Using Data Assimilation and ML/AI for Earth System Prediction
Stephen Penny, Sofar Ocean, focused on how data assimilation can be combined with ML/AI for Earth system prediction. Given a set of noisy observations and imperfect theoretical understanding of a physical system, data assimilation is used to reconstruct a four-dimensional trajectory of the system dynamics, or a probability distribution of such trajectories, and make predictions of that system. Because of the many similarities between data assimilation and ML/AI formalisms (e.g., Abarbanel et al., 2018), there is an opportunity to use ML/AI to re-evaluate the fundamental assumptions and formulations in the field of data assimilation.
While an ultimate research goal is to solve a data assimilation and forecasting problem statement directly with one single solution, a more attainable goal in the near term is to strategically replace the most expensive components of the data assimilation cycle with ML/AI approaches. One way to do this would be to leverage “data generators,” in which the vast quantities of model simulation data and historical reconstructions of past Earth system states that have already been produced and archived could be used as data inputs to ML models. Another approach would be to try to learn model dynamics directly from observational data, though this approach would have minimum data requirements. As one example of using ML/AI in the data assimilation workflow, Penny showed how surrogate models can be used to replace an expensive forecast model with low-cost ML/AI models that can be trained in advance and updated, designed for specialized hardware, and hold a possibility of building in physical constraints (e.g., Arcomano et al., 2020; Lin and Penny, 2021; Penny et al., 2021).
Penny noted that some key challenges already have effective solutions—for example, utilizing ML/AI models that are successful at reproducing intransient properties of dynamical systems, such as the average error growth rate in a system (Penny et al., 2021; Platt et al., 2021). However, challenges in the details of scaling up ML/AI methods and handling noisy real-world data remain. In order to best utilize all available Earth science information, Penny identified a number of needs including new global optimization methods; robust ML/AI methods that are less sensitive to noise and biases in training data; the ability to scale to high dimensions; and the identification of the best types of training data.
Penny concluded with an example of the transformational potential of using data assimilation and ML/AI together. In the current data assimilation workflow (Figure 4, top), an increasing volume of observations has been used to learn and understand the Earth system, partial differential equation (PDE) models have been developed, and data assimilation has been used to combine information from observations and models. Re-analysis datasets are a product that describe the history of the system and can be used to make forecasts of the future. Moving forward, Penny showed how data assimilation, PDE models, and ML/AI could be integrated into a single approach that takes observations from the historical record and builds a scientifically validated history of the Earth system (Figure 4, bottom).
Bridging the Gap between Domain and ML/AI Scientists
Several participants emphasized the importance of bringing together domain and ML/AI scientists in order to achieve progress. Gentine raised the challenge of making connections between ML/AI experts and domain scientists, whose fields historically have been distinct with different vocabularies and limited collaboration. In Dueben’s opinion, the single most important
development to achieve progress would be to bring domain scientists and ML/AI scientists together in the next 2–5 years to cultivate ML/AI domain scientists. To underscore why domain scientists should to be part of the ML/AI conversation, Barnes noted that for Earth system applications, scientists would rather trade 1 percent of accuracy for getting the answer right for the right reasons, which is not typically factored into the way the commercial sector is using ML/AI tools and calls for a new way of thinking.
With respect to education, Barnes suggested moving beyond ML/AI short courses for early career scientists and fully embracing data science as fundamental to Earth system science education. In practice, that could include incorporating data science and statistics into academic classes and structures in a fundamental way rather than as an afterthought. Barnes offered a vision in which all scientists across the career spectrum would have basic working knowledge of data science tools by offering and incentivizing training for all career levels. Dueben added that in order to train ML domain scientists, the Earth science community could train domain scientists to
become ML/AI experts, attract ML/AI scientists to work on Earth science applications, and work with the private sector to facilitate this exchange. Gentine shared the example of the new NSF Center for Learning the Earth with Artificial Intelligence and Physics6 that aims to forge a new climate data science discipline and develop a convergence approach between education and research. The other objective of this center is to increase the representation of underrepresented minorities in climate data science in order to have the broadest reach required to tackle climate change, which is affecting everyone.
EMERGING OPPORTUNITIES FROM SOCIAL AND HUMAN ENGINEERED SYSTEMS
Panelists in this session discussed opportunities for using ML/AI to understand social and human engineered systems, and the prospects for using ML/AI to integrate social and human engineered system science into Earth system science. Panelists were specifically asked to provide examples of emerging or future opportunities for using ML/AI to understand human and engineered systems, explore the prospects for using ML/AI to integrate social and human engineered system science into Earth system science, and consider how ML/AI could bridge the gap between physical and social sciences to advance Earth system science.
Auroop Ganguly, Northeastern University and Pacific Northwest National Laboratory (PNNL), discussed how ML/AI in combination with science, engineering, social science theory, and policy frameworks can tackle the challenging complexities at the intersection of coupled natural, human-engineered, and social systems. Trends and natural variability in Earth and natural systems impact and are impacted by human engineered systems (e.g., infrastructure, energy, transportation), and natural and human engineered systems interact with social or human systems (e.g., financial services, regulatory frameworks, governance mechanisms). Physics-guided, uncertainty aware, spatiotemporal ML methods have shown promise in natural and Earth systems in translating predictive understanding to stakeholder-relevant products (e.g., Reichstein et al., 2019). Probabilistic, graphical ML methods have the potential to interlink infrastructure and social systems (e.g., Buldyrev et al., 2010). Agent-based models, gaming exercises, and participatory modeling have made strides in the human and social sciences (e.g., Ganguly et al., 2018; Pagan and Dörfler, 2019). However, Ganguly argued that holistic risk and resilience frameworks that consider the flow of uncertainty and causality are still lacking.
Ganguly shared the example of risQ,7 a climate analytics startup (subsequently acquired by a Fortune 500 company) that has been attempting to develop real-world solutions at the intersection of climate and cities. risQ develops three product suites: (1) physics-guided data sciences to ground climate model simulations and observations in the needs of urban stakeholders, (2) climate-risk-to-investment benchmarking through economics informed data sciences, and (3) climate justice products that examine social impacts on underserved populations based on geospatial data sciences. Ganguly shared a lesson learned through his experience with risQ: Solutions should be developed in tandem with incentives and social impacts.
The work of Abigail Snyder, PNNL, looks at integrated human-Earth systems modeling through the Global Change Intersectoral Modeling System,8 which includes a multisector model
___________________
6 See https://leap.columbia.edu
7 See https://www.risq.io
8 See https://gcims.pnnl.gov/global-change-intersectoral-modeling-system
that integrates representations of both human and physical systems and their interactions. Snyder emphasized that humans are the largest driver of uncertainty in the future of the Earth system because humans are affected by and can affect the dynamics of the Earth system. There has been extensive work on how human decisions and systems impact the physical Earth system, and how changes in the environment impact human systems and behaviors. Two-way feedbacks and coupling between human and Earth systems are more challenging to study, and ML/AI tools offer the opportunity to develop new models and datasets to better understand these interconnections.
Snyder shared several emerging opportunities for ML/AI tools to represent interactions between human and Earth systems. The first is the emulation of Earth system model outputs and other human-relevant process models for more computationally efficient coupling or integration. ML can be a helpful tool to represent systems for which researchers lack hypotheses in the structure of input-output relationships. Snyder echoed other workshop speakers by discussing the value of developing these system representations in collaboration with domain experts and the importance of interpretable and trustworthy AI when doing human-relevant modeling. Another emerging approach would be to use unsupervised learning on large ensembles of model results for knowledge discovery—an area of focus for the multisector research community.
David Rolnick, McGill University, distilled many of the session ideas in a recent report (Rolnick et al., 2022; Figure 5) about opportunities for AI in advancing climate science, mitigation, and adaptation, part of the greater Climate Change AI global initiative.9 AI is a powerful tool that can distill raw data into actionable information, improve predictions, optimize complex systems, and accelerate scientific modeling and discovery. Beyond applying existing AI tools to climate problems, climate-relevant problems can also be valuable in motivating fundamental innovations in AI, creating valuable use cases for hybrid physical models, transfer learning, interpretable and causal ML, and uncertainty quantification. However, Rolnick cautioned that AI is not applicable in every context, and simple methods may be sufficient in some cases. AI may provide incorrect information if the question has not been framed properly, highlighting the importance of developing close collaborations among experts in AI, experts in Earth sciences, and additional stakeholders.
___________________
Jennifer Chayes, University of California, Berkeley, described a vision for creating an integrated ML climate platform that includes not only physical laws and technical solutions but also behavioral changes, economic constraints, and geopolitical factors (Figure 6). Chayes highlighted the “Jupyter meets the Earth” project funded by the NSF EarthCube program10 that is using geoscience use cases to drive the advancement of computational technologies for interactive geoscience research characterized by large datasets and computationally complex models. There are many examples of applying ML/AI tools to, for example, identify extreme climate events, predict the economics of renewable energy, and understand the economic impacts of climate change. Chayes offered a vision that would bring together these different applications in a single platform where users would have similar sets of tools and could utilize different model outputs as inputs, and, for example, use reinforcement learning—a type of ML—to look at behavioral changes. Furthermore, such a platform would have a low code/no code front end that would allow decision makers to have the best available predictions about a type of policy intervention.
Ganguly explained that scientists are interested in characterizing uncertainty for a number of reasons: to support the credibility of a discovery, to help make risk-informed decisions, and to translate scientific outcomes to multi-stakeholder acceptance. In an effort to better understand the large uncertainties in human behavior, Chayes suggested using massive multiplayer games to see how individuals react and make decisions, including the unintended consequences of a policy intervention. Ganguly agreed that further opportunities exist to use participatory modeling and games in the climate action space to involve stakeholders and early adopters directly.
More broadly, the panelists agreed that attention should be paid to the ways problems at the intersection of human and Earth systems are being framed. Rolnick pointed out that there are AI and climate problems not being adequately studied because problem definitions and priorities can be skewed by the people involved—for example, more attention has been given to using AI to forecast and respond to forest fires, which often affect the Global North, than to
___________________
10 See https://www.nsf.gov/awardsearch/showAward?AWD_ID=1928374
forecasting and preventing locust outbreaks, which tend to affect the Global South. Snyder added that in order to address biases in the regions scientists choose to study, large ensembles of scenario modeling could be useful, in which perturbations across different variables in different sectors could generate a large ensemble of model outputs, upon which unsupervised ML methods can be deployed. Snyder suggested that instead of using models that do not currently represent all types of processes simply as tools for prediction, models can be thought of as tools to provide an array of possible futures that interact with each other.
This page intentionally left blank.
