Summary
The Clean Air Act (CAA) (42 U.S.C. 7408) requires the U.S. Environmental Protection Agency (EPA) to set National Ambient Air Quality Standards (NAAQS) for ambient air pollutants, or “criteria pollutants,” that are reasonably expected to present a danger to public health or welfare. The term “criteria pollutant” derives from a CAA provision (Section 108(a)(2)) that directs EPA to review and document the scientific criteria and develop the scientific foundation for the NAAQS. That takes place during what is currently referred to as an Integrated Science Assessment (ISA), the general framework for which is defined in EPA’s Preamble to the Integrated Science Assessments, published in 2015.1 EPA conducts ISAs to evaluate the scientific evidence used to determine health and welfare effects related to criteria pollutants. Examples of human health effects that may be related to exposure to pollutants include respiratory, cardiovascular, neurological, birth outcome, or developmental effects. The impact on public welfare includes effects on climate, visibility, manmade materials, vegetation, wildlife, and other ecological attributes. The six criteria pollutants, the provisions of identification for which are provided in the CAA, are carbon monoxide, lead, nitrogen oxides, sulfur oxides, particulate matter, and ozone.
The framework, as articulated in the Preamble, outlines the essential processes and considerations used currently to evaluate, integrate, and synthesize the body of scientific evidence underlying causal determinations for criteria pollutants. It provides a broad overview of most steps of the causal determination process, integration of evidence across study types and disciplines, evaluation of evidence for health and welfare outcomes, and characterization of the weight of evidence for health and welfare effects to determine the causal category for the pollutant and each of the examined outcomes. Five causal categories are defined in the Preamble—ranging from “causal relationship” to “not likely to be a causal relationship.” These are used to classify the relationship of a criteria pollutant and its associated health and welfare outcomes. Causal categories are determined by a weight of evidence approach that combines review of the scientific literature with expert judgment built on a set of considerations known as the Bradford Hill aspects of association.
___________________
1 See https://cfpub.epa.gov/ncea/isa/recordisplay.cfm?deid=310244 (accessed June 1, 2022).
Numerous causal inference design and analysis tools exist, with many new advances since the 1970s. Critics have questioned the use of some of these tools for causal assessment in NAAQS reviews. Given the importance of ISAs in establishing the NAAQS, the EPA Office of Research and Development requested the National Academies of Sciences, Engineering, and Medicine (the National Academies) to consider existing frameworks and new advances and tools for integrating, documenting, and evaluating scientific evidence for assessing causality of health and welfare effects, describe how EPA’s methods for classifying the weight of evidence for causation (i.e., what EPA terms “causal determinations”) might be refined, and make recommendations related to the development and use of a future causal determination framework.
The National Academies convened an ad hoc committee of experts to respond to EPA’s request. The committee was asked to suggest the types of evidence most useful for forming causal determinations and the emerging tools and approaches for integrating and synthesizing evidence across studies and scientific disciplines that might be applied in the future. It was asked to identify issues concerning potential confounders (i.e., factors other than the pollutant of interest that are associated with both the pollutant and the outcome and not on the causal pathway between pollutant and outcome) that EPA might consider when assessing causality for an individual criteria pollutant, including those that are part of a complex atmospheric pollutant mixture. Furthermore, the committee was asked to consider whether a single framework and set of practices is appropriate for assessing causality for both health and welfare effects. This resulting report includes recommendations related to the development and use of causal determination frameworks for causal determinations as part of ISAs and describes priority research needed to improve those frameworks in the future.
In addition to the current causal determination framework, the committee reviewed nine other frameworks that are or have been used for determining causal relationships, or include major aspects of that process. Alternative methods to assess causality were also reviewed. The committee concluded that the fundamental structure of the weight of evidence approach described in the 2015 Preamble allows effective determination of causality for both health and welfare effects. Although the committee was not asked to evaluate individual ISAs, it did consider tools and processes to support causal determinations that were introduced in ISAs conducted after 2015. Those appear scientifically robust although not formally integrated into a causal determination. Based on its assessments, the committee concluded that the Preamble’s framework could be improved by providing additional detail regarding how to conduct the various steps in the ISA process. More detailed guidance can improve confidence in the scientific foundation for the setting of NAAQS.
THE WEIGHT OF EVIDENCE APPROACH
Critics of the ISA causal determination framework describe a lack of a systematic review criteria for selecting, evaluating, and weighting individual studies for inclusion in the weight of evidence approach. There are concerns that without a systematic criteria, sources of bias could be overlooked. Some critics suggest that studies selected for inclusion in the ISA should apply models that yield invariant results over a wide variety of settings and hypothetical interventions. Other concerns have been raised regarding whether the body of evidence examined supports the hypothesis that reducing exposure reduces risk of effects. The causal categories themselves have been described by some critics as lacking clear and testable boundaries, and that sources of bias that might affect causal determinations could be overlooked following current methodologies. Some concerns appear to be more closely related to specific applications of the framework during the conduct of a particular ISA rather than the framework itself, or about processes that define aspects of an individual ISA that occur prior to its conduct (e.g., the identification of the scientific questions in the Integrated Review Plan [IRP] that will be addressed in the ISA). Critics have also raised concern that not
enough attention is paid to at-risk populations or welfare endpoints related to sensitive ecosystems or endangered species. Additionally, there are calls for the selection of individual studies based on specific study designs, but such studies have not been conducted. On the other hand, many reviews of the framework demonstrate support for the overall weight of evidence approach and the five causal categories that have been refined over many review cycles. Suggested alternatives to the current weight of evidence approach also have issues and have not been subjected to any formalized consensus evaluation. The ISA causal determination is not a procedure that can be tested objectively or evaluated against the ground truth.
In consideration of the weight of evidence approach described in the Preamble, the approaches used in other causal determinations, and a variety of alternative methods to assess causality including those advocated by critics of the current causal determination framework, the committee concludes the following:
Conclusion: A single study will rarely definitively and comprehensively address issues associated with the determination of causality that are examined in ISAs. A weight of evidence approach—combining assessment of study quality with expert judgment—allows EPA to draw conclusions that integrate scientific findings across multiple study designs and disciplines, as required by the CAA. Increased transparency in how evidence is integrated would improve confidence and understanding of the ISA causal determinations and other conclusions.
Drawing causal determinations in complex air pollution settings is challenging. Many study designs are used to study health and welfare effects, each with strengths and weaknesses. A weight of evidence approach, which combines assessment of the scientific literature with expert judgment to weigh that complex literature, is a scientifically defensible and reasonable approach for the causal determination.
THE FIVE CAUSAL CATEGORIES
The granularity of the current causal categories (i.e., “causal relationship,” “likely to be a causal relationship,” “suggestive of, but not sufficient to infer, a causal relationship,” “inadequate to infer the presence or absence of a causal relationship,” and “not likely a causal relationship”) is scientifically meaningful and useful in the context of the NAAQS reviews. The five-level framework qualitatively expresses the degree of uncertainty in the causal determinations, and the associated rubric summarizes key strengths and limitations of the body of evidence. The distinctions help EPA determine which health or welfare endpoints should be carried into the risk and exposure assessment stage of the NAAQS review process. The definitions of the categories provided in the Preamble are mutually exclusive and adequately reflect the uncertainties associated with the causal determination process. The categories cover the range of foreseeable scientific scenarios considered in NAAQS reviews, are scientifically defensible given the precautionary nature of the CAA, and are broadly consistent with those of other regulatory and health and environmental guidance groups.
A SINGLE FRAMEWORK FOR HEALTH AND WELFARE
Core scientific principles of causal assessment cut across health and welfare effects. Questions of exposure assessment, study design and quality, and transparency and replicability are equally relevant to assessing causality of health and welfare effects, even given that the types of studies that inform health or welfare assessments may differ substantially.
Conclusion: The Preamble’s causal determination framework can adequately guide causal determinations for both health and welfare as long as comprehensive and well-defined scientific questions and pollutant exposure-outcome relationships are identified and addressed.
Using the same framework to assess causality for both health and welfare effects allows a uniform approach to be used in assessing these two areas of air pollution impacts; benefits of this include enhanced ability to examine and integrate the linkages between health effects and welfare effects. However, there is opportunity to improve the causal determination framework so that the resulting ISAs more effectively convey the state of scientific understanding. Such improvements include providing guidelines in the framework to ensure that causal determinations are adequately supported and explained and that the significance of relevant exposure patterns and welfare endpoints is made clear. The framework might be modified to include assessment of the relevance of fundamental scientific questions identified during the initial stages of the NAAQS review process and the assumptions behind them, particularly as literature is identified and reviewed during the ISA and causal determination. The adequacy of those scientific questions in support of the causal determinations needs to be conveyed and documented as part of the causal determination process to best support setting of the NAAQS.
ADDRESSING HETEROGENEITY IN EXPOSURE RESPONSES
There is heterogeneity in both the health and welfare responses of individuals, populations, species, and ecosystems being exposed to pollutants. Heightened human response can be due to age, comorbidities, or other environmental, socio-economic, behavioral, epigenetic or genetic factors. Similarly, species and subspecies responses to exposures can vary, and susceptibility and resilience can be influenced by environmental factors (e.g., soil characteristics, buffering capacity, heat and water stress).
Conclusion: Heterogeneity in the response of individuals and populations exposed to air pollution complicates causal assessments. The current framework separates description of vulnerable groups and sensitive ecosystems or species from causal determinations, potentially obscuring understanding of causal relationships for the more exposure-sensitive groups of subjects when the causal category determinations are presented at a broader level.
Recommendation 1: Include guidelines in the framework regarding how heterogeneity in exposure responses is considered to ensure causal determinations fully account for evidence of effects in sensitive groups of humans, other species, and ecosystems. To the degree practical, the framework should provide guidance on how evidence should be examined for key sensitive groups and then integrated across endpoints or subgroups in establishing causal determinations.
The framework explicitly guides consideration of heterogeneity in response to exposures only after a causal determination is made. Consideration of only overall average population or broad ecosystem effects can obscure causal relationships that exist for more sensitive subgroups or species, communities, or ecosystems. There are threatened or endangered species and critical habitats or ecosystems, including those at the global scale, that warrant heightened attention beyond that currently given in the causal determination framework. Furthermore, studies differ in their representation and treatment of subpopulations, and in their treatment of potential problems associated with multiple comparisons. For example, controlled human exposure studies generally exclude subjects
with serious health conditions, and controlled exposure and panel studies may not be able to recruit sufficiently diverse participants to allow generalization to groups at heightened risk.
Methods are needed for both health and welfare causal determinations to systematically assess possible bias due to lack of representation in individual studies and in the overall body of evidence with respect to groups, species, or ecosystems that are expected to be at heightened risk. Research is also needed to understand how causal determinations are influenced by such biases. New indices and map presentations are needed that can enhance communication and understanding of heterogeneity in causal relationships between air pollution and welfare effects, highlighting threatened or endangered species, sensitive and critical habitats or ecosystems, and important ecosystem services. New research and methods to assess the existing literature regarding heterogeneity in effects, and how that heterogeneity is represented in causal determinations, can help identify research gaps and prioritize future research investments.
STUDY QUALITY SELECTION AND EVALUATION
A causal determination framework needs to include information on how to assess and document the relevance and quality of individual studies that are ultimately selected for inclusion in the causal assessment. Included in the assessment of relevance and quality is consideration of different study designs, including how studies account for the variety of confounders that may be present.
Conclusion: The causal determination framework described in the Preamble provides general guidance for individual study quality evaluation, but minimal detail regarding determination of individual study relevance, study inclusion or exclusion, or influence on weight of evidence causal determinations. Specific criteria for assessing the quality and relevance of individual studies to inform a given causal determination are described in recent IRPs, but the formal framework provides little direction for developing or forming those criteria for particular causal determinations.
Recommendation 2: Include in the causal determination framework used for developing Integrated Science Assessments a set of foundational study design attributes and analysis approaches to be considered when selecting and evaluating studies used for causal determinations. Include discussion of the attributes examined and how the resulting examination (e.g., the specific study characteristics of interest) influences the consideration of individual studies in the weight of evidence approach.
Specifics of how studies will be evaluated and weighed will depend on the particular pollutant considered and causal questions examined, but there are foundational aspects of study quality as they relate to questions of causality. Providing explicit guidance—such as what aspects of a study will be considered in assessing its relevance and quality—in a framework for causal determination can help increase the transparency and replicability of the study selection, evaluation, and weighting process. Some recent ISAs have included narrative quality review summaries of studies or the outputs of new tools (e.g., study quality criteria tables) to inform the assessment of study relevance and quality, but none of these applications have been systematized. While the outputs of these tools should not be considered definitive benchmarks or criteria for including a study in a causal determination, the continued use and refinement of those tools in future causal determinations—and articulation of their aspects and goals in the causal determination framework—would clarify the study selection and evaluation process. Study strengths and limitations, and the relevance (or lack thereof) of the study for the causal question under consideration could be systematically documented. Guidance for such articulation is not currently provided in the causal determination
framework. Whereas no single study selection or evaluation tool should prescriptively include or exclude eligibility for inclusion of a study in the ISA and it may be inappropriate for the framework to prespecify use of a specific tool for making causal determinations, the framework could include a set of core scientific principles regarding study inclusion and quality to increase transparency and replicability.
Recommendation 3: Provide explicit guidance in the causal determination framework for (a) assessing the approaches used in individual studies to account for important and potentially biasing confounders, and (b) how the strength of those approaches might influence weight of evidence considerations in causal determinations.
The framework recognizes that co-pollutants may be confounding factors when assessing the potential effects of a criteria pollutant, but it is not explicit about other types of confounding, such as confounding by weather effects, other environmental factors, or socio-economic or demographic differences within populations. Guidance in the framework regarding a few key aspects of confounding would help to improve the scientific conclusions drawn in the weight of evidence approach. In particular, when evaluating individual studies, the weight of evidence approach could take into account:
- how well a study articulates concerns about confounding and what the relevant confounding factors are (e.g., whether the study includes a conceptual framework for potential confounders of the research question of interest);
- whether the relevant confounders are observed and adjusted for in the study design and analysis using scientifically reasonable statistical methods (e.g., whether key confounders defined in the scientific literature are addressed and how well the study design and analysis incorporate observed confounders); and
- whether analyses of the robustness of study results to an unobserved confounder have been conducted, and how such an unobserved confounder might change study conclusions.
TRANSPARENCY AND THE WEIGHT OF EVIDENCE APPROACH
Replicability indicates that consistent study results are observed across studies conducted with different data. Reproducibility—highly related to transparency—refers to the ability to obtain the same results given the same data. Concordance of findings across the scientific literature is the basis of ISA causal assessments and its importance is stressed in the Preamble. However, the Preamble’s causal determination framework does not describe how to assess and weight transparency and potential reproducibility of individual studies in the weight of evidence approach.
Conclusion: EPA recognizes the importance of replicability of individual studies when making causal determinations. However, the Preamble’s causal determination framework does not provide explicit guidance regarding how the potential reproducibility and replicability of individual studies should affect the influence of those studies on causal assessments.
Recommendation 4: Develop guidance for the causal determination framework for assessing individual study documentation of data, methods, and assumptions, and for how the use of that assessment informs the influence of the individual study in the weight of evidence approach.
Transparency and the ability to reproduce study results can help increase confidence in individual study results, and provide rationale for more influence in a weight of evidence approach. Attention to transparency and reproducibility of individual studies is particularly important as the complexity of study designs increases. For example, designs and analyses to account for nuanced research questions and data structures (such as time-varying, exposures, and large-scale data with large samples and multiple pollutants, covariates, and outcomes measured) are complex. Privacy considerations may make it infeasible to freely share all data, but statistical software and code used to analyze the data could be documented and shared. Similarly, results that are replicable and evident across variations in study designs or specific analysis choices can be viewed as more robust and as stronger evidence for a causal relationship. EPA should investigate how study transparency, reproducibility, and replicability should influence study quality and relevance assessments used in the weight of evidence approach. Guidance should then be articulated in the causality determination framework.
INTEGRATING EVIDENCE
The causal determination framework outlined in the Preamble offers a reasonable approach for evidence integration given the complex questions under study when making causal determinations in the ISA process. The approach is broadly consistent with those applied by many other groups studying similarly complex causal questions. A spectrum of approaches for integrating evidence for causal determination exists within the public health, medicine, and environmental science communities and are continually modified and improved. EPA staying abreast through investigation and monitoring of approaches from a variety of disciplines or other causal determination frameworks could inform EPA regarding future improvements of the Preamble’s causal determination framework.
Recommendation 5: Monitor (a) research in the scientific literature on evidence integration and (b) the evolution of other frameworks used to assess causality (e.g., the Integrated Risk Information System [IRIS2] and Office of Health Assessment and Translation [OHAT3] frameworks) to determine if any emerging approaches or characteristics of their research synthesis and evidence integration protocols might be adapted to improve Integrated Science Assessment causal determinations for health and welfare.
An area where comparable causal determination frameworks differ from the Preamble’s framework is in their use of more formal schemes to rate risk of bias in individual studies and aggregate the ratings within or across lines of evidence. The committee does not recommend the use of more formal methods of quantitative scoring for individual studies or preassigned rubrics for combining evidence as there is no evidence to show that application of more formal methods provides either a more reliable or explainable result than the consensus approach currently in use by EPA and in the other frameworks examined by the committee. The framework for causal determinations described in the Preamble offers a reasonable approach. Choosing studies on which to apply risk of bias tools, and how to consider what to do in the face of heterogeneity in results ultimately depend on expert evaluation. These tools may be particularly challenging to apply in complex settings, such as environmental epidemiology. Where different study types are combined within a line of evidence, or when combining evaluations across lines of evidence, no single formal tool is available that does
___________________
2 EPA’s Integrated Risk Information System; see https://www.epa.gov/iris (accessed June 1, 2022).
3 The National Toxicology Program’s Office of Health Assessment and Translation’s approach for Systematic Review and Evidence Integration; see https://ntp.niehs.nih.gov/ntp/ohat/pubs/handbookmarch2019_508.pdf (accessed June 1, 2022).
not ultimately depend on expert evaluation for its inputs, and none of the various causal determination frameworks examined by the committee recommended use of any such tools.
Recent advances in decision making such as those in multi-attribute decision analysis may eventually be useful for the ISA process, but will require more research and assessment for application in causal determinations. Similarly, formal meta-analysis or risk of bias tools may be appropriate for certain subsets of the evidence, which then could be integrated with the other streams of evidence. EPA should monitor related fields such as risk assessment, systematic review, and decision analysis to identify any advances in practical application of expert judgment acquisition and documentation, which remains a vital part of the weight of evidence approach. However, the committee does not currently recommend a transition to a more structured approach due to the range and complexity of questions the ISAs must address and the variety of types of evidence that need to be considered.
EXPERTISE
A hallmark of the current causal determination framework is the extensive, iterative review of the underlying evidence by a broad range of EPA-appointed and public reviewers and commenters. However, the science and methodologies for assessing study relevance when making causal determinations, and of the many scientific facets of how pollutants impact public health and welfare, is rapidly evolving. For EPA to stay abreast of scientific advances, there is a need to identify all the scientific disciplines necessary to develop and thoroughly review causal determinations for each step of the ISA process. The Preamble does not detail the processes for assuring that the expertise and range of perspectives needed is identified, although EPA does have procedures for identifying experts to participate in drafting the ISAs and in forming the EPA-appointed bodies that conduct the reviews.
Recommendation 6: Articulate in the causal determination framework a clear process for identifying and incorporating the necessary expertise—including expertise in relevant emerging areas—for each step of the causal determination (e.g., development, individual study selection and assessment, through final review).
Given that causal determinations being made as part of a NAAQS review typically include a large number of endpoints for both public health and welfare, scientifically well-founded causal determinations made via a weight of evidence process requires expertise in a wide range of scientific disciplines. These may include areas such as exposure science, epidemiology, social sciences, clinical trials, mechanistic studies, ecology, agricultural sciences, and [bio]statistics including causal modeling (note that this list is not exhaustive and may differ for each ISA). Developing and articulating a formal process in the framework for identifying the scientific disciplines and range of perspectives within those disciplines needed at the different phases of the ISA and causal determination process will help ensure that causal determinations objectively incorporate the latest advances in causal determination methodologies from all necessary areas of research.
EMERGING METHODS
The continued vitality of the causal determination framework outlined in the Preamble and the ISA process depends on the recognition that designs and analytic techniques for studies that contribute to causal determinations are in a continuous state of dynamic development. The current process incorporates emerging methods reasonably well into the ISAs through engagement
of leading experts during the review process. However, there have been significant shifts in how research and study goals are framed, and how data are collected and analyzed in ever increasing volumes and complexity. Systematic guidance related to assessing quality of individual studies that utilize various emerging technologies and methods (such as causal inference techniques and machine learning approaches) may increase the utility of emerging methods and hence improve causal determinations. Examples of the most salient features of emerging research methods useful for causal determinations include those for exposure assessment, selection and control of measured confounders (including to ensure that they are not on the causal pathway from exposure to outcome), novel estimation of causal effects, dealing with post treatment variables and unmeasured confounders, and handling multiple exposures. Focus should be on weighing the relative merit of emerging tools in relation to EPA’s core goals of synthesizing rigorous scientific evidence across studies for both health and welfare.
MOVING FORWARD
The EPA causal determination framework, currently described in the Preamble, has some areas in need of improvement, but its fundamental structure (a weight of evidence approach based on review of the scientific literature, extensive internal and external review, and five causal categories) successfully supports the NAAQS review process. Improvements needed include increasing transparency in the process and providing clearer articulation of various aspects of the framework. A single causal determination framework for both health and welfare effects is reasonable given appropriate identification and attention to both health and welfare endpoints. The number and description of the causal categories provided in the Preamble’s framework is scientifically defensible and reasonable for the NAAQS process. The framework needs to provide more guidance regarding addressing uncertainties related to heterogeneities in exposure response of individuals, populations, and sensitive ecosystems directly into the determination of causality. It may need to be modified so that ISAs are able to address the relationships between exposure metrics that are scientifically relevant and well defined to support causal determinations for public health and welfare endpoints.
The committee considered the effectiveness of numerous frameworks and emerging methods developed for inferring causality. The weight of evidence approach applied by EPA is consistent with causal determination frameworks employed in similar applications and can lead to robust results. The robustness of the causal determination process would be more apparent if the framework better articulated strategies to make the process and determinations more transparent. This would increase confidence in the causal determination process, and yield greater confidence in and utility of the ISA causal determination results as part of the NAAQS reviews.
This page intentionally left blank.
