6

Example Critiques of the ISA Process and Causal Determination Framework

The U.S. Environmental Protection Agency’s (EPA’s) Integrated Science Assessments (ISAs) and causal framework have been subject to review by EPA committees (e.g., Clean Air Scientific Advisory Committee [CASAC] and CASAC expert panels) and external individuals and groups. Such critiques provide valuable input which EPA often acts on to improve the ISA process. As part of its information gathering efforts, the committee considered comments provided by CASAC and CASAC panels from recent ISA reviews and invited input from individuals representing different organizations and points of view to speak about their critiques of various ISA processes (see Appendix B for meeting agendas). The committee did not conduct an exhaustive review of critiques of the ISA process, but rather sampled from among those thought to have representative points of view from different sectors of stakeholders (e.g., the public- and private-sector points of view) or from different types of ISA consumers (e.g., those responsible for complying with or enforcing the National Ambient Air Quality Standards [NAAQS]). Inclusion of specific critiques in this chapter is not intended to imply endorsement or raise prominence of a particular point of view.

This chapter focuses on those critiques by first summarizing CASAC comments, and then summarizing input from stakeholders in sections on the topics of the weight of evidence approach, systematic review, appropriateness of the causal categories, on issues related to populations at increased risk and the selection of endpoints, and on issues related to external review. The final section of this chapter synthesizes the critiques and provides committee conclusions based on them.

CASAC COMMENTS FROM RECENT ISA REVIEWS

As described earlier, EPA’s current framework for assessing causality in the ISAs was initiated in 2008 in response to CASAC comments and refined and enhanced over the following decade based on additional CASAC and public comments. CASAC sometimes disagreed with specific causal determinations suggested in draft ISAs and was sometimes critical of the extent to which EPA adhered to or was sufficiently transparent in its approach to implementing various aspects of the framework, but in general, many different CASAC expert panels were positive, encouraging, and supportive of EPA’s continued use of the causal determination framework. However, the

CASAC reports on the external review drafts of the PM (CASAC, 2019) and ozone (CASAC, 2020) ISAs differed sharply in their tone and content from previous reports of CASAC and were marked by some notable disagreements among the committee’s members.

The CASAC report on the draft PM ISA (CASAC, 2019) began by stating that the draft did “not provide a sufficiently comprehensive, systematic assessment of the available science relevant to understanding the health impacts of exposure to particulate matter (PM).” The three major highlighted points were (1) a lack of systematic review criteria for evaluating the quality of individual studies; (2) inadequate treatment of causality; and (3) a failure to connect the evidence with biological mechanisms. The report then recommended reconstitution of the CASAC PM panel to add reviewer expertise in several key areas. This panel, which had supplemented the seven-member chartered CASAC in all previous reviews, had been recently abolished by former Administrator Pruitt, for reasons that remain unclear.

The overall comments of CASAC were supplemented by individual reviews from each of the seven CASAC members (EPA, 2019c, Table 7-6). A particularly critical review of the causal relationship between PM_2.5 and mortality recommended a “manipulative causation”¹ viewpoint (CASAC, 2019, p. A-11). The review recommended more attention be given to accountability studies and those based on quasi-experiments (CASAC, 2019, pp. A-27–A-30). In the environmental epidemiology field, quasi-experiments are generally observational studies in which some external action, such as a new government regulation or the closure of a factory, changes the pattern of emissions. In such circumstances, and with some other assumptions, “before” and “after” comparisons of health effects can be likened to the treatment and control groups of a randomized treatment allocation, allowing causal determinations. The CASAC review was also critical of EPA’s criteria for study selection and recommended a list of inclusion and exclusion criteria (CASAC, 2019, pp. A-36–A-37). In contrast, the report indicates that other members found “evidence supporting the causal relationship between PM_2.5 exposure and mortality is robust, diverse, and convincing” (CASAC, 2019, p. 3). Individual reviewer comments strongly recommended the reappointment of the PM panel, noting that the committee had no experts in epidemiology among other areas (CASAC, 2019, p. A-81). That same review recommended strengthening the conclusions in the ISA in several respects, such as more recognition of pulmonary vascular effects and cardiopulmonary interactions.

For the CASAC review of the draft ozone ISA (CASAC, 2020), EPA again did not appoint a panel of external experts to interact directly with CASAC members, but instead convened a panel of eight external consultants to respond to written questions from CASAC. That panel included two university professors (an atmospheric chemist and an epidemiologist) and six experts who were either independent consultants or industry scientists. In response, CASAC recommended that EPA “consider restoring a traditional interactive discussion process in which the CASAC can interact directly with external expert panels, while also keeping the option of obtaining written responses from external experts to specific questions” (CASAC, 2020). CASAC found “that the Draft Ozone ISA, while providing useful reviews of many aspects of ozone exposures and human health effects in selected studies, does not provide a comprehensive, systematic assessment of the available science relevant to understanding the public health impacts of changes in ambient concentrations of ozone” (CASAC, 2020). Further comments from both CASAC and individual CASAC members elaborated on the need for greater clarity in criteria used to assess study quality and in explaining the meaning of causal conclusions and how they were derived (CASAC, 2020, p. 2). Individual comments criticized the “downgrading” of the causal evaluation for short-term total mortality and

___________________

¹ This use of “manipulative causation” is a little different from the definition of the term provided in Chapter 3 of the present report. CASAC (2019) defines “manipulative causation” as “whether and how changes in exposure would change harms, holding specified other variables fixed at specified levels.” This viewpoint is not representative of the majority opinion among epidemiologists because it is considered too narrow a basis for determining causality.

cardiovascular effects from “likely” to “suggestive,” which was partly attributed to the exclusion of a large number of relevant studies conducted outside North America (CASAC, 2020, p. A-17).

CRITIQUES RELATED TO THE WEIGHT OF EVIDENCE APPROACH

A series of papers criticize EPA’s approach to causal assessment in air pollution studies (e.g., Cox, 2017, 2018, 2019b, 2021). Cox (2019b) is a concise summary of the views presented in the CASAC report (CASAC, 2019) and elsewhere that criticizes EPA’s weight of evidence approach. Cox argues that EPA’s causal classification criteria are ambiguous, lack clear boundaries, “are logically incoherent and lack testable (or potentially falsifiable) implications” (Cox, 2019a, p. 1). Furthermore, the factors EPA uses to determine causality “do not address whether or to what extent available evidence supports the hypothesis that reducing exposures would reduce risks of adverse health effects,” which he called interventional (or manipulative) causation (Cox, 2019a, p. 3). Continuous rather than discrete classification of causal effects is advocated because the latter leads to arbitrary boundaries between categories. As an alternative, Cox advocated the invariant causal prediction approach of Peters et al. (2016). The central idea is to find models that are invariant in their predictive accuracy across a wide variety of settings and hypothetical interventions. Methods to develop such models include Bayesian networks combined with conditional independence testing. As an example, Cox (2019a) gave an analysis of a dataset consisting of daily mortality, PM_2.5 and meteorology (daily minimum, maximum and mean temperature, and dew point) in the city of Boston, showing how to construct a Bayesian network. This approach showed no association between PM_2.5 and mortality when conditioning on the other variables. In this analysis a single-city dataset was used, similar in structure (though not in analysis methodology) to several papers by Schwartz and others in the 1990s (Schwartz, 1993; Schwartz and Dockery, 1992; Schwartz and Marcus, 1990). It is unclear how the methods would apply to multi-city studies or to any of the long-term analyses that have been conducted over the past 20 years.

A follow-up paper by Cox (2021) includes a thought experiment with dominos used to illustrate some conceptual points about causality. The author presented this thought experiment to the committee during one of its public meetings. Suppose 100 dominos are stood upright and close together, so that when one is pushed over, the whole set collapses. In the set one of the dominos is colored yellow. To what extent is the collapse of the set “caused by” the fall of the yellow domino? Cox likened this to whether a single molecule of a pollutant can be said to have caused the resultant adverse health effect. He proposed a solution that once again involved a graphical representation of the interactions between elements of the system and the use of causal conditional probability tables to test causality (Cox, 2021). The objective he characterized as “to address the more pragmatic question of how changing exposures (or other factors) would change outcome probabilities”—in other words, manipulative causation.

A counterargument to the manipulative causation requirement was provided by Goldman and Dominici (Goldman and Dominici, 2019). They acknowledged that such an approach could address the issue of confounding bias in epidemiological studies. However, they also argued that this objective is already addressed by EPA’s weight of evidence approach, and it would be a mistake to focus entirely on one approach that is largely untested in epidemiology. They also argued that the manipulative causality approach fails to acknowledge the many ways in which existing epidemiological studies address confounding through study design, through the explicit incorporation of confounders into regression models, and through methods based on matching or quasi-randomization. This was further rebutted by Cox (2019b), who characterized their approach as “assert[ing] that current ambient levels of fine particulate matter kill without presenting causal analyses” and stated that they “refer[red] to ambiguous and conflicting evidence of association in the presence of incompletely controlled confounders as if it supported these conclusions.” However, Goldman

and Dominici (2019) did, in their analysis, acknowledge the importance of adequately controlling for confounders. Support for EPA’s weight of evidence approach in the ISA’s causal determination framework is reflected in comments submitted to the regulatory dockets on the October 2018 External Review Draft of the ISA for PM (EPA, 2018a) and the September 2019 External Review Draft of the Ozone and related photochemical oxidants ISA (EPA, 2019b) from health researchers and medical or public health advocacy organizations. Notably, those commenters include previous members and chairs of the CASAC and its most recent PM review panel (Frey et al., 2018; Samet et al., 2018). The comments on the draft PM ISA from former members of CASAC’s PM review panel point out that

According to the same group, as the framework has been refined “comments by CASAC have transitioned from less about the framework itself toward ensuring transparent application of the framework to particular determinations” (Frey et al., 2018, p. 23).

Similarly, a comment letter on the draft PM ISA from the Health Effects Institute (HEI) specifically addresses and supports the causal determination framework as presented in the 2015 Preface. HEI’s letter highlights the following features of the conduct of ISAs:

• Well stated criteria for causal determination presented a priori in the Preface of each ISA;
• Careful evaluation of evidence from all strands of research: exposure assessment, toxicology, clinical studies, and epidemiology, rather than reliance on any one strand of evidence or solely on statistical causal analyses; and
• Explicit acknowledgment of the uncertainties attendant in each case. (HEI, 2018)

As another example, a joint comment letter on the 2018 draft PM ISA from the California Air Resources Board (CARB) and California Office of Environmental Health Hazard Assessment commended EPA for the quality of the draft ISA (CARB, 2018). The letter indicated agreement with “the overall conclusions for most PM exposure and health endpoint combinations, and with the use of the causal weighting approach that incorporates evidence from human and toxicological studies and considers study quality in the weighting” (CARB, 2018). While not disagreeing with the overall framework, other groups criticized the relative weighting of evidence from toxicology, controlled human exposure, and epidemiology studies in specific cases. The American Thoracic Society (ATS), CARB, and the California Office of Environmental Health Hazard Assessment (OEHHA) indicated that EPA over weighted one or the other type of study in regard to a particular endpoint (ATS, 2019; CARB, 2018). Others have suggested that EPA needed to better explain how it weighted and combined these lines of evidence in making some assessments (Frey et al., 2018, pp. 27–28). Commenters also question the degree of aggregation or disaggregation of outcomes for which causal determinations are made, and in some cases question the outcomes for which lines of evidence are relevant. For example, in their comments on the draft ozone ISA, CARB and OEHHA suggest that the “catch-all” grouping of developmental effects obscures the strength of the evidence for more specific outcomes, such as pre-term birth or fetal growth (CARB, 2018). Another example is the ATS comments on the draft ozone ISA, which urge EPA to consider evidence on the relationship between ozone exposure and dyslipidemia in assessing causality for cardiovascular effects, whereas in the draft ISA EPA had considered this evidence only in regard to metabolic outcomes (ATS, 2019, p. 4).

A conservation group representative who presented comments to this study committee stated

that the basic causal determination framework was working for welfare endpoints, but recommended enhanced consideration of cumulative impacts on ecosystems, and greater attention to synergistic effects of other stressors, including climate change. She supported use of critical loads as a synthesis tool and recommended greater emphasis on sensitive species and ecosystems, including climate refugia.²

The courts have reviewed the scientific underpinnings of EPA’s NAAQS decisions, based on challenges to EPA’s reliance on epidemiological studies and its weight of evidence approach. The courts have repeatedly upheld EPA’s reliance on “weight of evidence” determinations that synthesize findings from dosimetry, toxicology, epidemiology, controlled human exposure, and/or mechanistic studies, without necessarily requiring each individual study or line of evidence to stand on its own (e.g., American Farm Bureau Federation v. EPA 559 F.3d 512 (D.C. Cir. 2009); Coalition of Battery Recyclers v. EPA 604 F.3d 613 (D.C. Cir. 2010); Murray Energy Corp. v. EPA (D.C. Cir. 2019)). EPA’s use and the courts’ approval of weight of evidence reasoning in its ISAs is broadly consistent with other administrative law settings involving toxic substances (McGarity and Shapiro, 2013).

CRITIQUES RELATED TO SYSTEMATIC REVIEW

Arguments have been made for a stronger use of formal systematic review methods to supplement EPA’s current weight of evidence approach (Goodman et al., 2013, 2018, 2020). Other representatives from various industries who participated in a public meeting of this committee also supported the weight of evidence approach. According to those commentators, the primary weakness of weight of evidence approaches is the lack of systematic criteria for selecting, evaluating, and weighing studies. This, in turn, means that well-known sources of bias, such as omitting relevant confounding variables and exposure measurement error, can be down weighted or overlooked. As a result, studies may be classified as supporting a causal effect when there are, in fact, substantial limitations that would raise questions about the contribution of the study in making a causal determination.

Goodman et al. (2020) proposed “adding transparent criteria for assessing study quality, as well as detailed methods for integrating evidence in a way that fully and systematically considers individual study quality and relevance, and the coherence of results across studies within and across scientific disciplines.” They illustrated these points by reviewing 50 other weight of evidence methodologies from EPA and other agencies. They also proposed a new set of criteria for in vitro studies. In earlier work, Goodman et al. (2018) proposed a “quality scoring” assessment of studies resulting in studies being classified into 3 tiers of varying quality (see Box 6.1 for more discussion on scoring).

An illustration of this approach was provided in a recent discussion by Johnson (2021). Focusing on three papers from the recent Supplement to the 2019 PM ISA, three reviewers gave explicit scores to the studies based on many criteria including the selection of participants, confounding, whether all participants were followed through to completion (no attrition), whether all the measured outcome variables were reported, quality of statistical methods, and model specification. In the reviewers’ assessment, none of the three studies examined could be classified as Tier 1, the highest quality assessment. Note however, that the results are heavily dependent on the subjective assessments of three reviewers. Their reproducibility, in independent assessments of the same evidence, has not been demonstrated.

Richmond-Bryant (2020) provided counterarguments to both Cox and Goodman. She noted the

___________________

² Remarks by Georgia Murray, Sept. 23, 2021.

list of criteria for excluding studies given by Cox’s commentary (CASAC, 2019), suggesting that the resulting manipulative causality tests address too narrow a question in relation to the established weight of evidence approach. She also criticized the quantitative scoring approach, noting that “evaluation of quality scores in systematic reviews has demonstrated that different scoring systems can produce drastically different results for the same set of studies,” and advocated a continuation of EPA’s qualitative approach. In response, Goodman et al. (2021) reiterated the lack of explicit guidance and criteria in EPA’s current ISA process.

Statements in support of EPA’s current framework include published commentaries (Carone et al., 2020; Goldman and Dominici, 2019; Richmond-Bryant, 2020) as well as remarks provided directly to the committee producing the present report.³ These supportive commenters generally encourage research using statistical causal inference methods, but caution against any change to EPA’s framework that would restrict evidence of causality to very specific types of studies (Carone et al., 2020; Goldman and Dominici, 2019). Both published commentaries (Savitz et al., 2019; Steenland et al., 2020) and remarks to this committee⁴ caution against changing the framework to require rigid application of risk of bias scoring instruments or other quantitative schemes for scoring or excluding individual studies.

___________________

³ Remarks by John Bachmann, Sept. 23, 2021; Ananya Roy, Sept. 23, 2021; Gretchen Goldman, Sept. 23, 2021; Dan Greenbaum, Sept. 2, 2021; Jon Samet, July 20, 2021.

⁴ Remarks by John Bachmann, Sept. 23, 2021; David Savitz, July 20, 2021; Dan Greenbaum, Sept. 2, 2021.

CRITIQUES RELATED TO THE CAUSAL CATEGORIES

Individuals from a variety of affiliations have provided input to the committee regarding the adequacy of the causal categories. A National Park Service (NPS) employee who spoke to the committee on his own behalf, for example, expressed support for the five causal categories to the committee.⁵ Support for or disagreement with EPA’s specific causality determinations, but not the categories themselves, was a common focus of scientific comments on the draft PM and ozone ISAs, potentially identifying areas for more in-depth review, discussion, and future research. Critics have provided input that specific causal determinations in ISAs should be upgraded or downgraded based on their own interpretations of the evidence. As examples, the HEI comment letter on the draft PM ISA supported EPA’s determination that long-term exposure to PM_2.5 is likely causal for lung cancer, but also suggested that EPA’s determination that ultrafine particulate matter (UFP) is likely causal for nervous system effects should be reconsidered and possibly downgraded (HEI, 2018). Scientists at OEHHA questioned EPA’s downgrading of the causality determination for mortality from short-term ozone exposure from likely causal in 2013 to “suggestive of, but not sufficient to infer, a causal relationship” in the 2019 draft ozone ISA (CARB, 2018). On the other hand, HEI’s comments on the draft ozone ISA expressed support for EPA’s downgrading, which was influenced by a HEI-funded controlled exposure study (HEI, 2018). The fact that external reviewers are engaging with EPA’s determinations of “suggestive,” “likely causal,” or “causal” categories may indicate that the distinctions between them are important (see Box 6.2). As a former EPA official noted in remarks to the committee, instances where the causal determinations have been downgraded from one ISA to the next indicate that the framework is working, in that it allows EPA to change its determinations in response to new information.⁶ On the other hand, former members of the CASAC PM review panel found that in the draft PM ISA, that “there is room for more transparent communication of specific causal determinations” (Frey et al., 2018).

CRITIQUES RELATED TO POPULATIONS AT INCREASED RISK AND SELECTION OF ENDPOINTS

The Preamble’s causal determination framework is aimed at assessing causal relationships between the specific pollutants and a variety of potential endpoints, highlighting the importance of

___________________

⁵ Remarks by Bret Schichtel, Sept. 23, 2021.

⁶ Remarks by John Bachmann, Sept. 23, 2021.

selecting relevant endpoints as part of the review process. Comments from health researchers and health advocacy organizations also encourage EPA to give greater attention to at-risk populations, including people with low socioeconomic status and communities of color, children, the elderly, and those with pre-existing diseases (Public Health and Medical Organizations, 2018). These comments are often directed at EPA’s findings regarding “at-risk” populations, which in the draft PM ISA are separated from the causal determinations. The need for greater attention to at-risk populations and environmental justice is also a major theme in comments on later stages of the NAAQS review process (ALA, 2021). However, the concern is also evident in EPA’s ISA causal determinations. For example, several commenters criticized EPA’s “suggestive” causal determination for short-term cardiovascular effects of ozone, which was strongly influenced by a new multisite controlled human exposure study of healthy adults that arguably contradicted other studies involving subjects with cardiovascular disease or with risk factors for cardiovascular disease (ATS, 2019; CARB, 2018). Representatives of health advocacy organizations who spoke to the committee also recommended that EPA needed to better account for at-risk populations in making causal determinations, including giving more attention to studies of at-risk groups that suggest causal relationships or considering where needed research was lacking for these groups.⁷ While identifying such research gaps might not change causal determinations in the near term, highlighting them could stimulate valuable research efforts.

Selection of welfare impacts has also come under criticism in connection with the framework EPA uses. In 2015, after EPA issued the draft Integrated Review Plan for the NO_x and SO_x secondary standards, conservation groups stressed the importance of using critical loads to provide a “science-based framework for policy determinations” (The Adirondack Council, 2015), the need to consider sensitive and essential ecosystems, and the need to consider the effects of these pollutants on endangered species (Center for Biological Diversity, 2015). The Air Resources Division of NPS also submitted comments on the draft Integrated Review Plan, emphasizing the need to use critical loads “for key biological indicators … to scale up site-specific results to eco-regional levels so that regional implications can be considered” (NPS ARD, 2015). The NPS comments also urged EPA to consider impacts on ecosystem services. Additionally, speaking to the committee on his own behalf, an NPS employee suggested that more consideration be given to health and well-being effects of welfare endpoints, such as the psychosocial effects of degraded visibility. He also suggested that the significance of welfare endpoints could be brought out more in the causal determination framework.⁸

CRITIQUES RELATED TO EXTERNAL REVIEW

Numerous commenters on the draft PM ISA and the draft ozone ISA raised concerns about procedural changes related to these reviews. A primary issue was elimination of the supporting expert panels that have traditionally augmented the expertise of the seven-member CASAC (CARB, 2018; Frey et al., 2018; Public Health and Medical Organizations, 2018; Samet et al., 2018).⁹ Commenters were also concerned about the truncated timeframes EPA set for NAAQS reviews and the attendant elimination of opportunities for external review of the ISA and other review documents (ALA, 2021).¹⁰ External review opportunities for the ozone NAAQS review were particularly short, with the Policy Assessment issued before the ISA had been finalized.

___________________

⁷ Remarks by Ananya Roy, Sept. 23, 2021; Gretchen Goldman, Sept. 23, 2021.

⁸ Remarks by Bret Schichtel, Sept. 23, 2021.

⁹ Remarks by Georgia Murray, Sept. 23, 2021.

¹⁰ Remarks by Georgia Murray, Sept. 23, 2021.

SYNTHESIS OF CRITIQUES AND REVIEWS OF THE ISA FRAMEWORK

In summary, many reviews of the framework demonstrate support for the overall weight of evidence approach and five causal categories that have been refined over many review cycles. The framework, in concept, is at least accepted by many commenters. This support is not unanimous, as demonstrated by the CASAC reports of 2019 and 2020, and some of the individual comments to this committee. In particular, some critics suggested that the causal categories are lacking clear, testable boundaries, and that more emphasis should be placed on studies that use formal causal inference methods. However, the alternatives to EPA’s weight of evidence approach that have been suggested raise their own issues, and would not appear to enjoy widespread support. Some critiques of the ISA causal determination framework summarized above describe a lack of a systematic review criteria for selecting and evaluating individual studies for inclusion in the weight of evidence approach. These commenters raised concerns that the lack of systematic criteria could result in overlooking sources of bias in individual studies. On the other hand, other commenters caution against overly rigid application of study scoring instruments and indicate support for more qualitative assessments of study quality, as EPA currently provides. Another cluster of criticisms focused on selection of endpoints for causality determinations, including the concern that not enough attention is paid to at-risk populations or welfare endpoints related to sensitive ecosystems or endangered species. Some individual concerns described to the committee were related to specific applications of the framework in specific ISAs, or about processes that define aspects of an individual ISA that occur prior to the conduct of that ISA (e.g., the identification of the scientific questions in the integrated review plan that will be addressed in the ISA). While recognizing that these comments did not address the Preamble framework per se, the committee considered how they might reflect on gaps or limitations in the framework.

In reviewing the Preamble, illustrative applications of the framework in recent ISAs, and CASAC and external commenters critiques, several themes emerge. The lines of evidence used to assess causality for both health and welfare endpoints allow scope for observational, controlled exposure, and mechanistic studies to be considered. As discussed in the Preamble and illustrated in recent ISAs, causal determinations for health consider controlled human exposure studies, epidemiology studies, and animal toxicology studies that are framed with a discussion of biological plausibility. Causal determinations for welfare are able to draw on laboratory and field experiments, time-series and gradient observational studies, paleoecological studies and process models, with critical loads assessments providing valuable insights for synthesis. Further application of formal causal inference approaches to observational datasets for health and welfare could lead to sharper conclusions about causality for individual studies. These approaches should be encouraged, but they should not become the sole focus of epidemiological studies, nor should they replace the numerous other lines of evidence employed in the weight of evidence approach.

While the Preamble includes a general discussion of strengths and limitations of each type of study, it lacks specifics on criteria for evaluating study quality. This limitation has been addressed to some degree—but not completely—with study quality tables included in recent (post-2015) ISAs for health endpoints, though not for welfare. The detailed narrative quality reviews published in the Health Assessment Workspace Collaborative (HAWC) registry for the most policy relevant health studies (supporting causal or likely causal determinations or change of causal category for the 2020 ozone ISA) are also helpful to constrain study quality criteria but need refinement. The introduction of the PECOS (Population, Exposure, Comparison, Outcome, and Study Design) tool in the 2020 ozone ISA (EPA, 2020a) adds a useful, objective basis for including or excluding studies based on relevance, and importantly, was applied to both health and welfare effects and presented for review and comment in advance of the ISA in the (first draft) ozone integrated review plan (EPA, 2018b).

Causal determinations are presented for relatively broad endpoints such as reproductive effects or increased tree mortality. This approach has the advantage of allowing coherence across more specific effects to be included as part of the causal assessment. However, it may cloud causal determinations for more specific health effects, effects in particular subgroups at heightened risk, or for harm to specific species or types of ecosystems.

For both health and welfare endpoints, ISAs published in the 2019–2020 timeframe illustrate how EPA uses the Bradford Hill aspects of association (Hill, 1965) in a qualitative approach to synthesizing the evidence for causal determinations. Causal determinations are not assigned by rigid rubrics, but rather are explained with a post hoc narrative. With this approach, having robust reviews of draft determinations by CASAC and external stakeholders are especially critical for ensuring that the determinations are adequately justified and scientifically sound. EPA’s reviews focus on using tabular and graphical comparisons across studies within a particular line of evidence. In Chapter 7 discussion of other frameworks for assessing causality from a body of evidence is provided, including some that use more quantitative approaches or specify in advance what causal classification would result from a particular type and strength of evidence.