5
Evidence Synthesis
In this chapter, the committee reviews the following chapters of the ORD Staff Handbook for Developing IRIS Assessments (the handbook): “Analysis and Synthesis of Human and Experimental Animal Data” (Chapter 9), “Analysis and Synthesis of Mechanistic Information” (Chapter 10), and the portions of “Evidence Integration” (Chapter 11) that discuss synthesis (EPA, 2020a). The committee’s review of those chapters considered whether the approaches to evidence synthesis are scientifically sound. It also examined whether the considerations presented in those handbook chapters are sufficiently broad to allow for application to the wide range of scenarios expected to be encountered when applied to individual Integrated Risk Information System (IRIS) assessments. In addition, the committee considered whether the methods presented in those handbook chapters are sufficiently clear in describing the intent to synthesize the relevant evidence and incorporate study evaluation conclusions, regardless of the study results (Question 6a, Appendix B).
OVERVIEW OF EVIDENCE SYNTHESIS AS COVERED IN THE HANDBOOK
The U.S. Environmental Protection Agency (EPA) IRIS process for evidence synthesis is presented in Chapters 9, 10, and 11. The purpose of these chapters is to summarize and interpret the results across all informative health effect studies within both human and animal streams and to integrate those data to draft hazard synthesis sections describing human and animal toxicity data. Approaches for analyzing mechanistic data on endpoints that lead to, or modify, the development of adverse outcomes are provided in Chapter 10. In addition to what was provided in the handbook, the committee relied significantly on EPA’s presentation (Thayer, 2021). Screenshots of the Health Assessment Workspace Collaborative (HAWC) within-evidence stream judgment process were critical for understanding the approaches incompletely described in Chapter 9 and then redundantly, though not consistently, reiterated in Chapter 11.
A goal of the handbook is to enhance transparency of EPA’s “guided expert-judgment” process. This is consistent with recommendations from the 2014 National Academies report Review of EPA’s Integrated Risk Information System (IRIS) Process to
develop templates for structured narrative justifications of the evidence-integration process and conclusion. The premises and structure of the argument for or against a chemical’s posing a hazard should be made as explicit as possible, should be connected explicitly to evidence tables produced in previous stages of the IRIS process, and should consider all lines of evidence (human, animal, and mechanistic) used to reach major conclusions. (NRC, 2014, p.106)
CRITIQUE ON EVIDENCE SYNTHESIS
The HAWC approach presented by EPA (Thayer, 2021), including factors that increase certainty and factors that decrease certainty, seems to be a meaningful move toward transparency. That said, Table 9-1 (p. 9-3) and Table 11-1 (p. 11-5) are inconsistent in descriptions of the considerations, and Table 11-1 appears more representative of the process. Chapters 9 and 11 are not clearly organized with respect to where synthesis stops and integration starts. As there is no conclusion at the end of Chapter 9, it is not clear what is accomplished in that chapter. If two chapters would cover synthesis/integration, the first chapter could address Step 1 (pp. 11-2 to 11-3; within-stream judgments) and the second chapter could address Step 2 (across streams).
Unit of Analysis
Given the committee’s understanding of the handbook, the process or method for evidence synthesis, particularly in Chapter 9, is not clearly or consistently described. With the understanding that HAWC is a dynamic system, Table 9-1 is not consistent with the column headers and criteria in the HAWC interface, namely Studies, outcomes, and confidence; Summary of key findings; Factors that increase certainty (consistency, dose-response, coherence of effects, larger or concerning magnitude of effect, mechanistic evidence providing plausibility, medium or high confidence studies, other); and Factors that decrease certainty (unexplained inconsistency, imprecision, lack of expected coherence, evidence demonstrating implausibility, low confidence studies). Furthermore, Table 9-2 (p. 9-6) introduces individual and social factors that may increase susceptibility to exposure-related health effects but without specifying how those factors are considered in the within-stream evidence judgment. Presumably in the “Factors that increase/decrease certainty” column both susceptibility and publication bias can be noted. These options could be more clearly categorized as “Other” if Table 9-1 was redesigned to match the HAWC judgment boxes. Also, the role of natural experiments in Chapter 9 is out of step with how risk of bias is discussed in the study evaluation chapter. Natural experiments are a study design, which can be evaluated using risk of bias criteria suited to that study design; they are not a consideration for evidence synthesis as stated in Table 9-1. In general, using HAWC, the methods associated with within-stream evidence judgments can be implemented in a transparent manner, but that is not yet clearly communicated in the handbook.
The evidence synthesis approach in general is sufficiently broad to allow for application to a wide range of scenarios. Yet, the committee often stumbled on different terminology used and that supported the need for glossary definitions for “endpoint,” “[health] outcome,” “health effect,” “bodies of evidence,” and “streams.” In a systematic review framework, the individual “bodies of evidence” need to be prespecified. It is recommended in Chapter 3 of this report that this be accomplished through the development of a series of “refined” population, exposure, comparator, outcome (PECO) statements informed by the literature inventory/evidence map (which utilized a “broad” PECO statement) and specified in the assessment protocol.
A related element of confusion is the “unit of analysis” of the synthesis step. In Chapter 9, it appears that there are only two “bodies of evidence”—animal and human—for each health effect. On the other hand, in Chapter 11, it is noted that “inferences” may be drawn from “a finer level of specificity of effect” and then used “to draw conclusions about the broader health effect categories.” Additionally, Chapter 11 states that “evaluation of strength of … evidence will preferably occur at the most specific health outcome level possible” (EPA, 2020a, p. 11-8). Indeed, in Table 11-1, it appears that the template evidence profile table suggests that “separate rows by outcome,” possibly also by study design, are permitted and that each “row” that undergoes synthesis results in a strength of evidence judgment. However, in EPA’s presentation, an example was presented where liver effects in animals were separated by “clinical chemistry” and “histopathology,” but only a single strength of evidence judgment was presented (Thayer, 2021).
A corollary issue is that in a systematic review framework, mechanistic and toxicokinetic (TK) data or pharmacokinetic (PK) and physiologically based pharmacokinetic (PBPK) models play a narrow role in evidence synthesis of animal and human bodies of evidence, primarily limited to the evaluation of consistency. However, as described in Chapter 3 of this report, in some cases, mechanistic endpoints may be considered as discrete endpoints that will undergo evidence synthesis. If those endpoints involve in vivo animal or human data, the approach to evidence synthesis for animal and human data streams is appropriate.
In vitro or non-traditional evidence streams present more of a challenge and are currently an area of rapid evolution. The handbook has outlined a reasonable approach to evaluating in vitro data in Chapter 10, with the synthesis approach shown in Chapter 11. These represent reasonable approaches but are likely to change substantially with time.
Choices of Methods
The evidence synthesis approach outlined in the handbook appears to be a “hybrid” of a guided expert judgment approach (e.g., Hill, 1965; IARC, 2019; Samet et al., 2020) and a more structured approach (e.g., Grading of Recommendations Assessment, Development and Evaluation [GRADE] [Balshem et al., 2011; Guyatt et al., 2011]; NTP, 2019). For instance, while the considerations delineated appear primarily to be based on the Bradford Hill considerations (with the addition of “study confidence,” which includes risk of bias), the structure of the analysis resembles GRADE but without the explicit up-rating and down-rating of evidence. The committee reiterates the view of the NRC (2014) report withholding an explicit recommendation to choose either guided expert judgment or a structured approach. The committee notes that the handbook’s evidence synthesis framework does not rely on the Bradford Hill considerations (1965) alone but also is aligned with multiple more modern evidence evaluation frameworks, including Cochrane (Higgins and Thomas, 2019), GRADE, NTP (2019), and the Navigation Guide (Woodruff and Sutton, 2014.
More problematically, however, the considerations outlined in evidence synthesis appear to mix in some elements of evidence integration, particularly through the concepts
of coherence and biological plausibility. The result is confusing, especially when combined with the challenging transition between Chapter 9 and 11, described above. Most systematic review frameworks do not include the concept of “coherence” in the evidence synthesis step, because that would involve bringing in information from outside the evidence being evaluated. Moreover, the application of “coherence” as described in the handbook appears more appropriate either (1) during planning of the assessment (the biological relationship among different endpoints) or (2) during evidence integration (through incorporation of mechanistic data). The concept of “biological plausibility” is largely absent in most systematic review frameworks except perhaps in the concept of “indirectness” used by GRADE and Cochrane. Overall, the applications of “biological plausibility” in the handbook appear to either (1) address considerations already covered elsewhere, such as consistency; (2) be more appropriate to consider during evidence integration; or (3) involve comparison with data on mechanistic effects. The latter two are more appropriate to incorporate during the evidence integration step.
Finally, the consideration of natural experiments is inappropriate in Chapter 9 of the handbook. They should already have been considered in “study confidence,” since the criteria used for study evaluation will depend on the design of the study and these criteria are discussed in Chapter 4 of this report.
The remaining considerations—study confidence, consistency, strength (effect magnitude) and precision, and biological gradient/dose-response—are closely aligned with existing systematic evidence evaluation frameworks, such as Cochrane, GRADE, NTP, and the Navigation Guide. This is appropriate because the Bradford Hill considerations encompass both evidence synthesis and evidence integration, while modern evidence evaluation frameworks make an explicit distinction between them in order to promote greater transparency and consistency.
Logistics
Chapter 9 of the handbook provides a limited discussion on data visualization (p. 9-7) including a forest plot for trichloroethylene, but there is much less emphasis on data visualizations within evidence streams compared to that in Chapter 6. Trend analyses could also be helpful in the analysis and synthesis of within-stream studies.
The committee noted a concern that the IRIS process could become very bogged down during the within-stream judgment process, given the statement in Chapter 9 that “evidence synthesis is an iterative process” (EPA, 2020a, p. 9-1). This concern is compounded by the lack of clarity around who is actually doing the analysis (“assessment team and disciplinary workgroups (as needed)”). It would be useful to more clearly define who is responsible for conducting the synthesis and indicate for quality control that some consensus is needed on the final judgments. Additionally, Table 9-1 suggests in “coherence” that “these analyses may require additional literature search strategies.” It is not clear to the committee why additional literature review would arise at this late stage of the assessment. Collectively, these concerns reflect the committee’s concern that the timeliness of evaluations could be adversely impacted based at least on how Chapter 9 of the handbook is presented.
FINDINGS AND RECOMMENDATIONS
Findings and Tier 1 Recommendations
Finding: The handbook is confusing as to the transition from the synthesis step (within a data stream) to integration (across data streams), in particular because many considerations for synthesis are repeated (with slight variation) in Chapters 9 and 11.
Recommendation 5.1: The handbook should consolidate its discussion of evidence synthesis in a single place. The discussion should include all of the considerations involved in making strength of evidence conclusions (currently in Chapter 9), as well as the criteria for different strength of evidence judgments (currently in Chapter 11). The handbook chapter describing synthesis of evidence should end with the methods for reaching strength of evidence conclusions for each unit of analysis, and how these are carried forward to evidence integration. [Tier 1]
Finding: The “unit of analysis” for evidence synthesis and the strength of evidence conclusion is unclear, in terms of the breadth or narrowness of the evidence being synthesized. Although the handbook states that the evaluation of strength of evidence “will preferably occur at the most specific health outcome possible,” the process for doing so is not clear (EPA, 2020a, p. 11-8). For instance, in a case where liver clinical chemistry and liver histopathology (both in animals) are synthesized separately, will they each receive a strength of evidence judgment, will there be a single overall strength of evidence judgment for liver toxicity, or will strength of evidence judgments occur both individually and together?
Recommendation 5.2: The unit of analysis for evidence synthesis and strength of evidence conclusions should be clearly defined as specified by the refined PECO statements recommended in Chapter 3 of this report. For example, a unit of analysis could be defined at the endpoint level (e.g., clinical chemistry) or outcome level (e.g., liver toxicity). If judgments may be made at both the endpoint and health outcome levels, details should be provided on how these judgments and the methods used to make them are distinct from each other. [Tier 1]
Finding: The guided expert judgment approach that EPA uses does not describe how to specify the initial rating for the strength of evidence evaluation. The approach does not include explicit criteria for up-rating and down-rating evidence and does not include sufficient information to operationalize the approach.
Recommendation 5.3: The handbook should provide justification for the initial rating for strength of evidence, as well as more detailed operationalization of the criteria used to upgrade or downgrade the evidence. [Tier 1]
Finding: The considerations around mechanistic and TK data and PK or PBPK models outlined in evidence synthesis appear to mix in some elements of evidence integration, particularly through the concepts of coherence and biological plausibility, because they rely heavily on evidence outside of the “unit of analysis.”
Recommendation 5.4: The handbook should restrict the applications of mechanistic and TK data or PK models in evidence synthesis and strength of evidence judgments to those relevant to each individual unit of analysis, such as addressing consistency and indirectness of evidence. Other applications of mechanistic and TK data or PK models, such as addressing coherence and elements of biological plausibility, could be addressed in evidence integration, either as a separate evidence stream or as support for the human or animal evidence streams. This recommendation should be implemented in the planning stage and reflected in the protocol (see Recommendation 3.9). [Tier 1]
Finding and Tier 2 Recommendation
Finding: Consideration of “natural experiments” is not part of evidence synthesis but rather part of study evaluation.
Recommendation 5.5: EPA should remove discussion of natural experiments in the context of evidence synthesis in the handbook. That discussion could possibly be repurposed in the context of the handbook’s Chapter 6 “Study Evaluation.” [Tier 2]
No Tier 3 recommendations were presented in this chapter.
