5
Next Steps and Opportunities for Applying Triangulation

To begin this session, Martyn Smith of the University of California, Berkeley, presented the key characteristics (KCs) approach in epidemiology, toxicology, and hazard identification. He noted that findings providing insights into mechanisms play an increasingly important role in hazard identification. The KCs provide the basis for a knowledge-based, objective, and systematic approach to evaluate mechanistic data in hazard evaluations. This approach contrasts with but also complements the more reductive mode of action or adverse outcome pathway (AOP) approach (detailed below). Recent International Agency for Research on Cancer (IARC) Monographs and evaluations by the U.S. Environmental Protection Agency (EPA), the California Environmental Protection Agency, and the National Toxicology Program (NTP) have illustrated the applicability of the KCs-based approach.

The KCs of carcinogens were identified 10 years ago. KCs vary with the type of toxicant and have been developed for non-cancer hazards including male (Arzuaga et al., 2019) and female (Luderer et al., 2019) reproductive toxicants, endocrine disrupting chemicals (La Merrill et al., 2020), hepatotoxicants (Rusyn et al., 2021), and cardiovascular toxicants (Lind et al., 2021) and are currently under development for neurotoxicants, immunotoxicants, and metabolic disrupters. Future opportunities include a comprehensive set of biomarkers and assays to measure these KCs, as well as additional studies of these KCs within human studies. These advances may greatly aid epidemiologists in their work and advance evidence synthesis.

In this context, triangulation or evidence integration is highly relevant for identifying chemical hazards. Given that some 350,000 chemicals are used regular commerce, reaching conclusions from epidemiology or animal studies alone is unlikely. High throughput studies generally focus on cytotoxicity, which is largely not relevant to carcinogenicity and more complex chronic diseases. Mechanistic data from studies in humans, animals, in vitro, and in silico may increase in importance and provide biological plausibility. The studies are both relatively easy to conduct and produce a voluminous amount of data.

As described by Schubauer-Berigan, IARC convened two workshops in 2012 that explored the challenges in evaluating mechanistic data. There was no method to search systematically for data relevant to carcinogen mechanisms, leading to a lack of uniformity in mechanistic topics considered across assessments. As a further challenge, the evidence is growing in volume in complexity and could encompass thousands of publications for agents such as polychlorinated biphenyls or benzene. “Most importantly, how could bias towards favored mechanisms be avoided?” Smith asked. Depending on who is in the room, different mechanisms will be proposed. To address these challenges, IARC identified the 10 KCs of carcinogens and later developed an evaluation approach based on the KCs. In this approach, the KCs of human carcinogens form a uniform basis for assembling and evaluating relevant mechanistic evidence to support cancer hazard identification.

The classical approach to evaluating mechanistic data is hypothesis-driven and typically involves developing the idea of a mode of action (MOA), or more recently an AOP. This approach includes a hypothesis, detailed knowledge, and measurements of the key events that are postulated to occur during progression of toxicity.¹ It also provides a focus on a favored mechanism or pathway that can introduce bias, especially on committees and public databases. Moreover, the MOA or AOP may be incomplete or

___________________

¹ Key events are hypothesized effects along a biological pathway after the molecular initiating event and characterize the progression of toxicity (https://ntp.niehs.nih.gov/whatwestudy/niceatm/comptox/ct-aop/aop.html). They are distinct from KCs of carcinogens, which are the empirically observed effects of known human carcinogens (https://pubmed.ncbi.nlm.nih.gov/30521319).

wrong, which was the case for di(2-ethylhexyl) phthalate (DEHP) (Rusyn et al., 2012). As another example, described by Haney, the MOA for chromium VI (CrVI) in the gut was proposed to involve cytotoxicity followed by cell proliferation. However, other effects also occur if CrVI is of a high enough concentration to produce cytotoxicity, consistent with its five known KCs, which are genotoxicity, epigenetic alterations, oxidative stress, and chronic inflammation, and therefore increased cell proliferation.

By contrast, the KC approach does not require hypothesizing a mechanism, but may inform an MOA or AOP if required for setting a regulatory standard. KCs are founded in a systematic evaluation of knowledge about known carcinogenic agents and comprise the properties of known human carcinogens. They are distinct from the hallmarks of cancer, which refer to the biology of the cancer cell. Established human carcinogens, including CRVI, have one or more, usually several, of these KCs. The KC approach is more agnostic, avoiding bias. It can identify data gaps for mechanisms and can be applied to nonchemical agents such as infectious agents and radiation.

Since 2015, IARC has applied the KCs in cancer hazard evaluations of many, mechanistically diverse chemicals (e.g., pentachlorophenol, styrene, acrolein) and complex exposures (e.g., welding fumes, nightshift work).² KCs have also been used by NTP, for example, to evaluate haloacetic acids as well as antimony trioxide.³ However, they may also be used, in line with the EPA Guidelines for Carcinogen Risk Assessment (2005),⁴ to inform understanding of the relevance of data generated from animals. The KCs can also be used as the basis of an unbiased search of the literature and help develop MOAs and AOPs, which work by Xavier Arzuaga at EPA recently showed (Arzuaga et al., 2019). However, for carcinogens, it is very difficult to determine the molecular initiating and key events, and even harder to measure them in humans. For example, it may not be possible to detect common key events (such as adducts and mutations in the TP53 gene). Furthermore, the events need not occur in a serial fashion and the outcomes are complex, with cancer involving at least 200 different disease states. These are limitations to applying the AOP paradigm to cancer and chronic diseases.

The KCs approach has been endorsed by National Academies reports. Using 21st Century Science to Improve Risk-Related Evaluations (NASEM, 2017) encouraged development of KCs for other hazards besides carcinogens, and Review of U.S. EPA’s Staff Handbook for Developing IRIS Assessments: 2020 Version (NASEM, 2022) encouraged use of the KCs for organizing mechanistic data and evaluating biological plausibility. Efforts are under way to apply the KCs to the 19 toxicological traits noted in legislation in California.

KC applications can facilitate systematic reviews of mechanistic data and identify data gaps. They can also potentially improve predictive toxicology and molecular epidemiology for disease prevention. However, more assay and biomarker development is needed (as noted by Fielden et al. [2018]). Within the pharmaceutical industry, the KCs have potential as a replacement for many animal studies. Lists of assays to measure the KCs in human studies, experimental animals, and in vitro has been published for the KCs of carcinogens (Smith et al., 2020) with plans to expand the list online by 2023.

More studies of key characteristic biomarkers could bolster epidemiological studies along with the development of best practices for future applications. As noted by Schubauer-Berigan, the KCs have been valuable to IARC, and its Preamble emphasizes mechanistic studies in exposed humans and human cells. Systematic evaluations of different KCs in exposed groups of people using appropriate biomarkers to inform authoritative bodies and regulators could strengthen this research and resulting policy decisions. Additionally, because decisions about the KCs are based on expert judgment, best practices would likely be helpful in standardizing decision-making. For example, expert judgment would rule out basing strong conclusions on studies at very high concentrations. Additionally, the KCs are not a checklist. They are useful only within the triangulation context, during the evidence synthesis process, and to inform integration of mechanistic results with epidemiology and animal findings.

___________________

² See https://pubmed.ncbi.nlm.nih.gov/29562322 and https://monographs.iarc.who.int/agents-classified-by-the-iarc.

³ See https://ntp.niehs.nih.gov/ntp/roc/monographs/antimony_final20181019_508.pdf.

⁴ See https://www.epa.gov/risk/guidelines-carcinogen-risk-assessment.

Next, Tracey Woodruff of the University of California, San Francisco (UCSF), described the Navigation Guide. The evidentiary basis used by clinical medicine differs substantially from that used by environmental health. Clinical medicine laws require testing of pharmaceuticals or interventions prior to their introduction in the marketplace. Thus, many human experimental studies, or randomized controlled trials (RCTs), are conducted and considered to be the gold standard. In contrast, environmental health primarily uses observational human studies, as well as animal studies, as the basis of decision-making. Because environmental chemicals are already in use, humans are already in a pre-exposed state, thus relevant decisions are after, rather than before, use.

Aiming to develop a method to apply lessons from clinical medicine to improve the basis of decision-making in environmental health about ongoing exposures, UCSF developed its Navigation Guide for systematic reviews in 2009. This development followed 18 months of work with a broad range of experts in clinical medicine, state and federal government, and evidence-based systematic reviews, including with the Cochrane Collaboration. The USCF work was motivated by the challenges with reviews at EPA, particularly that the evidence evaluation methods were not always transparent or consistent. USCF’s goal was to create a method that reduced bias in evidence evaluation in order to support better decisions regarding chemicals identified as harmful, and therefore improve environmental health, Woodruff explained.

Creation of the Navigation Guide coincided with an upswell in interest in environmental health systematic reviews because their transparency, consistency, and evidence evaluation were shown to improve the basis of decision-making. Multiple National Academies’ reports and reviews⁵ have discussed systematic review approaches, including those of EPA’s Integrated Risk Information System (IRIS) Program and NTP, and have endorsed or used the Navigation Guide. The World Health Organization and the International Labour Organization also applied the Navigation Guide in a 2-year systematic review of 15 observational human studies, involving 220 experts in 35 countries, in a first-ever publication on the Occupational Burden of Disease of Long Work Hours. As a result, this previously unrecognized burden of disease is now considered one of the highest burdens of disease globally.

The Navigation Guide differs from other approaches in some important ways, although it is similar to the methods of NTP’s Office of Health Assessment and Translation and EPA’s IRIS Program. With the Navigation Guide, each stream of evidence is evaluated separately. In addition, the Navigation Guide allows for a gradation of evidence, or a summary of the strength evidence and an acknowledgment of uncertainty of evidence, while providing a basis for moving forward with decision-making. By using the Navigation Guide's approach, non-human data (and later perhaps mechanistic information) can be evaluated in the same manner used for observational human studies, following this proposed process to evaluate strength of evidence:

1. Analytical plan or protocol development
2. A search for evidence, study selection, and data extraction. Bias is reduced by applying the Navigation Guide consistently across the three evidence streams, assessing different criteria to downgrade or upgrade evidence, after which a rating is assigned to all the evidence rather than scoring or removal of studies deemed to be biased. All evidence is considered. This approach does not rely on a checklist, computer program, or algorithm but provides a structured approach to evaluate the different domains that could influence the study outcomes in one direction or another. Different domains that have been developed can change over time, as new information becomes available. However, they are, in general, based on

___________________

⁵Review of EPA’s Integrated Risk Information System (IRIS) Process (NRC, 2014a); Review of DoD’s Approach to Deriving an Occupational Exposure Level for Trichloroethylene (NASEM, 2019); The Use of Systematic Review in EPA’s Toxic Substances Control Act Risk Evaluations (NASEM, 2021); Review of the Environmental Protection Agency’s State-of-the-Science Evaluation of Nonmonotonic Dose-Response Relationships as They Apply to Endocrine Disruptors (NRC, 2014b).

2. empirical evidence or judgments of methodological features that can influence the outcomes across studies in one direction or another.
3. A strength of evidence evaluation, which was developed based on IARC’s approach to synthesizing and integrating evidence.
4. In this process, the systematic review can also be used for dose-response, by identifying studies for meta-analysis and a body of evidence across evaluation to arrive at a conclusion, beyond non-cancer health effects.

Woodruff highlighted similarities between the Navigation Guide and other approaches, including the Hill considerations (see Box 2-1) and the Surgeon General’s 1964 report,⁶ citing a comparative table from a National Academies’ review of EPA’s IRIS Program (NRC, 2014a). To evaluate the overall body of evidence, the Navigation Guide uses criteria common to other methodologies including inconsistency, indirectness, imprecision, publication bias, and conflicted sources of funding. The Navigation Guide and NTP’s approach have integrated elements of Grading of Recommendations Assessment, Development and Evaluation (GRADE), with different domains that build on the Hill considerations (see Box 2-1) for the overall evaluation of evidence.

Woodruff illustrated use of UCSF’s Navigation Guide with a systematic review of polybrominated diphenyl ethers (PBDEs) exposure and neurodevelopmental outcomes in childhood. PBDE are flame-retardant chemicals to which most if not all Americans are exposed, including women during pregnancy. A narrative review was inconclusive concerning the association of PBDE exposure and neurodevelopmental outcomes (Gibson et al., 2018). In contrast, UCSF’s systematic review and meta-analysis found sufficient evidence of an association between developmental PBDE exposure and reduced IQ. The human evidence was rated as moderate across different domains. A counterfactual was also used to analyze whether any future published studies might influence the meta-analysis results, which was found to not be the case.

Woodruff advocated for a focus on improving or clarifying systematic review methods to address the issues identified and discussed during this workshop. She also endorsed upgrading observational studies, to ensure that they are appropriately valued and used in evidence reviews. She advised against use of a single metric to describe the risk of bias or of quantitative scoring. Instead, sensitivity analyses can evaluate the influence of different risk of bias domains. If one domain poses a high risk of bias, then quantitative analysis can help determine any influence on results. She emphasized the importance of including all studies. “Systematic review methods have evolved and continue to evolve. The next generation of systemic reviews can better accommodate the factors that we have talked about today, which would be a great improvement in our methods,” she observed.

Next, Lisa Bero of the University of Colorado School of Public Health and Senior Editor, Cochrane Public Health and Health Systems Network, explored questions and challenges about triangulation in the context of existing systematic review approaches. “The talks have raised a lot more questions for me,” she said. Among these are the following:

• “Is triangulation about strengthening causal inference, using as much information as possible, exploring potential biases?
• Is it a method or an approach used within a study, within a data set, across different types of data?
• Is it a method for risk of bias assessment?
• Is it a method for comparing studies or exploring heterogeneity, all to allow us to figure out if with finding what we are seeing, we can express more or less confidence in it?
• Is it a method used to synthesize evidence?

___________________

⁶ See https://profiles.nlm.nih.gov/spotlight/nn/feature/smoking.

• Or, is it used in one step beyond evidence synthesis, during the structured decision-making, such as on a regulatory committee that may consider more data than that in the evidence synthesis?”

She added that many of these methods or tools already exist in various forms and have already been addressed by systematic review methods. Other considerations include whether the IRIS Handbook should be modified to include triangulation.

Since its development 20 years ago, the Cochrane Handbook for Systematic Reviews has been updated once, but methods have evolved. EPA is very involved in publishing methodological advances. In a series in the American Journal of Public Health led by Bero, Cochrane reviewers and methodologists contribute articles that refine Handbook topics ranging from using logic models to guide a review, to involving stakeholders, to defining core outcome sets and performing qualitative evidence synthesis.

Bero addressed myths about Cochrane reviews, including that they are limited to RCTs. Instead, the reviews use abundant types of studies besides RCTs and conduct different types of systematic reviews, including living systematic reviews, overview reviews, and rapid reviews, among others. Another myth is that the reviews focus exclusively on clinical interventions, despite their coverage of public health interventions and implementation science. However, no reviews of chemical hazard assessments have been conducted to date, although one protocol is in development. Cochrane does not exclude studies based on risk of bias assessments: half a chapter in the Handbook is devoted to how to conduct risk of bias assessments using sensitivity analysis, descriptive reports, or grouping studies for subgroup exploration. In fact, the Handbook does not advise excluding studies. Lastly, Cochrane reviews are not the same as the committee decision-making process. They are a critical component of that process, but guideline and regulatory committees consider other factors in addition to findings from one or multiple systematic reviews to make a decision.

Bero addressed two major issues about triangulation: how to assess and consider different risks of bias (as addressed by Woodruff) and how to employ a reproducible and transparent method to integrate data from different sources to reach a conclusion about causality. The latter point considers how to obtain the best evidence for the question. The protocol for reviews, which Cochrane has published since 1998, addresses transparency about study selection. One aspect of transparency is definition and determination of which quantitative study designs to include in systematic reviews, for which Cochrane is developing guidance. Cochrane allows nonrandomized evidence if RCTs are not available, sparse, uncertain, of low quality, or indirect to the question at hand, or present other important limitations. Compared to RCTs, observational studies include more realistic exposure assessments and measurements; improve ability to assess important outcomes of interest that are likely to occur over longer time periods than typically studied in a trial; capture rare occurrences, which require a much larger population; and consider equity, because often RCTs are conducted in very narrow populations. However, Bero cautioned against study design labels (e.g., cohort or interrupted time series), and noted that Cochrane is moving away from such labels because these terms are defined differently and inconsistently. Instead, the guidance is to query aspects of a study and encourage reviewers to consider the study characteristics they seek.

As an example, Bero described a review (Anglemeyer et al., 2020) commissioned by the New Zealand government during the pandemic to assess its investment in a national digital contact tracing system for COVID-19. The review addressed the effectiveness of digital solutions in identifying cases compared to old-fashioned contact tracing methods. The review also considered the acceptability, privacy, and other ethical concerns surrounding the use of digital solutions that RCTs could not fully answer. The review considered a range of quantitative and qualitative research studies based on the study characteristics sought. Risk of bias was addressed at the individual study level (as explained by Woodruff), using a modified ROBINS-I tool. An RCT was not modeled because it was not realistic. Humans conducted the risk of bias assessments, not machine learning. Establishing what merits assessment and the likely direction of effect of any bias is part of the Cochrane review process.

Next, these diverse types of data were synthesized using SWIM (Synthesis Without Meta-Analysis) guidelines (Campbell et al., 2020). These guidelines outline methods to synthesize evidence and

explore heterogeneity. Studies can be grouped by exposure, characteristics, or individual risks of bias for these purposes. The synthesis method is mapped to the question asked, while considering the type(s) of data available. If effect size estimates are sought, meta-analytic methods can be used. If the objective is to assess any evidence of an effect, direction of effect and P-values may suffice. For the contact tracing study, modeling with different assumptions and scenarios indicated that digital solutions were successful. However, digital solutions may have equity implications for at-risk populations with poor access. Importantly, the synthesis did not offer a recommendation; a committee would take that next step and reach a decision on action.

Among the many remaining triangulation-related challenges is that systematic review methods, including risk of bias tools, can be misused and misunderstood. There may be insufficient or no data to quantify bias or to conduct analyses within a study, and obtaining quantitative estimates from diverse data can be challenging. Bero noted that evidence should be synthesized in a structured, narrative way. She also cautioned that use of a new and ill-defined term is problematic, noting that the National Academies’ review of the IRIS Handbook recommended a glossary to clarify terminology.

PANEL DISCUSSION

Planning committee member Nicholas Chartres, UCSF, moderated a panel discussion that included the session speakers as well as Ellen Chang, Exponent, Inc.; Lianne Sheppard, University of Washington; and Kyla Taylor, National Institute of Environmental Health Sciences.

In opening remarks, Chang stressed the importance of relevant subject-matter expertise in the triangulation process, citing her glyphosate and neurotoxicity work. Specifically, expertise in the specific exposure and health outcome proved essential to assessing risk of bias, because the exposure assessment was a key element in judging study quality. She noted that risk of bias tools are useful as guideposts for structure, transparency, and systematic methods, but expert judgment and expertise will continue to play a significant role. This comment echoed earlier remarks by Smith that conclusions regarding the KCs are based on expert judgment and that best practices may be needed, including regarding evaluation of the quality of the evidence.

Sheppard stated that exposure assessments pose a key challenge to interpreting pesticide studies. In contrast, air pollution exposure can be addressed through predictive models of ambient exposure measurements. She also cautioned against requesting endless effort of studies and evaluation, which can be a strategy for delay and a barrier to action.

Echoing earlier comments, Taylor noted that “there is nothing new in the concept of triangulation.” She noted, “most of its aspects are already being used by systematic review in the Navigation Guide, IRIS, NTP, IARC, and others. They all integrate information from different evidence streams and consider contributing factors to risk and bias and consistency. I would avoid calling this a new or emerging method.” Although triangulation approaches can improve current systematic review and evidence integration methods, particularly regarding transparency and communication, she noted the challenges to formalizing these approaches.

Panelists discussed key limitations to the current risk of bias approaches used in systematic reviews and opportunities to address them. They highlighted the need to better account for the direction and magnitude of bias in individual studies. Chang stated that current risk of bias tools help to direct focus on exposure assessment, outcome assessment, confounding, and selection bias; yet, flexibility is needed. “It will inevitably not be a one-size-fits-all situation,” she asserted. Predicting the direction of bias is also difficult, she added. Bero noted that, whereas prediction of direction is possible, “a big challenge with risk of bias tools is quantifying the bias.” Referencing earlier comments by Woodruff, she stated that methods can evolve, citing the example of elaboration of tools to capture important parameters about exposure by the Navigation Guide, NTP, and others. Woodruff concurred that exposure assessments must improve, referencing systematic reviews for air pollution and autism, which incorporated data on air toxins or criteria air pollutants. She noted that some exposure misclassification biases toward the null, a consideration that might help to inform evaluations.

Bero cited findings and recommendations from the National Academies’ review of the IRIS Handbook concerning risk of bias tools, including the need for transparency in the flow of studies from the question to synthesis and integration. Woodruff also noted the importance of including all studies in evidence synthesis. “But, introducing new terms like triangulation without a clear definition might add confusion. A larger priority may be focusing on specific aspects of the process,” she cautioned. As noted earlier by Bero, the National Academies’ review also suggested development of a glossary, because the IRIS Handbook uses many terms in unconventional or inconsistent ways. Like Taylor, Woodruff pointed to the utility of a protocol. She called for identifying the most important domains based on the topic as a critical first step, specifying in the protocol how those domains will be evaluated, and ascertaining whether the evidence is sufficient to do so. In other words, domains should not be treated equally.

Turning to causal inference, Sheppard noted the strengths of EPA’s recent particulate matter review, which used “novel methods for adjusting for confounding” to address the evidence from different points of view. Woodruff asserted that counterfactual analyses are needed in meta-analyses to add confidence. These and other quantitative approaches enable the exploration of alternative scenarios that might influence overall findings and can be incorporated into protocols as researchers learn more.