4
Case Studies of Triangulation Across Epidemiology Studies for Hazard Identification and Risk Assessment
This workshop session included specific case studies of triangulation and evidence synthesis, beginning with a presentation on air pollution by Hanna Boogaard of the Health Effects Institute. Describing a large review to synthesize evidence from traffic-related air pollution (TRAP) studies that bolstered confidence in prior conclusions about associations between TRAP and specific health outcomes, Boogaard discussed the difficulty of evidence synthesis for environmental exposures and mixtures. Drawing from 350 observational studies, the review used a systematic approach to quantitatively summarize epidemiological results where possible using meta-analysis techniques. It included evaluation of risk of bias in individual studies and a review protocol published online.1
She noted that several features characterized the review, including its large size: it is the largest effort of its type to date, requiring substantial scoping decisions. The researchers limited evaluation to the epidemiological literature and a subset of health outcomes including mortality, cardiovascular and respiratory morbidity, and birth outcomes. They included pollutants as indicators of the traffic-related mixture and assessed confidence in the evidence of an association using two complementary approaches considered to be equally valuable. The first approach was a modified Office of Health Assessment and Translation (OHAT) approach for assessing confidence in the quality of the body of evidence. This Grading of Recommendations Assessment, Development and Evaluation (GRADE)-type approach provided an initial rating based on study design features, which were up- or downgraded based on certain factors (including risk of bias, dose-response, or inconsistency). The second approach provided a narrative assessment for the presence of an association. Both approaches reached the same confidence conclusions.
One lesson learned is that observational studies can offer high confidence evidence in environmental health, where randomized controlled trials (RCTs) are generally not feasible, she said. The review started with a moderate confidence rating for observational study assessments because all types of cohort and case-control studies met three of four key design features—the one exception being the controlled exposure feature, which can only be met in RCTs. The review found this limitation to be an insufficient reason to begin with a lower confidence rating.
The broader narrative assessment complemented the review because of the shortcomings of the GRADE-based approach, that is, the mechanistic up- and downgrading of confidence related to certain factors, primary focus on the quality of a body of evidence, rather than the possibility of association, and heavy gearing toward studies entering meta-analyses. Consideration of the direction and magnitude of the association and the consistency of findings figured more prominently in the narrative assessment, she said. As a key tenet of triangulation, replicated associations across several studies, populations, pollutants, or methodologies were more likely to present a true association than single isolated observations from single studies. Though both the GRADE-based and narrative assessments share many evidence synthesis aspects, they display subtle but major differences. The fact that complementary assessments reached the same confidence conclusions with some exceptions was reassuring, according to Boogaard.
Another lesson learned is that all relevant studies in evidence synthesis should be considered, she said. The traffic review started with 350 studies with roughly being half meta-analyses. Other studies such as those for indirect traffic measures such as distance to roadways or traffic density were included because those measures contained relevant information for the confidence assessment. “Meta-analyses do
___________________
1 See https://www.healtheffects.org/announcements/panel-publishes-protocol-review-traffic-related-air-pollution.
not automatically increase confidence in the evidence. And studies not fitting into meta-analyses may be equally informative,” Boorgaard asserted, referencing work by Savitz (Savitz and Forastiere, 2021).
The review also showed that heterogeneity of the effect estimates among observational studies should generally not be used to downgrade confidence, she said. Statistical tests for heterogeneity have well-known limitations: they are less reliable when the number of studies is small, and legitimate reasons may plausibly account for variability in the magnitude of effect estimates such as different populations, exposure assessment methods, or follow-up time. However, the review did not downgrade because of inconsistency, because almost all individual studies reported positive associations. Heterogeneity primarily stemmed from the variability of the magnitude of the positive association—not its direction. GRADE does not upgrade for consistency—it only downgrades for inconsistency—while OHAT and nearly all environmental hazard and risk frameworks use consistency as an extra upgrading factor, which led to this decision, as outlined by Boogaard. Triangulation is part of this process (Pearce, 2019).
Boogaard noted that the review supported expanded exploration of publication bias, beyond use of statistical approaches. The review assessed publication bias through funnel plots and Egger tests when sufficient studies were available. Boogaard said that a minimum of 10 studies are needed, and 10 may be too few because Egger test results also depend on study size and magnitude of association; true heterogeneity in effect size may lead to asymmetrical funnel plots and significant Egger tests. Other useful approaches include subgroup analysis of multi-center studies with single city studies, or analyses of differences in effect estimates from earlier versus later studies.
Boorgaard highlighted the need to assess the impact of specific sources of potential bias and noted lack of consensus on the best approach for assessing risk of bias in observational studies. The review used an adapted version of the risk of bias tool and guidance from the World Health Organization (WHO) Air Quality Guidelines systematic reviews2 because its tool assessed risk of bias in observational air pollution studies. It assessed six domains including confounding and selection bias, and then, for most domains with subdomains, rated each subdomain and provided an overall rating per domain in three categories. No summary classification was derived across domains. Going forward, it may be useful to focus on identifying possible key biases, classifying each study based on how effectively it addressed each potential bias, and determining whether results differed substantially across studies in relation to susceptibility to key biases (Savitz et al., 2019). This approach would provide insight into the potential impact of each specific bias, identify a subset of studies likely to best approximate the true association, and suggest features needed to improve future research.
Finally, the review evaluated up- and downgrading factors independently following GRADE’s Quality and Environmental Noise Guidelines for systematic reviews, because GRADE generally does permit upgrading after a body of evidence is downgraded.
Roel Vermeulen of Utrecht University presented on triangulation of evidence concerning environmental exposures, including to diesel engine exhaust, power line proximity, benzene, and asbestos, and possible links to cancer. He noted that comparisons of different study designs and evidence over different domains may be useful to strengthen causal inferences. Artificial intelligence could help identify useful studies for systemic reviews and triangulation, by mining publications for relevant information.
Vermeulen anchored his talk in the notion that study designs in different populations, using different methods, or with different evidence bases would not share the same measured or unmeasured biases, and thus are informative when compared. This utility also applies to studies with different information layers, such as in epidemiology or toxicology, to integrate evidence to a point of reference. Thus, comparing results from studies with different observational study designs and from different domains through triangulation can help strengthen causal inference in the fields of hazard identification and risk assessment.
___________________
2 See https://www.euro.who.int/en/health-topics/environment-and-health/air-quality/publications/2020/risk-of-biasassessment-instrument-for-systematic-reviews-informing-who-global-air-quality-guidelines-2020.
Vermeulen discussed an example of diesel engine exposure and lung cancer (Lipsett and Campleman, 1999). Concerns about an over- or underestimation of effect if there was no adjustment for healthy worker or smoking effects motivated researchers to conduct a comparison of studies that had adjusted for these effects to those that had not. A comparison of studies on truck drivers to those on railroad workers exposed to diesel engine exhaust revealed similar results, suggesting that co-exposures or differences in lifestyles were unlikely to bias results. “This is triangulation’s strength: By using all studies—not penalizing on a study’s negatives or positives—using their positive and negative effects by quantifying the expected direction, we can infer what is happening and strengthen our evidence base,” he said. However, in explaining research on proximity to power lines and childhood leukemia, Vermeulen noted that studies that share similar designs, populations, and exposure assessments are susceptible to the same kinds of biases. He characterized risk of bias tools as a detective, not the judge.
Approaching research questions from different angles may foster effective use of triangulation approaches. Vermeulen noted that mechanistic data can be informative but also voluminous, and highlighted opportunities for efficiently identifying and clustering relevant data based on the key characteristics of carcinogens.3 He also noted that the quality of the exposure assessment in epidemiological studies is an important source of heterogeneity that could be more exploited in triangulation.
The next case study was presented by Amy Berrington de González of the National Cancer Institute and addressed triangulation approaches to ionizing radiation (Berrington de González et al., 2020). Ionizing radiation is an established carcinogen, but an important public health question remains about the risks and magnitudes of these risks from low-dose exposures. Health Risks from Exposure to Low Levels of Ionizing Radiation: BEIR VII Phase 2 (NRC, 2006), the last in a series on ionizing radiation as a long-standing carcinogen limited to low doses (fewer than 100 millisieverts), concluded that the available scientific evidence was consistent with risks from low-dose exposures, based primarily on epidemiological data from medium- and high-dose studies combined with radiobiology data. No direct evidence was available then from human data. Using different approaches to assess bias, study quality, and inferences can boost confidence in causal inferences from a particular study or group of studies, said Berrington de González.
Recent publications in low-dose radiation epidemiology have been facilitated by methods such as pooled analyses and electronic record linkages, and new studies have also emerged on susceptible populations and new exposures. Berrington de González and her colleagues undertook a systematic review to evaluate these epidemiologic findings on the health effects associated with low-dose ionizing radiation. They systematically assessed the impact of potential biases, including their direction and, if possible, magnitude. However, the variability across studies challenged their ability to address more formal triangulation questions. They included a variety of timings, primarily split between childhood and adult exposures, to evaluate exposures in different ways.
Additionally, instead of excluding studies for low-quality dosimetry and no adjustment for smoking, Berrington de González and her colleagues categorized them as having a high likelihood of bias. Following extensive evaluation of each bias in each study, only a few studies were potentially biased away from the null. Exclusion of these studies did not alter the conclusion, or even the summary risk estimate. It was far more likely that studies were biased toward the null, particularly because of the exposure assessment limitations. Thus, the summary excess relative risk estimate was quite possibly slightly underestimated. However, quantification of the magnitude of the dose error from the published data was challenging. Ineligible studies were also mostly positive, and publication bias was considered to be unlikely because the radiation epidemiology field is a small field and most studies are known. This new body of epidemiological data from all available, eligible studies with dose-response estimates, directly supports excess cancer risks from low-dose ionizing radiation exposure.
___________________
Turning to key lessons learned, Berrington de González noted that extensive efforts were required to consider duration and time of exposure, looking at childhood and adulthood exposure studies separately. In most cases, the direction of bias could be assessed, but assessment of the magnitude of bias with the published data was more challenging. The researchers found some contrast with biases likely going in different directions and subgroup independence, for case control and the timing of exposure. Moreover, occasionally independent subgroup assessments have multiple limitations that can make pulling apart biases difficult.
For the future, Berrington de González advised thinking carefully about whether types of approaches are sufficiently varied to enable meaningful contrasts, considering, for example, different settings. Other questions include whether the subgroups are independent (considering case-control studies and timing of exposure) and whether a qualitative comparison is sufficient. She also urged her peers to publish quantitative bias assessments within original study publications. “This really would be made much easier,” she said. “This not only took a long time, it took an enormous group of people with all sorts of expertise to achieve this.”
David Savitz of the Brown University School of Public Health began his presentation by positing that triangulation is not as complex as it may appear. He cited choices of a partner, job, or house as common examples of decisions that are based on convergent information. In assessing a spectrum of research, rigorous quantitative tools can assist but should not be treated “as a judge” per Vermuelen’s point. Rather, they have utility as part of the menu of options, to both strengthen causal inferences and inform research needs.
Savitz presented on the epidemiology of per- and polyfluoroalkyl substances (PFAS). He discussed grouping studies based on population, exposure, and health outcomes to identify methodological strengths and limitations. He explained that the Centers for Disease Control and Prevention’s (CDC’s) National Health and Nutrition Examination Survey (NHANES)4 data allow for large studies with the power to detect weaker associations. However, these types of studies cannot identify exposure sources and are susceptible to reverse causality. Conversely, studies with defined sources of PFAS exposure allow for potential mitigation but are costly and time-consuming and study size is limited. Savitz noted that considering the methodological strengths and limitations of these subsets of studies may be conducive to triangulation because the studies complement each other. This approach can also highlight domains where new studies are and are not merited.
Savitz also noted that studies of specific sources of elevated exposures to PFAS are more directly informative of risk, but can be corroborated by studies of biomarkers in the background range. Similarly, clinical health outcome studies are more directly informative of risk, while studies of subclinical health markers can be corroborative. Extensive research on perfluorooctanoic acid (PFOA) and perfluorooctanesulfonic acid (PFOS) includes large populations with high exposures and clinical outcomes. However, inferences about other forms of PFAS may benefit from biomarker studies of background exposure, and from reaching beyond epidemiology to incorporate toxicology and mechanistic research. Ultimately, for chemical groups such as PFAS with thousands of individual chemicals, additional toxicologic and mechanistic research can inform interpretation of epidemiologic studies.
The last presentation of the session, given by John Jackson at the Johns Hopkins Bloomberg School of Public Health, addressed triangulation in the context of health equity interventions. Jackson first established a distinction between health disparities and health equity. He noted that negative health outcomes may further disadvantage historically marginalized groups, which, in the context of health as a human right, becomes an issue of justice. Jackson focused on hypertension as an example of a chronic condition of greater prevalence among minorities. NHANES data from 1999 to 2016 showed that awareness of the ability to control hypertension through treatment displays an encouraging trend upward for all racial groups. However, significant gaps across racial groups in disease prevalence and control persist. Jackson stressed that this phenomenon is striking because the epidemiology of hypertension is well established, with known causes, many tools to address it, and a malleable gap.
___________________
A conceptual model developed by Lisa Cooper and colleagues at the Johns Hopkins Center for Health Equity explored various aspects that contribute to racial and ethnic disparities in hypertension prevalence and control. The challenges they identified at the patient level include poor health literacy, exposure to stress and discrimination, inadequate access to quality health care, other barriers to adherence, and healthy lifestyles at the family level. At the health care provider level, knowledge of social needs, cultural competence, and communication skills can profoundly affect which clinical information is elicited and the adequacy of treatment decisions. At the organizational level, organization of care, the care model, and available resources to track disparities and address equity issues also play a role. At the community level, issues include the food environment, housing segregation, and stressful living conditions. Jackson noted that the state and national policy environment can greatly affect the availability of health care and incentives that drive the care models used by health institutions. All of these aspects represent how different systems work together to reproduce inequity, which some call systematic or structural racism. This model lays bare health disparities in very complex and mutually reinforcing ways.
“In light of this complexity, many have recognized that there’s really no silver bullet to achieving health equity,” said Jackson. “The thinking is that we stand a greater chance of reducing disparities if we take a multipronged, multilevel approach, which means targeting more than one factor simultaneously through various channels” at the patient, provider, community, and policy levels. The Rich Life project5 serves as one example of this approach. Nearing publication, this project comprises a pragmatic cluster randomized trial that evaluates a collaborative stepped care intervention that involves a nurse care manager, community health worker, and special team to deliver individualized care to patients. Such an intervention taps different levels of the aforementioned social ecological model.
Jackson proposed a thought experiment to consider how the extent of disparity in hypertension control may change with distribution of a risk or protective factor, such as treatment intensification. Jackson also highlighted the consistency assumption, which is growing in recognition but perhaps historically overlooked. Market forces and socioeconomic differentials can reinforce racial inequities, even after interventions, policies, and strategies are attempted.
He also noted opportunities for observational studies in the health equity context, as such studies can assess factors that, if equalized across social groups or race, appear to reduce disparities. This can aid identification of the differential distributions of factors that appear to be implicated in the etiology of disparities. However, Jackson noted some potential limitations. Because observational epidemiological studies are designed to measure the confounders relevant to the hypotheses at hand, they may be more limited as data sources to study health disparities. Studies on health disparities are often conducted using secondary data and are thus at greater risk for unmeasured confounding. One should not assume that all relevant factors have been measured among people facing challenges in maintaining optimal health. In addition, a key driver of disparity may be tightly associated with race, resulting in the importance of that factor in driving disparities being underestimated.
In spite of these limitations, Jackson asserted that observational studies should not be discounted or abandoned; rather, researchers and public health practitioners must be willing to explore a wider range of evidence. Jackson highlighted the importance of triangulation and integrating evidence across different designs, including:
- Pragmatic trials, such as health outcomes based on insurance coverage
- Randomized studies, especially quasi-studies where the randomization already exists and does not have to be assigned
- Simulation models using data from different locations
- Qualitative data, especially if used to contextualize and synthesize results
___________________
5 See https://www.jhsph.edu/research/centers-and-institutes/johns-hopkins-center-for-health-equity/what-wedo/research/current-projects/rich-life.
PANEL DISCUSSION
The panelists discussed the challenges of causal inference for mixtures. These challenges can be addressed by focusing the triangulation approach toward the answer sought and overall objective, said Vermeulen. Savitz agreed, noting that the aim may range from understanding the biological effects of one or more component chemicals or informing regulation of the mixture. For instance, air pollution questions may concern nitrogen dioxide particularly, or proximity to roadways more generally. For PFAS, “Science ought not wait for definitive or even highly informative epidemiology for dozens of different chemicals. It’s not feasible,” he cautioned. Vermeulen agreed, adding that, broadly, the source and related intervention may matter most. The mixture’s individual components may not be particularly important if the planned intervention for health prevention concerns the source side, which is shared. Jackson suggested studying trajectories or longitudinal pathways of exposure to many exposures at once to identify a constellation of experiences or risks that are imbalanced across social groups. Identifying these pathways could help assess the effects of multiple risks together, rather than individually, because ultimately, joint intervention is necessary.
Varied opinions emerged on whether triangulation, as part of a broad category of various strategies for drawing conclusions, should be more formalized within a causal structure. Savitz noted no constraint by the geometry of the triangle, calling different evidence streams complementary threads of knowledge that inform one another. Savitz added that evidence bases are complementary, not necessarily in their support of one another, but in their pertinence to the overall interpretation. Berrington de González agreed, noting that with a hierarchy of information, human data will usually outweigh animal experiments, even if they consistently point in the opposite direction, citing the ionizing radiation and radiobiology examples.
As for challenges, evidence may be insufficient for triangulation or different evidence streams may completely disagree (e.g., epidemiology and toxicology). Vermeulen asserted that even within such an evidence base, one can triangulate, referencing his low-frequency field and childhood leukemia work. Looking at the issue from different angles may point to inference, and different streams of evidence can suggest connections and possible alignment, he said.
Panelists broadly objected to the formalization of triangulation in the near future, calling, instead, for more expert judgment. As an example, Boogaard highlighted difficulties using the OHAT approach for the traffic review due to its mechanistic up- and downgrading of certain factors, hence their development of a narrative approach. Savitz concurred, arguing against a formal, structured approach. Berrington de González disagreed, calling out tremendous progress in the bias assessment field, and in the increased understanding and recognition of the need to use new tools for rapid analyses, conceding that checklist tools should be avoided. Berrington de González called for a more formal framework that technically triangulates some sets of epidemiological data—though this work can be challenging and time intensive, she said. Boogaard concurred on the topic of time and effort, noting that methods may not exist. Vermeulen said that a checklist cannot replace verifiable argumentation; however, he queried, “How does one formulate argumentation and make it verifiable?” Vermeulen asserted that better argumentation means a much more a priori consideration of the specific evaluation planned—not a specific checklist. Savitz added that analogous exposures can provide relevant information for different diseases. “It’s akin to looking at everything that contributes rather than distilling it,” he said, encouraging more holistic or comprehensive consideration rather than algorithms.
Although multiple speakers noted the usefulness of clearly defined assessments of evidence quality and bias, some speakers expressed concern about introducing a seemingly new concept to established approaches to causal inference and evidence integration. Savitz warned that triangulation could potentially be applied to support or refute any conclusion and give an opinion an unwarranted “sheen of credibility.” Boogaard expressed concern that applying triangulation could create endless assessments with no clear conclusions. However, Vermeulen noted that if the science is rigorous and
approaches are established before data are evaluated, some of the potential pitfalls of triangulation may not be as concerning as other speakers posited. Summarizing the drawbacks voiced by speakers, committee moderator David Richardson stated, “I made a brief list of the dangers: anarchy, entropy, endless Sisyphean tasks, fear, and sleeplessness. So those are the fears or the cautionary tales we have.”
