2
Triangulation: Background, Methodologies, and Applications

To begin the workshop, the U.S. Environmental Protection Agency’s (EPA’s) Kristina Thayer provided context for the agency’s interest in triangulation by highlighting the concerns around excluding studies during human health assessment evaluations. “If you use a risk of bias tool to exclude studies, then you are fundamentally undercutting this ability you might have to look across studies, look at strengths and weaknesses across a collection of studies and reach conclusions,” she explained. Yet, although triangulation is appealing as a concept, it is difficult to operationalize with a standardized approach. Therefore, Thayer welcomed both an exploration of the concept of triangulation to determine whether its use might reduce bias and enable inclusion of more studies, and a repository of examples to consult when considering more routine use of triangulation.

Speakers during the first workshop session introduced triangulation and described various considerations and statistical approaches for its application. Triangulation has taken different forms and used different methods over the centuries, but attempts within epidemiology to formalize the concept to be more encompassing, systematic, and quantitative are new, said Deborah Lawlor of the University of Bristol. Lawlor explained that triangulation, if properly applied, can help to address pressing epidemiological questions more quickly and efficiently because of its inherent strength in identifying consistency, particularly as more data become available from biobanks and other sources. However, successful application of triangulation requires more transparency in published studies and reviews, especially regarding the methodology and how study biases were addressed.

To anchor her talk, Lawlor defined triangulation as bringing together results from two (but ideally more) different research approaches with potentially different and unrelated key sources of biases, while being explicit about the likely sources and direction of bias. Comparing results across those different approaches can lead to better causal understanding, she said. Triangulation’s premise is that all studies, including case-control studies, clinical trials, and laboratory experiments, have different underlying assumptions and different strengths and weaknesses. Triangulation seeks to balance biases across studies to reach conclusions without eliminating evidence. Different types of studies or approaches with different biases (particularly if directionally opposed) could strengthen causal inference if they arrive at the same conclusion. Conversely, considering results or conclusions from different approaches with different sources of bias would increase confidence in the sources of bias and identify possible future research.

Referring to a comment by Thayer, Lawlor acknowledged triangulation must be formalized in order for epidemiologists to use it in practice. She discussed her 2016 paper (Lawlor, 2016) that was in turn referenced by other speakers. Eric Tchetgen Tchetgen of the Wharton School at the University of Pennsylvania referred to the paper as “a welcome clarification of the potential utility of an older idea as it applies to etiological epidemiology,” and “a nice contribution toward formalization of triangulation as a useful tool for evidence evaluation and knowledge integration.” Neil Pearce of the London School of Hygiene & Tropical Medicine echoed this sentiment, welcoming efforts to make triangulation more explicit to bolster existing causal inference approaches. Pearce also pointed to the Preamble to the IARC (International Agency for Research on Cancer) Monographs¹ and the Bradford Hill considerations (see Box 2-1) as two best approaches to formalizing expert judgment. Similarly, Kyle Steenland of Emory University noted that discussion of how triangulation differs from other types of evidence synthesis and sensitivity and bias analyses will inform understanding of the approach.

___________________

¹ See https://monographs.iarc.who.int/wp-content/uploads/2019/07/Preamble-2019.pdf; https://pubmed.ncbi.nlm.nih.gov/31498409; and Samet et al., 2020.

Tchetgen Tchetgen highlighted the criteria that may be key to applying triangulation:

• Including different approaches with unrelated sources of potential bias
• Targeting the same well-defined causal question of interest
• Targeting the same exposure timing and duration
• Explicitly acknowledging key sources and direction(s) of bias

Multiple presenters stated that triangulation may be applied more successfully for evidence integration when incorporating studies with different study designs. Lawlor highlighted the importance of publishing study protocols and clear methodologies so that in the future researchers can more easily use data from prior studies to fortify or negate their own study findings. Lawlor asserted that predetermined study protocols avoid issues of excluding challenging studies and later added that multidisciplinary teams are best equipped to apply triangulation in a systematic way. During his presentation, Pearce echoed the need for relying on multiple studies, “In general, we assess causation by putting all of the evidence together and there is rarely, if ever, a single study, which establishes causation.” Steenland agreed, stating, “If you get the same answer using different types of studies subject to different types of biases, that gives you some comfort.”

The speakers discussed applications of triangulation across and within studies using different analytical methods. Across-study triangulation may rely on slightly different research questions, while within-study triangulation may use multiple approaches to answer the same question. Regarding the latter, Tchetgen Tchetgen advocated for a robust causal identification method to examine the study parameter of interest using multiple causal models that enable inference if one of the identifying strategies holds true. Steenland later noted the usefulness of this within-study approach. Lawlor addressed an example of prepregnancy body mass index and maternal and fetal health outcomes, which were examined with different approaches (i.e., conventional multivariable regression and Mendelian randomization). Across-study triangulation also benefits from consideration of the exposure or intervention being evaluated. Lawlor suggested using large databases and consortiums such as the UK biobank² to enable multiple analyses with standardized data. Pearce noted that a variety of study designs are useful in triangulation and evidence synthesis if they include quantitative exposure measurements.

Finally, as discussed repeatedly throughout the workshop, triangulation and evidence integration should acknowledge sources and directions of bias to accurately inform causal inference. Lawlor commented that part of the systematic approach of triangulation requires noting the sources of bias for the included studies. Tchetgen Tchetgen agreed, highlighting the tradeoff between more robust inferences and

___________________

² See https://www.ukbiobank.ac.uk.

reducing bias. “True reflection of your uncertainty and understanding should not be viewed as less efficiency but rather actually a more accurate expression of uncertainty about possible sources of bias,” he affirmed.

Pearce cautioned against using identification and evaluation of bias to score or exclude studies. He noted that different study designs can have different sources of bias, which can be identified and balanced. “Ultimately, I think triangulation is important and it is part of a more general approach as advocated by David Savitz and others, which is to start by looking at all of the evidence and then thinking about the possible sources of bias rather than just going through and scoring studies and then throwing them out,” Pearce advised.

PANEL DISCUSSION

Given ongoing dialogue about what triangulation is and what it encompasses, the panel discussed its distinguishing features in the context of other approaches for epidemiologic evidence and synthesis. Lawlor asserted that other forms of evidence synthesis primarily focus on one type of study, particularly randomized controlled trials (RCTs), or assess observational designs. Pearce noted the huge overlap between studies, while clarifying that triangulation delivers different data, citing work by Lawlor showing the effects of breastfeeding in the United Kingdom and Brazil where one would expect the confounding structure to be different. This similarity also applies to selection bias, Pearce added, citing an example in which the selection bias in different studies was expected to be in opposite directions and gave the same results. “Triangulation is more involved in comparing different sets of data with different assumptions or biases, whereas sensitivity analysis works with one set of data to vary assumptions,” he said.

Tchetgen Tchetgen described triangulation as a method that entails making varied assumptions to deliver robust inferences. By contrast, sensitivity analysis can be many things that all share an anchoring in a particular analysis choice. Tchetgen Tchetgen suggested anchoring analyses by varying assumptions not identified by the data and assessing how much inferences might vary if the assumptions were in fact true. By combining these two approaches the conclusion is such that if either method provided valid inferences it would yield a single, valid answer. One might also run various analyses separately and compare results to the multiple robust analysis.

Lawlor noted that triangulation has been used both within her specific field of reproductive epidemiology and in the larger field of epidemiology. A well-known historical example of triangulation’s value is Ignaz Semmelweis’s work on infection control in obstetrical clinics.³ In 1846, Semmelweis explored survival outcomes among pregnant mothers and their children in two maternity wards, one staffed by midwives and the other staffed by medical students and doctors. Mortality rates were markedly higher in the ward staffed by doctors. Switching and tracking patients in both wards, as well as accounting for the fact that a doctor who performed autopsies was infected and ultimately died, revealed that the high rates of mortality could be attributed to infections. “Midwives who suggested this study weren’t satisfied with any one method. They wanted to bring all the data together,” she said.

However, Pearce noted that triangulation as a specific approach is often not expressly used, hence the push for formalized decision-making. “But unfortunately, I think the way it has been done is by introducing these systems of scoring individual studies whereas what we need are more formal systems so that the decision path is clear that it more formally incorporates triangulation and other methods,” he said.

Regarding triangulation’s limitations, Lawlor cited more noise or disagreement based on diverse methods, additional inconsistency, more time and effort expended, and the need for collaboration. Moreover, Lawlor noted, in an academic system that judges PhDs and postdoctoral students on the number of papers produced irrespective of their quality, science might suffer. Tchetgen Tchetgen concurred, highlighting the need to address incentives in academia and industry. “Publication biases, method analyses and knowledge integration are very vulnerable to the winner’s curse,” he said. “Do we

___________________

³ See https://www.npr.org/sections/health-shots/2015/01/12/375663920/the-doctor-who-championed-hand-washingand-saved-women-s-lives.

really understand the extent to which such selection bias or the lack of null results being published as widely as results that tend to report a finding, the extent to which this really affects our ability to advance knowledge?” he asked. “This demands self-reflection and acknowledgment.”

Among other limitations is the need to find the right data to apply to the method, Pearce said, citing an example of asthma in low- and middle-income countries. Many experts in Western nations ascribe asthma to allergy, but his work in low- and middle-income countries suggests otherwise. The international patterns and time trends are a key part of the picture that is missing, more so than insights afforded by improved studies in the United States or in the United Kingdom. “We have to think globally,” he said. Lawlor also voiced concern about a preponderance of available data originating in high-income countries, which may not be generalizable or transportable, underscoring the importance of establishing large cohorts in different populations. She cited promising examples of efforts to conduct cohort studies among adults in Malawi and children in Pakistan. Still, many results suffer from selection bias.

Regarding potential next steps for triangulation, Steenland asserted that the definition of triangulation and how it differs from sensitivity or risk of bias analysis remain unclear. Additional examples that demonstrate triangulation’s effectiveness could be beneficial, as could studies that identify situations where triangulation shows discordant results. Tchetgen Tchetgen concurred, adding that wider engagement of researchers, including through workshops, would help them to understand sources of bias and ways to integrate and account for them. However, he noted that this effort will require changes in how younger researchers are trained and incorporated into incentive systems. “A cultural shift needs to happen,” he said.

Lawlor called for more frequent use of triangulation, even in the same study. “This will take more time. But it will provide a greater understanding,” she said. Also needed is a focus on the possible range within which the causal result may lay and being open to uncertainty, she said. Pearce likened a formalization of triangulation to turning around a ship and suggested several future focus areas: developing a methodology for triangulation (also cited by Lawlor and Tchetgen Tchetgen); returning to thinking about epidemiology as studying populations—citing smokers, healthy eaters, and exercisers as distinct populations, for example; prioritizing international comparisons rather than narrowly focusing studies; and educating committees about triangulation and the importance of generating evidence from a variety of approaches. He cautioned,

During a discussion about how selecting certain studies might bias the outcome (i.e., finding a causal effect), Lawlor noted the benefits of publishing pre-analysis protocols or statistical analysis plans before data are made available and a study begins. Here, researchers cannot pre-select studies to use. Excluding a study would not be possible based on expectation of a particular answer. Working within consortia also helps to control for this possibility.

Lawlor also welcomed registration of observational human studies to ensure that epidemiologists use the appropriate methods to facilitate the use of observational studies in other contexts. She noted ongoing efforts to secure prior publication of protocol and analysis plans before data are released. These plans must stipulate study intent and purpose, as well as how the study will compare, contrast, integrate, and conduct sensitivity analysis, she said. Collectively, this transparency could help future researchers design and conduct studies. Yet, ongoing challenges persist, including difficulties with governance, more so than for RCTs whose questions are usually limited. Moreover, UK cohort studies will not be funded

___________________

⁴ For additional background on GRADE, see https://bestpractice.bmj.com/info/toolkit/learn-ebm/what-is-grade.

unless they are widely available to academia, industry, and higher education students to use and analyze. Early attempts include the UK biobank, which requires a proposal, as does the Avon Longitudinal Study of Parents and Children cohort. Although the increased transparency will likely be helpful, this process could slow the pace of research, and, by extension, triangulation work, because registering and publishing data and methodologies should be done within each individual cohort, in each triangulation framework and protocol, and with trust that no one would exclude studies or data that are complex or have limitations, she said.

A similar development is the increasingly common practice of journals requiring published meta-analyses to facilitate risk of bias appraisals. Regarding whether this practice serves its purpose, Pearce commented that many situations for which triangulation of all available data might be most useful lack sufficient quality data. These scenarios might include questions about whether exposure to a certain chemical causes cancer or heart disease or, alternatively, whether a meta-analysis is using the best available studies for a quantitative estimate.

Considering practices to integrate findings from toxicology, mechanistic studies, and epidemiologic studies, Pearce called IARC’s approach—one of a few examples of a formal process—important and valuable. Steenland agreed, noting the utility of having some evidence, including toxicology studies and mechanistic data, when there is an observed association between an agent and disease in observational epidemiologic studies. For example, mechanistic studies of ethylene oxide provided support for its evaluation as a Group 1 carcinogen. Mary Schubauer-Berigan concurred on IARC’s method, noting its use for more than 50 years in the Monographs Program. Many noteworthy developments have been drawn from mechanistic evidence evaluation, she said, including the identification of the 10 key characteristics (KCs) of carcinogens. KCs comprise the properties of the known human carcinogens and reflect the different mechanisms through which cancer is caused. For instance, ethylene oxide is genotoxic, whereas 2,3,7,8-tetrachlorodibenzo-para-dioxin modulates receptor-mediated effects, and etoposide alters DNA repair.⁵ Schubauer-Berigan stated that the KCs have facilitated learning and transferring knowledge from established carcinogens to help to identify new carcinogens (Smith et al., 2016). As later explained by Martyn Smith, the KCs of carcinogens were established 10 years ago, and KCs of other types of toxicants (including endocrine-disrupting chemicals and cardiotoxicants) have since been developed.

Next, Aisha Dickerson, the session’s committee moderator, asked the panelists whether scoring and ranking studies based on risk of bias is useful when tools such as the Office of Health Assessment and Translation and the Navigation Guide evaluate bias without requiring scoring of studies. Pearce noted that scoring and ranking in this case are usually not helpful in part because case-control studies always score lower than cohort studies. He advised instead to use Savitz’s approach to listing possible biases for each key question in each study, and then assessing each study separately for a particular type of bias, its likelihood of occurrence, and likely strengths and direction. “That helps one contrast and put the evidence together,” he explained. One notable exception for ranking might be when, for example, based on associations, a meta-analysis is preferable. “Then … choose the best, most reliable studies—but with lots of checks and sensitivity analysis,” he advised.

Tchetgen Tchetgen noted the challenges of addressing life course epidemiology with current methods, notably the timing of exposure, to which Lawlor had alluded. Many cohorts start later in life, he said, resulting in “a huge truncation of the relevant part of the exposures in childhood or younger ages [that] may confound exposures later in life.” Unfortunately, many tools developed from causal inference literature do not apply to many of these cases. He pointed to marginal structural models and Mendelian randomization, which often fail in the context of life course epidemiology because they fail to address exposure that occurred prior to the start of the cohort. Lawlor concurred, noting that exposure timing and life course are often issues.

___________________

⁵ See https://monographs.iarc.who.int/iarc-monographs-preamble-preamble-to-the-iarc-monographs.