6
Final Thoughts and the Future of Triangulation
Faustman moderated a summary of workshop sessions. Moderators of the earlier sessions provided an overview of key themes and reflections, which was followed by a panel discussion of the full committee.
First, Dickerson summarized the introductory session of the workshop. She noted that triangulation has been widely used in several scientific enterprises, including but not limited to math, physics, sociology, and epidemiology. Specifically, it has been used as a conceptual framework to integrate multiple sources of evidence with known properties to infer a measure that cannot be observed directly. For this reason, Dickerson noted, Lawlor and her colleagues suggested in their 2016 paper that triangulation may be used as a method to strengthen causal inferences and etiologic studies of epidemiology. By integrating results from several studies that used different approaches instead of depending on individual studies, one could better synthesize information and potentially infer causality. However, with triangulation, it may be useful to account for duration and timing of the exposure and be explicit about the key sources of potential bias unique to each study included in this multi-parameter synthesis.
Dickerson noted that limitations to triangulation appear to be the same as those commonly present in epidemiologic studies, including misclassification and measurement error, selection bias, and missing data, all of which can present challenges for causal inference assessments. She noted that Tchetgen Tchetgen indicated that these limitations need to be addressed comprehensively, rather than as an acknowledgment of residual bias at the end of the process. Triangulation often compares studies with varied study designs. Furthermore, Tchetgen Tchetgen proposed use of multiple robustness techniques to combat these issues. In this way, triangulation could help to identify causal parameters’ interest at the individual study level, to obviate the need to perform cross-study triangulation to address confounding and then synthesize the studies.
Additionally, Dickerson noted that Lawlor had acknowledged that triangulation is primarily based on selected qualitative comparison. However, Pearce had pointed out that this selection is still compounded by issues with publication bias. Pearce had also highlighted that many studies are limited by the populations that can be accessed, which, in turn, limits the populations to which triangulation results can be applied. Until more studies are conducted in low- and middle-income populations, results of triangulation will not be truly valid. However, triangulation’s strength is that it eliminates issues with ranking and exclusion of studies based on perceived study quality, a theme also noted by others. Moreover, many participants concurred that the practice of ranking in study quality evaluation should not be used for future environmental health study evaluations, or synthesis.
Next, planning committee member Laura Beane Freeman of the National Cancer Institute summarized the session on health authority approaches to evidence synthesis. Several workshop speakers noted that a comprehensive and transparent method may be necessary for environmental epidemiological assessments. Triangulation approaches were discussed on multiple levels: within and across studies, across evidence streams both within epidemiology, and across other toxicology and mechanistic data streams. It may seem to be obvious that consistency across evidence streams increases the confidence in the overall assessment.
Another theme highlighted multiple times is that several participants consider high-quality exposure assessment to be critical. When considering inferences, bias is thought of in the context of positive associations when none exist. However, in truth, non-differential misclassification of exposure as a bias will often reduce the ability to detect an association. Among next steps, expanding knowledge of
mechanisms can help inform understanding and confidence in assessments, as noted by Schubauer-Berigan in her presentation of the International Agency for Research on Cancer’s reviews and elaborated on by Martyn Smith’s presentation on the key characteristics (KCs). Future assessments will likely incorporate formal bias assessments not only within but also across studies and will not only note that bias may exist but also incorporate information about its direction and potential impact. Panelists also noted the need to develop methods to assess multiple exposures, including mixtures. Statistical approaches exist to incorporating such methods into individual studies. “But how should one consider these in human risk assessments or hazard identification frameworks, as well as methods for incorporating effects due to cumulative risk?” Beane Freeman asked.
Following Beane Freeman’s summary, Richardson summarized the triangulation case studies session. He noted that the discussion began with triangulation as a way to integrate evidence from different approaches rather than a way to consider the weight of evidence. That is, triangulation is neither simply about pulling together all available evidence from different streams, nor simply assessing heterogeneity or effect measure modification. Triangulation encompasses thoughtful and explicit consideration of the different types of biases that pull results in different directions in key studies. In a review, inclusivity is favored because some clearly flawed studies suffer bias but in a predictable direction. These studies may be set against a different study that might yield a consistent and unbiased estimate.
The definition of triangulation had generated lively discussion in Session IV. Several participants noted that triangulation entails pulling together evidence from different approaches, particularly those with different key sources of bias, and then comparing results. Thus, multiple presenters explained approaches to leveraging differences in designs, methods, and analyses. This thinking could also inform researchers’ approaches to new studies, to address aspects that are unavailable with different populations, designs, and methods—for example, by performing a new study with measured confounders, a valid instrumental variable, or a negative control outcome. Among next steps, some speakers proposed formalizing the argumentation.
Whether the word triangulation or a different nomenclature is used, an a priori set of questions and formalized lines of argumentation may be helpful. Specifically, what is needed for valid negative control exposure, a valid negative control outcome, a valid instrumental variable, or genetic instrument for a valid natural experiment? Collectively, these questions fall under the rubric of what is considered to be triangulation, that is, a series of methods for making logical argumentations. An a priori set of questions can help to formalize the argumentation, which can in turn inform an a priori understanding of key questions, many of which are standard concerns in epidemiologic reviews about measurement error, selection bias, and confounding.
Finally, Chartres summarized the workshop session on possible next steps and opportunities to apply triangulation. Chartres noted that, among key takeaways from this session, rigorous systematic reviews address many of the issues that triangulation strives to address, also cited by Bero, Woodruff, and Taylor with parallels in earlier presentations by Nachman and Lunn. Assessments are currently being done by looking at different study designs and different populations, accounting for different potential confounders as well as considering non-differential exposure misclassification and by applying meta-analytic techniques. A common theme emerging from the workshop was the need to use the best study designs to answer the question. A salient example was Bero’s reference to Cochrane’s movement away from labels to more specifically assess the attributes of different studies. Appropriate valuation of observational studies is important. Persistent challenges within systematic reviews pertain to (1) their definition and use, (2) the misuse of risk of bias tools, and (3) a lack of data to quantify bias because of reporting limitations in many studies, also mentioned by Schubauer-Berigan. More standardized ways to report how studies were conducted merits thought, to provide more rich data for risk of bias assessments.
Regarding next steps, Samet and many panelists had suggested first examining previous recommendations by the National Academies and others to improve systemic reviews. This could be a first step, as opposed to exploring a new approach. Though triangulation may have some validity, several panelists noted that the construct is not yet well defined or agreed upon.
Another possible next step involves including all studies in evidence synthesis, and moving away from risk of bias checklists or approaches that remove studies from the evidence base. Instead, researchers could judiciously review a study’s individual strengths and limitations before starting a review. Balancing expert judgment both in systematic reviews and risk of bias processes and ensuring that important biases are captured when considering individual study evaluations, are important next steps, as also noted by Chang. The direction and magnitude of biases within individual studies also deserves additional thought, as does the potential impact of these biases on the effect estimate. Already, experts in this field note that sensitivity analyses aid in this endeavor. Involving different relevant experts in this process will likely help ensure that the question posed maps to the answer sought, and by extension, to the exposure and outcome. Rigorous consideration of important biases, within a framework and structure, could point to the way forward.
Lastly, the KCs provide a basis for an objective approach to evaluating mechanistic data. This more agnostic approach based on the KCs helps to avoid bias and identify mechanisms with incomplete data. The KCs can help advance current thinking about mechanistic data, and more effort on best practices, including for evaluation of the quality of evidence relevant to KCs, is important as emphasized by Smith.
Themes emerging from the committee discussion included opportunities to improve publications, mixtures, and pooling of data. Planning committee member Joyce Tsuji of Exponent. Inc., suggested that reviewers and editors could request supplemental information on aspects of data collection, results, and different sub-analyses to improve the utility of publications. Other experts external to the study could then assess and incorporate these elements into a bigger analysis when combining studies. Faustman concurred, urging sufficient metadata to support such analysis, given the global emphasis to save data in a reusable manner. Planning committee member Chirag Patel from Harvard Medical School encouraged development of standards and tools to make the data provided by study authors more amenable to computational approaches and facilitate systematic reviews, which are resource intensive. However, as with triangulation, “Experts with knowledge will be needed to make sense of this. More elaborate tools do not mean better answers,” Patel cautioned.
Regarding mixtures, Dickerson cautioned that the U.S. Environmental Protection Agency’s work is generally toxicant specific. However, Faustman countered that this is not always the case, citing the example of benzopyrene. Beane Freeman reflected that defining the right question for the goal is critical: “Is the goal measuring nitrogen in the atmosphere, or assessing exposure to traffic pollution? Is the goal the science or changing people’s risks through regulation?” In this vein, Beane Freeman noted that the formaldehyde example showed that triangulation can be used for co-exposures that may exist in some circumstances but not in others. Arriving at the same answer adds confidence that the co-exposure does not confound that relationship. Chartres and other panelists reflected that this example more broadly points to the need for an a priori list of considerations for assessments. “Might other cohort studies be informative, or serve as example for the question for an answer that a priori is sought? What might be available and helpful?” Faustman asked. Chartres concurred, asking “But how to do this in a systematic way, not hand selecting studies, but finding studies with contrasts, and those that include different populations or confounding factors?”
Dickerson noted that new methods, developed progressively, could inform how to handle exposure to mixtures. She and Beane Freeman reflected that the considerations range from complex mixtures such as air and water pollutants to co-exposures in occupational settings, as well as encompass additional factors such as psychosocial stressors. On the topic of sufficient similarity in mixtures, planning committee member Scott Auerbach at the National Toxicology Program identified the assessment of the diversity of mixtures as a key challenge. However, because people are exposed to consistent and sometimes relatively uniform sets of mixtures (with some variability), synthesizing data across a sufficient similarity paradigm could help address some mixture issues.
Faustman added that new approach methodologies in toxicology, in silico approaches, more advanced quantitative structure-activity relationship information, and metabolomics might offer new opportunities to address mixtures, in addition to other evidence streams. However, these approaches
might complicate rather than simplify assessments of mixtures. Tsuji called out the complexities of co-exposures because their effects are observed well after the time of exposure. “Things will worsen, as seen in the exposome example, because the prenatal experience may make one more susceptible to something later in life. That aspect is rarely considered in epidemiologic studies. Many co-exposures or co-experiences influence, including socioeconomics and location. These aspects are very difficult to incorporate in studies,” she said. By extension, other questions include “How [do we] consider other factors like nutrition? Some environmental justice issues touched on and more holistic contexts query the definition of well-being and other exposures in the context of tribal nations. What might be the extent of effort for these broader constructs?” Faustman asked.
Patel introduced the topic of pooling: “Is the inter-individual variation cast at our meta-analyses and systematic reviews, such as covariate selection, co-mixtures, co-exposures too much to ask for some of these high stakes’ systematic reviews?” He identified pooled data sets and the required resources for such studies as “his central wish.” He also urged shared observation of how the variation in study designs, assumptions and inclusion criteria can funnel into a single summary statistic that eventually makes it into systematic reviews—and that, at a minimum, the assumption of using pooled versus not pooled data are tested. Faustman added that pooling data is a topic that may warrant further discussion, specifically regarding exposures across cohorts and internationally. Richardson raised concern about the time-consuming nature of pooling studies and noted barriers to data access, especially for studies from multiple countries.
Pooling offers the opportunity to perform analyses under common protocols and statistical models, using common variable selection and operationalization. It may require avoiding derivation under the model fitted to the published literature when conducting only meta-analyses of a risk estimate, Richardson said. Although a better direction, this approach can substantially slow the analysis. Freeman and Dickerson concurred. “The challenges of getting access to and pooling data are not trivial,” said Beane Freeman. She noted that different methods of exposure assessment across studies can make pooling challenging because they may estimate (measure) different aspects. In addition, even if studies use common estimates of exposure, the distributions may be different and non-overlapping in different populations. For example, all the highly exposed people may come from one study population and all of the low-exposed people from another.
Faustman highlighted opportunities afforded by the National Health and Nutrition Examination Survey (NHANES) and National Institute of Environmental Health Sciences support of the Human Health Exposure Analysis Resource projects to address exposure components across cohorts. “Such data is just now becoming available. This means that their exposure measures may be more elaborately standardized and characterized than those at NHANES,” she explained. Richardson added that this context provides stronger and cleaner logic for in-study triangulation because when pooling individual-level data, in-study triangulation requires fewer assumptions than does cross-study triangulation. The idea of pooling followed by triangulation within a study sample may merit exploration.
Faustman closed the workshop with reflections on several workshop themes. First, triangulation is not new; epidemiologists have used this construct for many years. “But is triangulation using science-based common sense or, as cited in key papers (Lawlor et al., 2016), provid[ing] examples of criteria to be considered in addressing sources of bias?” she asked.
Second, many definitions of triangulation exist. While it may be beneficial to consider and consolidate some of these definitions, researchers may avoid reinventing the wheel by fine-tuning and integrating the approach into systematic review practices. At the start of the review process, considerations that align with triangulation methods could ensure that an evaluation addresses key questions posed. The critical aspect of thoughtful problem formulation in hazard risk assessments is a key message. Faustman noted the importance of ensuring that science drives these considerations while aligning efforts with key questions to make such activities more applicable.
Finally, the case study examples highlight a variety of important considerations and lessons learned. Panelists raised concerns about the use of checklist approaches, the limiting of studies to randomized controlled trials, and the exclusion of studies from evidence integration. They also stressed
the need to appropriately value studies of different designs and account for the impact of the magnitude and direction of different biases across studies during evidence integration. Together, the lessons learned can be applied to framing some of the narrative at the start of evidence integration, to be more explicit. Important considerations are the different types of data streams that can be integrated, including differing exposure information (e.g., traffic-based air pollution, radiation) and mechanistic data. Aggregate exposure pathways and KCs are useful for reflecting complexities. “What might this mean in a much broader consideration of factors like nutritional and social factors, health disparities, equity issues and a more holistic framing (e.g., in the context of tribal nations, as presented by Jamie Donatuto)?” Faustman asked. Triangulation examples are not well developed for these considerations and are encouraged as possible next steps.
