3

Scoping and Problem Formulation

Several National Academies of Sciences, Engineering, and Medicine reports have highlighted the importance of scoping and problem formulation in risk assessment. For example, Science and Decisions: Advancing Risk Assessment noted that “increased emphasis on planning and scoping and on problem formulation has been shown to lead to risk assessments that are more useful and better accepted by decision-makers” (NRC 2009, p. 6). As shown in Figure 3-1, these are the first steps in performing a toxicological assessment. Scoping and problem formulation help to define the purpose of the assessment, to identify what health outcomes to evaluate, and to determine the type of evaluation to be performed. These steps play an important role in all risk assessments that rely on literature searching and data gathering methods, whether they are narrative or systematic reviews, because they established the parameters and techniques for the subsequent analyses. In the sections below, the committee evaluates the U.S. Department of Defense’s (DOD’s) conduct of these steps.

SCOPING

Scoping exercises are used to survey the literature and other pertinent information (e.g., reviews by authoritative bodies¹) to determine the extent, range, and nature of information available on the topic, identify data gaps, and consider whether additional research might be needed. Scoping should consider potential actions that might be taken (e.g., establishing an occupational exposure level [OEL]) that will help determine the level of scientific depth and rigor required to inform any such actions. Scoping may also include stakeholder input from toxicologists, industrial hygienists, and others within DOD, as well as from stakeholders external to DOD. Examples of input that stakeholders could provide might relate to concerns about certain exposures or populations of concern.

___________________

¹ Authoritative bodies include the International Agency for Research on Cancer (IARC), the National Institute for Occupational Safety and Health (NIOSH), the National Toxicology Program (NTP), the U.S. Environmental Protection Agency (EPA), the U.S. Food and Drug Administration, the National Academies, the Agency for Toxic Substances and Disease Registry, and the American Conference of Governmental Industrial Hygienists (NASEM 2018).

DOD’s objective was to develop an OEL for trichloroethylene (TCE). Given the large amount of data on the chemical, the agency did not conduct a completely new toxicological review but used an assessment of TCE conducted by the U.S. Environmental Protection Agency (EPA) for its Integrated Risk Information System (IRIS) program (EPA 2011) to identify older literature (published before 2010), and conducted its own literature searches to identify newer studies. The committee agrees that this was a practical approach, but it was difficult to understand from the presentation of the evidence how DOD concluded that TCE “could produce toxic effects on the nervous system, liver, kidney, immune system, reproductive system and the developing fetus” (Sussan et al. 2019, p. 1). A more transparent method of documenting this process would have been facilitated by the inclusion of an evidence table or map that provides a descriptive or visual summary of data availability from the EPA assessment or a formal scoping exercise (NRC 2014; Miake-Lye et al. 2016). Evidence tables or maps help to identify data-rich subjects on which reviews might be conducted and identify data gaps that might need to be addressed before conducting the hazard assessment. Alternatively, a more formal scoping review may be considered at this stage, particularly when a systematic review is planned (e.g., Colquhoun et al. 2014).

PROBLEM FORMULATION

Problem formulation is the process of defining the scope of a problem, formulating one or more specific questions about it, and establishing the assessment methods needed to address the questions. Problem formulation should also lead to the development of clear eligibility criteria (inclusion/exclusion criteria) and identification of methods to be used for data management and extraction. Several National Academies reports have provided guidance on the problem formulation step in decision-making (NRC 2009, 2014; NASEM 2017). Important elements in problem formulation include specifying the agent of concern, the relevant routes of exposure, the health end point(s) to be evaluated, and the types of evidence that will be considered (NASEM 2018). Evidence for hazard assessments may come from human studies, animal studies, and mechanistic² studies. Each form of evidence has its own inherent strengths and weaknesses that need to be considered (NRC 2014). As mentioned earlier, problem formulation can be applied for many types of reviews. DOD recognized the importance of problem formulation in its process for deriving an OEL for TCE. Indeed, the first step in its process is defined as “Develop problem formulation (PECO) criteria,” which suggests DOD’s intent to perform a systematic review.

A primary goal of problem formulation is the development of the review question. A properly formulated question is important in designing the literature search strategy and setting eligibility criteria for screening the literature. Defining specific review questions is especially useful for data-rich chemicals like TCE. An example of a focused review question that could be developed for a TCE hazard assessment is: Does developmental exposure to TCE result in cardiotoxicity? The draft document reviewed by the committee does not explicitly present the review questions to be addressed by the study. Developing explicit review questions will strengthen the TCE hazard assessment, improve transparency, and increase process efficiency. DOD should avoid developing overly broad research questions, which can result in retrieving heterogeneous studies that complicate the analysis and may prevent the effective integration of the evidence needed to reach a conclusion (NASEM 2017).

Another goal of problem formulation in a toxicological systematic review is the development of a Populations, Exposures, Comparators, and Outcomes (PECO) statement that helps guide the review. Box 3-1 provides the verbatim PECO statement developed for DOD’s TCE assessment.

A major challenge in the problem-formulation step is determining which adverse outcomes are of potential concern. DOD’s PECO statement for TCE reflects this challenge by stating that “all relevant adverse outcomes” will be considered. The decision to include all adverse outcomes unnecessarily increases the scope and complexity of the TCE hazard assessment. An adequately performed

___________________

² Studies for understanding the cellular, biochemical, and molecular mechanisms of toxicity, including toxicokinetics. They may be in vitro or in vivo laboratory studies.

scoping and problem formulation step can help narrow the focus of the TCE assessment to an appropriate subset of outcomes for determining a health protective inhalation OEL. Another alternative is to convert a single large review into several smaller reviews to address specific populations, exposures, or outcomes of interest. For example, DOD could perform separate reviews of the immunotoxicity and cardiotoxicity data if scoping and problem formulation identified these outcomes as the main drivers for deriving the OEL.

The committee notes that some methodological considerations (e.g., “weight” or “value” of studies) are currently included in DOD’s PECO statement. Inclusion of methods in the PECO statement unnecessarily complicates the statement and can complicate the review process. A properly formulated PECO statement is restricted to characterizing the question to be addressed thus specifying the criteria that will be used to determine study relevance for inclusion or exclusion from consideration. Eligibility of a study should be based on its relevancy and merits and not on what other evidence is available.

The following guidance on problem formulation should be considered by DOD for identifying outcomes to focus on and developing the review questions:

• Work with an information specialist (i.e., specially trained librarian) to design and conduct a broad literature search for scoping to identify possible outcomes associated with the chemical under investigation. Engaging an information specialist could strengthen the use of previously published authoritative assessments, such as those produced by EPA’s IRIS program (EPA 2011). Information specialists can also assist DOD in incorporating references identified in a previously published assessment into the new review.
• Compile an evidence table or map to guide the formulation of specific questions to be addressed in the review(s). The evidence table or map could build on summaries published in a previously conducted review. This exercise would document which toxicological outcomes have been examined scientifically and have sufficient evidence that might warrant further analysis in a review.
• Once a topic is selected for further analysis, the next step is to formulate the questions to address and develop an approach to the investigation, which will involve consideration of the scientific evidence, expert judgment, and relevant stakeholder perspectives.
• Each hazard-specific question should specify (1) the specific chemical(s)³ being evaluated and sources and pathways of exposure, (2) the general types of studies of interest (e.g., in vitro, animal in vivo, human clinical, and epidemiological studies), and (3) the outcomes of interest and the organ system potentially affected.
• Decisions about which specific outcomes should be further evaluated requires expert judgment and should consider whether the potential outcome is likely to occur at doses encountered by workers. DOD should also consider what the significance of the outcome will be if the potential association suggested in

___________________

³ Metabolites could also be identified at this step.

DOD’s problem formulation does not clearly indicate how in vitro and in vivo laboratory studies at the cellular, biochemical, and molecular levels and other mechanistic data might be used in the assessment. Mechanistic information often provides support for the biological plausibility of in vivo observations in animal or human studies. In some cases, mechanistic data can influence hazard identification and dose-response assessment (NRC 2014). Clearly documenting the role of in vitro data within the evaluation process upfront (ideally in a protocol) is necessary for ensuring transparency in the conduct of the review.

Identifying methods to assess individual studies and integrate findings across a body of evidence are other important results of the scoping and problem formulation process. For systematic reviews, these methods are pre-specified in a protocol (discussed further in Chapter 4 of this report). Problem formulation also helps to guide how the dose-response assessment will be performed. For example, the availability of a physiologically based pharmacokinetic model allowed DOD to plan for using it in the assessment (discussed further in Chapter 5).

SELECTION OF THE HAZARD ASSESSMENT METHOD

By the end of the initial problem formulation steps, DOD may consider different approaches for conducting the hazard assessment. Options DOD could consider include performing a new narrative or systematic review, relying on an assessment performed by an authoritative body, or updating a high quality narrative or systematic review. Each of these approaches has advantages and disadvantages. Table 3-1 provides a summary of the advantages and disadvantages associated with narrative and systematic reviews. Well-conducted systematic reviews are the “gold standard” of evidence synthesis. However, the higher level of rigor and transparency may come with the cost of increased time, resources, and effort to conduct the systematic review. Relying solely on an assessment performed by another body can be extremely cost effective and efficient but requires that the previous review meets the needs and objectives of DOD with respect to populations, exposures, and outcomes of interest. It is also critical that the previous review was completed recently and deemed complete by DOD. Updating an existing systematic review is an option that has been recommended by previous National Academies committees and a framework for this effort has been developed. For example, NASEM (2017) demonstrated this process by applying the ROBIS tool (Whiting et al. 2016) to evaluate risk of bias in a published systematic review. That report also points to the importance of having the objectives and PECO frameworks match the target question addressed by the updated systematic review. Another challenge is whether the methods used by DOD are compatible with those used by the previous review authors. Additional considerations regarding the selection of approaches are described in Chapter 6.

FINDINGS

The scoping and problem formulation steps in the draft DOD TCE assessment resulted in a broad PECO statement that considers multiple outcomes of interest. Unlike a formal systematic review, the draft DOD document did not articulate research questions that would be addressed. There is a critical need for DOD to consider what type of review (e.g., narrative, systematic, scoping, or other⁴) best meets its needs to support hazard identification, given the time frame, resources, and policy determinations. This decision is best made early in the scoping and problem formulation processes. If a systematic review best meets DOD needs, then adherence to best practices for the conduct of a systematic review will result in the most scientifically sound and defensible product (see Chapter 4).

TABLE 3-1 Advantages and Disadvantages of Narrative Review and Systematic Review Methods

___________________

⁴ Some authors have suggested that there are nearly 15 different types of reviews (Grant and Booth 2009).

The committee identified the following ways in which the draft TCE assessment could be strengthened:

• Better description of the methods used to scope the literature, including whether and how an information specialist assisted in the process. For example, the assessment should explicitly explain how the IRIS TCE assessment (EPA 2011) was used to scope (e.g., search, assess, analyze) the literature and formulate the problem.
• Create an evidence table or map from scoping to help DOD refine the review to a more limited number of outcomes. Narrowing the focus of the review can improve efficiency. The decision process used by DOD for determining which outcomes to evaluate should be carefully described.
• Develop one or more focused research questions to help drive the review process. The review questions and their associated PECO statements should be clearly linked to improve transparency.
• Clarify how mechanistic data might be identified and used in the hazard assessment.

Although these recommendations apply to DOD’s TCE assessment, the committee anticipates these suggestions will improve future assessments performed by DOD to derive OELs for other chemicals, for either narrative or systematic reviews.

REFERENCES

NRC. 2014. Review of EPA’s Integrated Risk Information System (IRIS) Process. Washington, DC: The National Academies Press.

Sussan, T.E., G.J. Leach, T.R. Covington, J.M. Gearhart, and M.S. Johnson. 2019. Trichloroethylene: Occupational Exposure Level for the Department of Defense. January 2019. U.S. Army Public Health Center, Aberdeen Proving Ground, MD.