National Academies Press: OpenBook

Review of U.S. EPA's ORD Staff Handbook for Developing IRIS Assessments: 2020 Version (2022)

Chapter: 7 Hazard Considerations and Study Selection for Deriving Toxicity Values

« Previous: 6 Evidence Integration
Suggested Citation:"7 Hazard Considerations and Study Selection for Deriving Toxicity Values." National Academies of Sciences, Engineering, and Medicine. 2022. Review of U.S. EPA's ORD Staff Handbook for Developing IRIS Assessments: 2020 Version. Washington, DC: The National Academies Press. doi: 10.17226/26289.
×

7

Hazard Considerations and Study Selection for Deriving Toxicity Values

In this chapter, the committee reviews Chapter 12 “Hazard Considerations and Study Selection for Deriving Toxicity Values” and Chapter 13 “Derivation of Toxicity Values” of the ORD Staff Handbook for Developing IRIS Assessments (the handbook) (EPA, 2020a). The committee considered whether those handbook chapters provide appropriate considerations for identifying data sets for dose-response analysis based on systematic review conclusions. The committee also considered whether the basic methods for dose-response modeling and deriving toxicity values are consistent with the state of the science and are presented in the handbook with sufficient clarity (Question 8 in Appendix B.)

OVERVIEW OF THE HANDBOOK’S CHAPTERS 12 AND 13

Chapter 12 of the handbook summarizes the considerations and approaches involved in arriving at hazard judgments to set priorities among health outcomes and select the studies to be used for toxicity value derivation. That chapter also outlines considerations to be used for study selection and information for combining dose-response data.

Chapter 13 discusses the processes used to derive various toxicity values, referencing additional U.S. Environmental Protection Agency (EPA) guidance documents as needed. The chapter also includes directions for characterizing sources of uncertainty and confidence in the derived toxicity values.

RESPONSIVENESS TO PREVIOUS NATIONAL ACADEMIES REPORTS

Previous reviews by the National Academies commented heavily on the processes used for selecting studies, and the current handbook attempted to address these concerns. The National Academies report Review of EPA’s Integrated Risk Information System (IRIS) Process (NRC, 2014) recommended that EPA develop criteria for determining when evidence is sufficient to derive toxicity values. Pertinent to this recommendation, the handbook states that “for both cancer and noncancer hazards, preference is given to health effects (or outcomes) and cancer types with stronger evidence integration conclusions” (EPA, 2020a, p. 12-4). Chapter 11 of the handbook discusses the evidence integration judgments used for noncancer endpoints and cites EPA’s Guidelines for Carcinogen Risk Assessment (EPA, 2005) to provide additional information for evaluating carcinogenic evidence.

The 2014 National Academies report also recommended that EPA “continue its shift toward the use of multiple studies rather than single studies for dose‒response assessment” by developing formal methods for combining data from multiple studies (NRC,

Suggested Citation:"7 Hazard Considerations and Study Selection for Deriving Toxicity Values." National Academies of Sciences, Engineering, and Medicine. 2022. Review of U.S. EPA's ORD Staff Handbook for Developing IRIS Assessments: 2020 Version. Washington, DC: The National Academies Press. doi: 10.17226/26289.
×

2014, p. 129). The handbook includes a section that provides additional considerations and methods for evaluating and combining dose-response data from multiple studies as an alternate approach to using data from a single study. The 2018 National Academies report Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation cautioned that “care must be taken when combining results within or between studies in developing dose‒response relationships inasmuch as multiple mechanisms, each with its own potential dose‒response relationship, might be involved” (NASEM, 2018, p.11). As indicated in Chapter 13 of the handbook, methods for the derivation of toxicity values are detailed in several historical guidance documents and are largely limited to those previous documents. The 2014 National Academies report recommended that EPA develop IRIS-specific guidelines for communicating uncertainty, and, in response, Chapter 13 of the handbook includes a section on characterizing uncertainty and communicating confidence in the derived toxicity values.

CRITIQUE OF METHODS FOR STUDY SELECTION AND DERIVING TOXICITY VALUES

Chapters 12 and 13 of the handbook focus on steps to be taken after systematic review, specifically the processes by which studies are selected and toxicity values are derived, and many aspects of sections in those chapters positively add to the overall assessment process. Chapter 12 aims to standardize the selection of studies for toxicity value development by providing considerations and case-specific recommendations. Chapter 13 addresses development of toxicity values, with most of the sections referring to other EPA guidance documents. Several aspects of both Chapters 12 and 13 would benefit from clarification. Overall, although improvements are evident, the level of detail in these chapters is not sufficient to provide a step-by-step guide for those involved in developing IRIS assessments.

The committee considered whether the handbook provides appropriate considerations for identifying data sets for dose-response analysis based on systematic review conclusions. The methods described to select studies, although not always completely clear, generally align with recommendations from previous National Academies reports. For instance, the handbook indicates that, in general, toxicity values are developed when available evidence is judged either to “demonstrate” or “indicate (likely)” that a chemical causes a health effect in humans under relevant exposure circumstances. However, the handbook is not clear as to which judgments would have been made during hazard identification and which are to be made at this point in the assessment process. For instance, while strength of evidence is mentioned in the introduction of Section 12.2 of the handbook, there is no discussion of how the strength of evidence conclusions, or the considerations that went into drawing those conclusions, affects the selection of endpoints or studies. Additionally, some of the criteria that are described in Chapter 12 as additional factors in study selection, such as “relevance of the test species,” “relevance of the studied exposure to human environmental exposures,” and “quality of measurements of exposure and outcomes,” would have already been considered in either the synthesis or individual study evaluations. Because the handbook does not clearly distinguish between evaluations made

Suggested Citation:"7 Hazard Considerations and Study Selection for Deriving Toxicity Values." National Academies of Sciences, Engineering, and Medicine. 2022. Review of U.S. EPA's ORD Staff Handbook for Developing IRIS Assessments: 2020 Version. Washington, DC: The National Academies Press. doi: 10.17226/26289.
×

as part of hazard identification and considerations specific to toxicity value derivation, an analyst may be led to repeat evaluations made earlier in the assessment process.

Chapter 12 also lacks clarity on how studies should be selected for toxicity values development. The process of study selection is likely to vary on a case-by-case basis, and Chapter 12 does attempt to describe the possibilities that may arise. As judgment is used to select these studies, detailed information will need to be provided in the individual IRIS assessments in order for the process to remain transparent. The handbook states that high-quality or medium-quality studies will be used primarily for toxicity value derivation, and yet qualifications indicate that low-quality studies may also be important, without details on when this may apply. Similar to the previous chapters, Chapter 12 is unclear as to how mechanistic data and other supporting studies should be used, or how this information should be incorporated into the assessment process. The chapter suggests that summary tables be used to document study selection and graphical comparisons be used for toxicity value selection, but no examples are provided, leaving the design up to those conducting an IRIS assessment. Specific studies are described that should not be taken through the assessment, such as non-relevant routes of exposure (excluded in PECO) and studies that do not support point of departure (POD) derivation.

Key questions about how the study selection process incorporates (1) the results of the systematic reviews that support the overall hazard conclusions and (2) additional factors not previously considered in hazard identification include the following:

  • What evidence streams are eligible to be selected? The handbook does not address whether a minimum “summary strength of evidence judgment” (e.g., of at least “moderate”) is generally required for studies within an evidence stream to be considered.
  • Which endpoints within an evidence stream are eligible? The handbook does not address how the factors that increase/decrease certainty in the evidence (see Table 11-1 in the handbook (p. 11-5)) are incorporated into the study selection process.
  • Which individual studies that are relevant to a selected endpoint are eligible? Although the handbook states that studies of “high or medium” confidence are preferred, it does not address how the ratings within each domain of study evaluation are incorporated into study selection.
  • What additional study characteristics are to be considered that are specific to toxicity value derivation? The handbook describes numerous attributes in Table 12-2 (p. 12-6) that are used to evaluate studies for toxicity value derivation, but some of these (e.g., “measurement of exposure”) may overlap with risk of bias or adequacy of reporting that may have been addressed earlier in the assessment. Removing from this step those aspects that have been previously evaluated as part of hazard identification would ensure that they are not “double counted” and that resources could be focused on toxicity value-specific evaluation criteria.
Suggested Citation:"7 Hazard Considerations and Study Selection for Deriving Toxicity Values." National Academies of Sciences, Engineering, and Medicine. 2022. Review of U.S. EPA's ORD Staff Handbook for Developing IRIS Assessments: 2020 Version. Washington, DC: The National Academies Press. doi: 10.17226/26289.
×

The handbook in many places describes what is “preferred”; however, it is not clear whether and how these attributes are tracked and integrated into confidence ratings or rankings of the resulting toxicity values. A summary table or evaluation for each study alongside these attributions would provide transparency and a basis for selection of overall toxicity values.

The committee also considered whether the basic methods for dose-response modeling and deriving toxicity values are consistent with the state of the science and presented with sufficient clarity. In Chapter 13, most of the sections appropriately refer to other EPA guidance documents, but the process of developing a toxicity value is not easy to follow. For toxicity value development, the methods are based on historical guidance documents and generally have not been updated in this handbook. The primary reference in the handbook for noncancer toxicity values, A Review of the Reference Dose and Reference Concentration Processes (EPA, 2002), is not actually a guidance document but a technical review of the (historical) IRIS process for developing reference concentrations (RfCs) and reference doses (RfDs). The technical panel tasked with preparing that review was created in 1999, and the report specifically states that it “is a review, not guidance, but it does make recommendations that should be considered in the implementation of changes in the current process and/or development of needed guidance” (EPA, 2002, p. xiii). Although methods for evaluating multiple endpoints and combining multiple studies are provided, it is unclear how multiple values should be used to develop a final toxicity value, which may add to the length and complexity of some assessments. Chapter 13 would better serve users engaged in the derivation of toxicity values by citing EPA documents or other published literature that describe details of the extensive methods involved, rather than repeating the details provided in those other documents. When the handbook addresses a concept or method being updated or that is not covered elsewhere, it would need to provide enough detail for guiding a user’s implementation of the approach.

Section 13.2 is confusing because it contains highly varied levels of detail and the ordering of information is hard to follow. For instance, there is a substantial amount of detail on physiologically based pharmacokinetic (PBPK) modeling and exposure route-to-route extrapolation, even though these are not routinely applied or available. Much of the detailed discussion in the chapter of procedures and approaches for those topics involving pharmacokinetic modeling could be streamlined by citing other comprehensive documents or published articles, or moving detailed discussion to an appendix.

The handbook is unclear concerning the use of probabilistic approaches for toxicity value derivation, instead of the traditional approach based on deterministic uncertainty factors. (See NRC (1994, 2009) for a discussion of the two kinds of approaches.) EPA stated that these probabilistic approaches would be routinely applied when feasible, but the handbook only makes a cursory and oblique reference to them (Thayer, 2021). The committee endorses the recommendations from previous National Academies report (NRC, 2014) for the IRIS program to transition away from using traditional deterministic approaches for deriving reference values and instead use probabilistic methods to derive risk-specific values. Since the 2014 National Academies report, several publications have further refined probabilistic approaches (e.g., Chiu and Slob, 2015; WHO & IPCS, 2018)

Suggested Citation:"7 Hazard Considerations and Study Selection for Deriving Toxicity Values." National Academies of Sciences, Engineering, and Medicine. 2022. Review of U.S. EPA's ORD Staff Handbook for Developing IRIS Assessments: 2020 Version. Washington, DC: The National Academies Press. doi: 10.17226/26289.
×

and demonstrated their feasibility both for oral exposures (Chiu et al., 2018) and a case example for acrolein inhalation exposures (Blessinger et al., 2020).

FINDINGS AND RECOMMENDATIONS

Findings and Tier 1 Recommendations

Finding: Chapter 12 of the handbook lacks clarity about how the study selection process incorporates (1) the results of the systematic reviews that support the overall hazard conclusions and (2) additional factors not previously considered in hazard identification. Key questions include the following:

  • What evidence streams are eligible to be selected?
  • Which endpoints within an evidence stream are eligible?
  • Which individual studies that are relevant to a selected endpoint are eligible?
  • What additional study characteristics are to be considered that are specific to toxicity value derivation?

Because the handbook does not clearly distinguish between evaluations made as part of hazard identification and considerations specific to toxicity value derivation, an analyst may be led to repeat evaluations made earlier in the assessment process.

Recommendation 7.1: The handbook should provide a clearer, step-by-step description of study selection, using a framework incorporating the different steps of hazard identification (including study evaluation, synthesis, and integration) as well as new steps specific to toxicity value derivation. The handbook should provide a template for how IRIS assessments are to summarize (e.g., in a table) the study selection process as applied to each endpoint, health outcome, study, and evidence stream in order to provide transparency as to study evaluation for toxicity value derivation, and to support selection of overall toxicity values. It is especially important to capture study attributes for which EPA has designated an option as “preferred” versus “less preferred.” [Tier 1]

Finding: Chapter 13 provides important information relating to issues and considerations for deriving PODs for toxicity values, but it lacks a consistent level of detail for deriving and utilizing PODs. Most of the sections appropriately refer to other EPA guidance documents but sometimes the sections unnecessarily repeat information from those documents, rather than focus on new information to facilitate implementation of frequently used methods. There is a substantial amount of detail on PBPK modeling and route-to-route extrapolation, even though these are not routinely applied or available.

Recommendation 7.2: EPA should streamline Chapter 13 of the handbook, especially Section 13.2, to focus on the most common methods and approaches rather than detailing less common scenarios. For instance, although use of PBPK modeling

Suggested Citation:"7 Hazard Considerations and Study Selection for Deriving Toxicity Values." National Academies of Sciences, Engineering, and Medicine. 2022. Review of U.S. EPA's ORD Staff Handbook for Developing IRIS Assessments: 2020 Version. Washington, DC: The National Academies Press. doi: 10.17226/26289.
×

is designated as “preferred,” it requires too much detail in this handbook to provide instructions on development and application of such models; citing other documents is preferable. If there is important information that is missing from existing EPA documents or from the peer-reviewed literature, these could be provided in an appendix to avoid disrupting the flow of the handbook. Additionally, there may be concerns over providing duplicated information, as any future updates to the related EPA documents would require an update to the handbook as well. [Tier 1]

Finding: The handbook is unclear as to the use of probabilistic approaches to replace the traditional deterministic uncertainty factor-based approach for toxicity value derivation.

Recommendation 7.3: EPA should make it explicit in the handbook that probabilistic approaches to derive risk-specific doses will be routinely applied where feasible, referencing recent literature including a 2020 case study on acrolein (Blessinger et al., 2020. EPA should also consider when and how to transition fully away from the traditional deterministic approach to adopt risk-specific doses for its IRIS toxicity values. [Tier 1]

There are no Tier 2 recommendations in this chapter.

Finding and Tier 3 Recommendation

Finding: Chapter 13 of the handbook is primarily a reference chapter that directs users to various historical EPA guidance documents for deriving toxicity values, most of which have not been updated to reflect the most recently used procedures. The IRIS program and its handbook would benefit from being able to refer to updated EPA guidance.

Recommendation 7.4: EPA should consider updating existing EPA guidance documents, notably those used in the development of oral RfDs and inhalation RfCs. [Tier 3]

Suggested Citation:"7 Hazard Considerations and Study Selection for Deriving Toxicity Values." National Academies of Sciences, Engineering, and Medicine. 2022. Review of U.S. EPA's ORD Staff Handbook for Developing IRIS Assessments: 2020 Version. Washington, DC: The National Academies Press. doi: 10.17226/26289.
×
Page 76
Suggested Citation:"7 Hazard Considerations and Study Selection for Deriving Toxicity Values." National Academies of Sciences, Engineering, and Medicine. 2022. Review of U.S. EPA's ORD Staff Handbook for Developing IRIS Assessments: 2020 Version. Washington, DC: The National Academies Press. doi: 10.17226/26289.
×
Page 77
Suggested Citation:"7 Hazard Considerations and Study Selection for Deriving Toxicity Values." National Academies of Sciences, Engineering, and Medicine. 2022. Review of U.S. EPA's ORD Staff Handbook for Developing IRIS Assessments: 2020 Version. Washington, DC: The National Academies Press. doi: 10.17226/26289.
×
Page 78
Suggested Citation:"7 Hazard Considerations and Study Selection for Deriving Toxicity Values." National Academies of Sciences, Engineering, and Medicine. 2022. Review of U.S. EPA's ORD Staff Handbook for Developing IRIS Assessments: 2020 Version. Washington, DC: The National Academies Press. doi: 10.17226/26289.
×
Page 79
Suggested Citation:"7 Hazard Considerations and Study Selection for Deriving Toxicity Values." National Academies of Sciences, Engineering, and Medicine. 2022. Review of U.S. EPA's ORD Staff Handbook for Developing IRIS Assessments: 2020 Version. Washington, DC: The National Academies Press. doi: 10.17226/26289.
×
Page 80
Suggested Citation:"7 Hazard Considerations and Study Selection for Deriving Toxicity Values." National Academies of Sciences, Engineering, and Medicine. 2022. Review of U.S. EPA's ORD Staff Handbook for Developing IRIS Assessments: 2020 Version. Washington, DC: The National Academies Press. doi: 10.17226/26289.
×
Page 81
Next: References »
Review of U.S. EPA's ORD Staff Handbook for Developing IRIS Assessments: 2020 Version Get This Book
×
Buy Paperback | $25.00 Buy Ebook | $20.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

The U.S. Environmental Protection Agency's (EPA) Integrated Risk Information System (IRIS) program develops human health assessments that focus on hazard identification and dose-response analyses for chemicals in the environment. The ORD Staff Handbook for Developing IRIS Assessments (the handbook) provides guidance to scientists who perform the IRIS assessments in order to foster consistency in the assessments and enhance transparency about the IRIS assessment process. At the request of the EPA, this report reviews the procedures and considerations for operationalizing the principles of systematic reviews and the methods described in the handbook for determining the scope of the IRIS assessments, evidence integration, extrapolation techniques, dose-response analyses, and characterization of uncertainties.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!