1
Introduction
The U.S. Environmental Protection Agency’s (EPA) Integrated Risk Information System (IRIS) Program is the part of the agency’s Office of Research and Development (ORD) that develop human health assessments that focus on hazard identification and dose-response analyses for chemicals in the environment. IRIS assessments cover cancer and noncancer outcomes and include toxicity values (reference values and unit risks) that can be combined with exposure estimates to develop quantitative estimates of risk. The assessments are highly important as they are used to inform risk assessments and risk management decisions throughout the agency, including EPA regulatory programs. The assessments are also used by federal, state and tribal agencies as well as community organizations and agencies in other countries to inform decisions concerning health risk assessment and management.1
Over the years, questions had been raised about the scientific basis of toxicity values reported in some IRIS assessments and the extensive amount of time taken to complete assessments. A 2011 report from the National Academies of Sciences, Engineering, and Medicine that reviewed a draft IRIS assessment of formaldehyde offered recommendations for improving the IRIS process (NRC, 2011). In 2014, the National Academies issued a report that reviewed the IRIS program process and expanded on those recommendations. Although many other recommendations were provided in the 2014 report, it concluded that “substantial improvements in the IRIS process have been made, and it is clear that EPA has embraced and is acting on the [National Academies] recommendations” (NRC, 2014, p. 9). In 2018, a National Academies report reviewed progress toward transforming the IRIS program and concluded that substantive progress has been made in transforming the IRIS program to date. It also found the program to be on track toward full implementation of systematic review and transparency in IRIS assessments (NASEM, 2018).
A major recommendation of the 2014 National Academies report was to adopt systematic review methods throughout the IRIS assessment process, including the steps on scoping, problem formulation, protocol development, study identification and evaluation, and evidence integration (NRC, 2014). Systematic review is “a scientific investigation that focuses on a specific question and uses explicit, prespecified scientific methods to identify, select, assess, and summarize the findings of similar but separate studies. The goal of systematic review methods is to ensure that the review is complete, unbiased, reproducible, and transparent” (IOM, 2011, p. 1).
The 2014 National Academies report recommended that EPA develop a handbook to “provide a single detailed guidance document for all those involved in the development
___________________
1 Additional information on the IRIS program is available at https://www.epa.gov/iris.
of IRIS assessments” and make the IRIS process transparent to stakeholders (NRC, 2014, p. 23). The 2014 report acknowledged the challenges of having one guidance document to meet the needs of EPA staff and consultants who conduct IRIS assessments, as well as stakeholders.
A 2021 National Academies report that reviewed EPA’s Toxic Substances Control Act (TSCA) Risk Evaluations (NASEM, 2021) also recommended the development of a handbook for EPA’s TSCA review and evidence integration methodology. TSCA risk evaluations are conducted by EPA’s Office of Pollution Prevention and Toxics (OPPT), which is separate from the IRIS Program. The 2021 report also recommended that OPPT review its approach and incorporate components of the National Institute of Environmental Health Science’s Office of Health Assessment and Translation systematic review and evidence integration method (NTP, 2019), the University of California San Francisco’s Navigation Guide (Woodruff and Sutton, 2014), and IRIS methods.
The ORD Staff Handbook for Developing IRIS Assessments (the handbook), issued in November 2020, provides guidance to scientists who perform the IRIS assessments in order to foster consistency in the assessments and enhance transparency about the IRIS assessment process (EPA, 2020a; Thayer, 2021). The handbook does not supersede existing EPA guidance and does not serve as guidance for other EPA programs. It also provides an opportunity for stakeholder communities to become aware of the processes and policies guiding those who are drafting IRIS assessments.
THE COMMITTEE, ITS TASK, AND ITS APPROACH
EPA requested that the National Academies conduct a review of the handbook (EPA, 2020a). In response, the Committee to Review EPA’s IRIS Assessment Handbook was convened to evaluate the operating procedures, as described in the handbook, for hazard identification and dose-response assessment of potential human health effects from exposure to environmental contaminants. The committee was also asked to review the procedures and considerations for operationalizing the principles of systematic reviews, and the methods described in the handbook for determining the scope of the IRIS assessments (e.g., health outcomes, routes of exposure), evidence evaluation, evidence integration, and dose-response analyses. The committee’s verbatim statement of task is provided in Box 1-1. The committee included experts in epidemiology, pharmacology, toxicology, statistics, environmental health, dose-response modeling, pharmacokinetic and pharmacodynamic modeling, systematic review, and risk assessment. (See Appendix A for biographical information on the committee.)
EPA provided the committee with a list of detailed questions to further guide the committee’s review that, with one exception (see Chapter 6 of this report), fall within the general outline of the statement of task. Appendix B provides brief descriptions of the organization and content of the handbook chapters and a listing of EPA’s questions for the committee.
In carrying out its task, the committee held eight meetings, including information-gathering sessions on February 11, 2021, and April 16, 2021, to hear presentations from Jonathan Samet, Colorado School of Public Health, and Kris Thayer, EPA, respectively.
The committee also heard public comments during each meeting (see Appendix C for meeting agendas). EPA’s presentation addressed questions from the committee regarding the handbook (Thayer, 2021). In addition, the committee received various written materials, including EPA’s written responses to the committee’s questions and the public comments on the handbook that were submitted to the EPA docket.2
The committee reviewed the handbook, recognizing that the primary audience is internal to EPA but also includes external stakeholders. It developed a list of considerations (Box 1-2) to guide its critique of each chapter of the handbook. The list includes transparency; clarity; organization; terminology; areas of controversy about the method; and whether the described method was accurate, complete, and could be applied by those responsible for conducting IRIS assessments. Not all considerations were relevant to each handbook chapter.
The committee’s review of the handbook is independent of the 2021 TSCA review, although the committee’s findings and recommendations could help inform OPPT’s responses to recommendations in NASEM (2021) and harmonize EPA’s approach to hazard evaluation.
ORGANIZATION OF THE REPORT
The committee structured its report chapters according to the major steps in the IRIS process for developing assessments. Chapter 2 focuses on an overview of the handbook’s organization and content. Chapter 3 considers the handbook’s coverage of planning IRIS assessments, including evidence mapping, scoping, protocol development, the refined evaluation plan, and organization of the IRIS assessment. Chapter 4 focuses on the handbook’s treatment of evaluation of individual studies within a data stream. Chapter 5 covers the handbook’s handling of evidence synthesis and judgments about the strength or cer-
___________________
2 A 90-day public comment period was associated with the release of the handbook. See Federal Register vol. 85, No. 230, Nov 30, 2020. 76566–76567.
tainty of the synthesized evidence within a data stream. Chapter 6 focuses on the handbook’s treatment of evidence integration across data streams and the judgments used to describe the strength of conclusions regarding hazard identification. Chapter 7 covers the handbook’s steps of selecting studies for toxicity derivation and deriving toxicity values based upon dose-response analyses. Table 1-1 maps the chapters of the committee’s report with the questions from EPA and relevant handbook chapters.
As requested by EPA (see Appendix B), the committee organized its recommendations to indicate their relative importance during EPA’s revision of the handbook:
- Tier 1: Recommended Revisions—Highest priority recommendations the committee believes are critical to improve the scientific rigor and/or clarity of the document.
- Tier 2: Suggestions—Recommendations that EPA should consider to strengthen the document.
- Tier 3: Future Considerations—Topic areas that may inform future developments.
These recommendations are outside the immediate scope and/or needs of the current document under review.
TABLE 1-1 Mapping of Committee’s Report Chapters and EPA Questions
Committee Report Chapter | Questions from EPAa | Relevant Handbook Chapter |
---|---|---|
2. Overview of Organization and Content of the Handbook | Q1. Comment on the overall organization of the handbook, in particular on whether key aspects of the assessment process are represented. | Entire Document |
3. Planning Assessments | Q2. Are systematic review approaches used by the IRIS Program (outlined in Chs. 1-5) clearly described and consistent with methodologies considered to be state-of-the-science by experts in the field? | 1-7, 10 |
Q4. Does the handbook clearly lay out a state-of-the-science approach for refinement of the scope and analyses of the IRIS assessment? Are there specific areas for improvement (recommended alternatives)? | ||
Q5. Review and assess the handbook’s process for evaluating and integrating mechanistic data. [Sections 2.2, 4.3.3, and 6.6, and Ch. 10] Are there specific areas for improvement (recommended alternatives)? | ||
4. Study Evaluation | Q3. Are the study evaluation methods in Ch. 6 for individual human studies (epidemiology and controlled exposure), animal studies, mechanistic evidence (pilot testing approaches), and pharmacokinetic models adequate? If not, how can the methods be improved? | 6 |
5. Evidence Synthesis | Q6 (part a). Are the approaches to evidence synthesis described in Ch. 9 and Ch. 10 scientifically sound? | 9-11 |
|
6. Evidence Integration | Q6 (part b). Are the approaches described in Ch. 11 scientifically sound and appropriate for integrating the various types of evidence relevant to investigating the potential for human health effects from exposure to environmental chemicals? | 11 |
Q7. Comment on approaches using five categories versus three categories for drawing evidence integration conclusions. Which approach is recommended and why? Are there any specific refinements for improvement? | ||
7. Hazard Considerations and Study Selection for Deriving Toxicity Values | Q8. Does the handbook provide appropriate considerations in Ch. 12 and Ch. 13 for identifying data sets for dose-response analysis based on systematic review conclusions? Are the basic methods for dose-response modeling and deriving toxicity values consistent with the current state of the science, and presented with sufficient clarity? | 12, 13 |
a Chapter and section numbers listed in the questions refer to those of the ORD Staff Handbook for Developing IRIS Assessments (EPA, 2020a). See Appendix B of this report for additional information.