5

Lessons Learned and Reflections on the Statement of Task

During the course of its deliberations, the committee developed a general strategy to collect and evaluate information regarding low-dose effects (see Chapter 2). The committee then applied two aspects of its strategy (investigation and analysis) to consider whether exposure to phthalates (see Chapter 3) or polybrominated diphenyl ethers (PBDEs; see Chapter 4) was associated with low-dose endocrine effects. The committee principally used systematic-review methods to complete and document these aspects. This decision was driven partly by the statement of task, which specified that the committee should complete systematic reviews of human and animal toxicology data for two or more chemicals that affect an endocrine hormone system. The committee’s reliance on systematic-review methods is also consistent with recommendations made by previous National Academies committees (NRC 2014a,b).

The focus of the report on systematic review does not imply that it is the only evaluation tool needed to address all questions about low-dose toxicity. As described in Chapter 2, other options for investigation and analysis are available for use instead of or in conjunction with systematic review. Selection of the approaches will depend on the nature of the question and the potential health, social, and economic implications the answer will have. The committee strategy also describes potential options for future actions. For example, one potential action could be an update of an existing toxicity assessment. Although this type of action is consistent with the iterative strategy developed by the committee, completion of the additional steps needed to support this action were not pursued because the committee was not asked to complete risk assessments for any individual chemicals or classes of chemicals.

The purpose of this chapter is to provide additional discussion of lessons learned as they relate to committee efforts to address the statement of task and related issues concerning low-dose effects of endocrine active chemicals (EACs). It also provides lessons learned from performing the systematic reviews and integrating the human and animal evidence.

COMMITTEE STRATEGY

Development of a generic strategy for evaluating evidence of low-dose effects in Chapter 2 was deliberate and reflected the committee’s desire to provide a framework that could be applied to many agents of concern regardless of the at-risk population, toxicity end point, or mechanism. Application of this generic strategy into one that meets the US Environmental Protection Agency’s (EPA’s) need to evaluate low-dose endocrine effects will require a combination of scientific and policy decisions on the agency’s part. For example, implementation of an active surveillance program will likely necessitate that EPA identify specific EACs, dose ranges, populations, and end points to be monitored. The agency might need to perform multiple scoping exercises that lead to the development of specific questions. The problem formulation activities to address these questions will ultimately result in investigation and analyses tailored to address issues related to low-dose endocrine effects. Box 5-1 provides some examples of targeted analyses the committee performed to answer questions that arose during the course of working on the case example of phthalates in Chapter 3.

CHARACTERIZING ADVERSITY

The committee discussed whether the effects of exposure to phthalate or PBDE exposure are adverse. The issue of adversity has been the subject of debate, and a number of definitions have been proposed (Kerlin et al. 2016). There have been recent attempts to develop clear criteria to evaluate whether an effect seen in nonclinical toxicology studies is adverse (Kerlin et al. 2016; Palazzi et al. 2016; Pandiri et al. 2017). The committee considered that guidance when drafting its definition of adverse.¹

A determination of whether an effect is adverse requires expert judgment and should be based on evaluation of the effect in both the animal and human literature. For example, in the case of phthalate-induced effects, hypospadias was considered adverse because it represents a morphologic effect that

___________________

¹ Adverse effect: a biological change in an organism that results in an impairment of functional capacity, a decrease in the capacity to compensate for stress, or an increase in susceptibility to other influences (adapted from IPCS 2004).

might affect reproductive performance or behavior (Bubanj et al. 2004; Schlomer et al. 2014). Whether changes in either fetal testosterone concentrations or anogenital distance (AGD) are adverse has been the subject of discussion in the scientific community (Howdeshell et al. 2017). Several mechanisms, including androgen receptor antagonism and inhibition of androgen synthesis enzymes, can contribute to phthalate-induced reductions in fetal testosterone production (Howdeshell et al. 2017). In animal models of phthalate toxicity, multiple studies have found no effect on apical reproductive end points when treatment-related reduction in fetal testosterone was less than 40%, suggesting a point below which apical effects in studies in rodents are not observed (Gray et al. 2016). Likewise, a reduction in AGD is a biomarker of reduced in utero androgen concentrations (Thankamony et al. 2016) and might exhibit a similar threshold before changes in apical reproductive end points are seen. Several studies have reported that newborns born with hypospadias and cryptorchidism have shorter AGDs than infants without abnormalities (Hsieh et al. 2012; Jain and Singal 2013; Thankamony et al. 2014). In addition, several studies in adult males have reported that men who have reduced fertility—including lower sperm concentration, count, and motility—have shortened AGDs (Eisenberg et al. 2011, 2012; Mendiola et al. 2011, 2015; Eisenberg and Lipshultz 2015). At this time, the degree of AGD shortening necessary to observe apical end points remains unknown. It is therefore unclear that changes in testosterone or AGD in the absence of other apical reproductive end points would meet the first part of the committee’s definition of an adverse effect—“a biological change in an organism that results in an impairment of functional capacity.”

The committee also considered whether phthalate-induced changes in fetal testosterone or AGD might increase the susceptibility of an organism to other influences. Sufficient androgen activity is required for proper male reproductive-tract development, and this activity is a consequence of multiple inputs at the molecular level. In the rat model, data have shown that a phthalate-induced reduction in fetal testosterone made the animal more susceptible to the effects of another chemical (linuron) that also targeted the androgen signaling system, although the reduction by itself was insufficient to produce an adverse reproductive effect (Hotchkiss et al. 2004). On the basis of those data, a phthalate-related reduction in fetal testosterone or reduction in AGD would be considered an adverse effect because they meet another part of the definition: a biological change that results in “an increase in susceptibility to other influences.”

The committee considered the PBDE-related effects on cognitive function to be adverse. The committee’s systematic review supports the findings made by Lam et al. (in press) of an inverse association between developmental PBDE exposure and effects on IQ in children. Lam et al. reported a decrease of 3.7 IQ points per 10-fold increase in serum PBDE concentration. It is important to differentiate between effects seen at the individual and population levels (NRC 2009). First, the magnitude of response to PBDEs and other EACs can vary within a population because some individuals might be more affected and others less so by the same exposure (NRC 2009). At the individual level, large changes in IQ (e.g., 10 points) are considered adverse. Smaller changes on an individual level, such as a shift of 3-4 points, might not be associated with a functional impairment in cognitive function and are often within the range of variability when an individual is retested on an IQ test (Watkins and Smith 2013). However, given widespread exposure across the population, small changes in IQ could shift the entire population distribution in the direction of decreased function, which has quantifiable consequences integrated over the population, such as increased fraction of population with very low IQ or reduced aggregate economic output (Axelrad et al. 2007; Bellinger 2012). Therefore, PBDE-related reduction in IQ would be considered an adverse effect because they meet the first part of the definition: “a biological change in an organism that results in an impairment of functional capacity.”

The committee’s reasoning as to whether the changes produced by phthalate or PBDE exposure are adverse is likely to be applicable to other chemicals and end points. Some lessons learned from the committee’s deliberation include the following:

• There is usually agreement regarding adversity of more severe outcomes because they fit into the first part of the definition concerning “impairment of functional capacity.”

• Changes in continuous end points, such as hormone levels or biomarkers, might not lead to demonstrable “impairment of functional capacity” at the individual level. However, such effects might fit into the second or third parts of the definition of adversity relating to “a decrease in the capacity to compensate for stress” or “an increase in susceptibility to other influences.” Decisions to label such changes as “adverse” would be strengthened by experimental data that demonstrate reduced compensatory capacity or increased susceptibility. In the absence of such data, scientific judgment (e.g., analogy to other end points) and policy considerations (e.g., the severity and magnitude of the possible effect) might be involved.
• The small magnitude of change of a continuous end point might not have demonstrable “impairment of functional capacity” at the individual level but could have quantifiable consequences on functional capacity when considered over the entire population. Such shifts in the population distribution might result in more individuals in the tail of the distribution or reduced aggregate function over the population. Therefore, when evaluating end points with respect to the first part of the definition, consideration needs to be given to effects over the population.

REFLECTIONS AND LESSONS LEARNED FROM THE SYSTEMATIC REVIEWS

Selection of Example Chemicals

The specification that the systematic reviews be performed on chemicals that “act through an endocrine-mediated pathway” and “that affect the estrogen, androgen, or perhaps other endocrine systems” led to extensive discussions about how stringent the committee should be about using mechanistic data to guide chemical selection. The committee’s goal was to identify candidate chemicals that were presumed to be associated with endocrine effects. Although that approach seems straightforward, mechanisms are poorly understood for some of the candidate EACs that were initially considered. The committee eventually selected two chemical classes (phthalates and PBDEs) that had varying amounts of mechanistic information to illustrate different approaches that EPA might need to use when assessing low-dose endocrine effects. Other factors that influenced chemical selection were whether relevant animal and human data were available and the types of end points associated with each candidate chemical.

The committee also recognized that it might be advantageous for EPA to build on existing systematic reviews that are published in the peer-reviewed literature. During the course of its discussions, the committee became aware of a systematic review being conducted by Lam et al. (in press) on developmental exposures to PBDEs and human neurodevelopment. The willingness of Lam and coworkers to share an early draft of their study provided the committee with a unique opportunity for the committee to meet its objective of demonstrating how to build on a published systematic review. The committee remains indebted to Lam and coworkers for their generosity.

Methods

The committee initially performed a series of scoping exercises—conducting informal literature reviews and hosting a workshop—that helped define chemicals of interest, populations of concern, exposure windows, health end points, and other factors that led to the development of the specific research question and the appropriate PECO (Population, Exposure, Comparator, and Outcome) statements that guided the committee’s investigation (Higgins and Green 2011; IOM 2011). Broadly stated, the phthalate research question asked whether in utero exposure to phthalates was associated with reproductive effects (as assessed by changes in fetal testosterone during gestation or at delivery, AGD, or the incidence of hypospadias) in male nonhuman mammals or humans. The PBDE research question asked whether developmental exposure to PBDEs was associated with neurobehavioral effects (as assessed by changes in learning, memory, attention, or response inhibition) in nonhuman mammals or humans. The systematic reviews performed by the committee were therefore hypothesis driven and designed to answer a set of focused questions. Although the project was intended to address issues surrounding low-dose effects, the

committee did not constrain exposure to low dose in the PECO statement and therefore did not use it as an eligibility criterion. That decision was a deliberate choice because the committee did not want to a priori constrain the investigation by any preconceived notion of low dose. Instead, the committee chose to address whether effects occurred at a low dose as a separate subsequent step in the process.

The committee discussed at length whether it could provide EPA with advice about when a systematic review should be performed but decided it could not be more specific because that decision will depend on the availability of data and resources, the anticipated actions, the time frame for decision making, and other factors. As the committee can attest, one disadvantage in conducting a systematic review is that it can be time and resource intensive, particularly for individuals that have not previously conducted a systematic review. Some steps are inherently resource consuming. For example, two individuals independently perform most steps in the systematic-review process, such as data abstraction and risk of bias determinations. The committee therefore recognized that there was a need for the inclusion of investigation methods that do not rely on systematic-review methods. One form of accelerated evidence synthesis that has been suggested by practitioners of systematic review is a rapid review, in which components of the systematic-review process are simplified or omitted (e.g., the need for two independent reviewers) to produce information in a more timely manner (Khangura et al. 2014; Polisena et al. 2015; Tricco et al. 2015). The committee recognizes the need to streamline the process; however, evidence suggests that rapid reviews should not be viewed as a substitute for a systematic review and that the time savings correlate with decreased methodologic quality or robustness (Harker and Kleijnen 2012; Featherstone et al. 2015). For some questions, systematic review methods are not expected to be time intensive (e.g., questions with small literature bases or data-poor chemicals). For other questions, the ability to streamline, increase efficiency, or automate portions of the review process might be considered by EPA. Efforts are currently under way to automate individual tasks in a systematic review, such as the literature search, study selection, and data extraction (Tsafnat et al. 2014; Jonnalagadda et al. 2015). The committee recognizes that the methods and role of systematic review and meta-analysis in toxicology are evolving rapidly and EPA will need to stay abreast of these developments, strive for transparency, and use appropriate methods to address its questions.

The committee used the National Toxicology Program’s Office of Health Assessment and Translation (OHAT) method (Rooney et al. 2014; NTP 2015) to evaluate the confidence in the evidence. Overall, the committee found the OHAT method was relatively easy to implement but identified a few challenges. For example, a body of evidence from experimental animal studies was given an initial confidence rating of “high” on the basis of the study-design issues that are inherent in experimental studies. That approach is consistent with the GRADE (Grading of Recommendations, Assessment, Development, and Evaluation) approach for considering randomized controlled trials. Multiple factors were then considered to determine whether to downgrade (e.g., risk of bias concern or unexplained inconsistency in the results) or upgrade (e.g., large magnitude of effect or evidence of a dose-response relationship) confidence in the body of evidence. In practice, the committee found that bodies of evidence had multiple risk of bias issues and found it challenging to determine whether to downgrade confidence by one or more levels on the basis of the problems with study design or conduct observed. The method would benefit from clearer guidelines or examples to illustrate the consideration of factors for downgrading or upgrading confidence.

Several other lessons became apparent during the course of the study. They include the need for improved reporting of study design, conduct, and results on the part of scientists to facilitate risk of bias evaluations; approaches for displaying behavioral and categorical data within a systematic review; and inclusion of subject-matter expertise on the review team, including expertise in meta-analysis and other statistical approaches. It is often not clear if a study used best practices in the study design and conduct but failed to report those good practices, or if there were issues in study conduct that would result in potential bias. Researchers should be encouraged to follow reporting guidelines (e.g., the ARRIVE guidelines for reporting of animal studies and the STROBE statement for reporting of observational studies in epidemiology). They should also use methods to minimize bias in research conduct particularly for issues where there is empirical evidence that risk of bias practices can affect the effect size (e.g., the key issues of randomizing animals to treatment or and blinding of outcome assessors to study groups).

The committee also found that quantitative data analysis methods can be valuable in evaluating a body of evidence. For phthalates, the use of common end-point measures across most studies enabled extensive use of meta-analysis techniques, the results of which contributed to evaluations of the precision, heterogeneity, magnitude of effect, and presence of a dose-response gradient across studies. Use of meta-analysis techniques is much more robust than relying on individual study results, because they account for potential heterogeneity across studies and the statistical power of each individual study. For PBDEs, the variety of test methods and end points limited the use meta-analysis, although the committee did demonstrate its use for one of the more commonly reported end points. In this case, the committee’s preferred option would have been to conduct benchmark dose modeling on individual studies; however, many studies did not report sufficient details on the study design (e.g., group sizes) or results (e.g., standard deviations) to estimate benchmark doses. Thus, the committee had to rely on reported lowest-observed-adverse-effect levels (LOAELs) and no-observed-adverse-effect levels (NOAELs) based on pair-wise statistical significance in evaluating studies. The use of LOAELs and NOAELs is less than ideal because they depend highly on individual study-design characteristics; therefore, apparent differences among studies might be explained by design differences, such as sample size or dose spacing, rather than true inconsistency. Indeed, the committee found that some of its concerns about consistency and heterogeneity in the PBDE studies were ameliorated by the results of the meta-analysis performed on the Morris water maze data.

As discussed above and elsewhere in the report, the committee used existing methods and software for performing the systematic reviews, the meta-analyses, and the evidence integration. The selected methods and software were primarily chosen because of the committee’s experience with using them and should not be taken as an indication of an endorsement of them or that they are the recommended approaches for EPA to use. The committee recognizes that other software, statistical packages, and evidence integration approaches could fit EPA’s needs.

LESSONS LEARNED FROM EVIDENCE INTEGRATION

In keeping with the statement of task, the committee demonstrated how human and animal data streams can be integrated and used to determine whether a likely causal association is supported by the evidence. Determinations regarding causality were constrained by the PECO statements that were described earlier. The committee synthesized the animal and human evidence and reached hazard conclusions using an OHAT framework (NTP 2015). Earlier chapters describe the data integration steps in more detail. Using the OHAT framework and language used to describe the strength of the evidence, the committee reached a number of conclusions regarding the endocrine toxicity of phthalates and PBDEs. Two example causality statements are provided below.

• Diethylhexyl phthalate (DEHP) is presumed to be a reproductive hazard to humans, and there is moderate evidence that decreased AGD in humans occurs following low-dose exposure to this phthalate.
• Developmental exposure to BDE-47 is presumed to pose a hazard to intelligence in humans, and there is a moderate level of evidence that effects on IQ occur following low-dose exposure to this congener.

The Use of Mechanistic Data for Evidence Integration

The committee considered mechanistic data, which included pharmacokinetic information, in reaching its final hazard identification conclusions for each end point. For example, the committee used mechanistic evidence to support the finding that DEHP effects on AGD in humans were biologically plausible. The OHAT hazard identification scheme also allows consideration of mechanistic data to upgrade or downgrade the initial hazard determination. The committee found that the guidance in the OHAT handbook on the level of evidence needed to upgrade or downgrade the initial hazard determination on the ba-

sis of mechanistic data was somewhat lacking, and there are few OHAT monographs or published systematic reviews that have used the approach. The method would benefit from additional clarity in the guidelines or particularly from examples to illustrate mechanistic evidence that would provide strong support for biological plausibility of the observed effect sufficient to justify upgrading or downgrading the hazard conclusion.

Mechanistic data were also used to evaluate whether there was dose-response concordance between humans and animals. In the case of DEHP effects on AGD, the committee noted significant species differences in phthalate metabolism and clearance, and effects on fetal testosterone (Ito et al 2005; Gaido et al. 2007; Johnson et al. 2012). In addition, qualitative and quantitative differences in phthalate pharmacokinetics also occur between rodents, nonhuman primates, and humans (Kessler et al. 2004; McKinnell et al. 2009; Kurata et al. 2012). The committee found comparing evidence on dose-response relationships between animal and human studies to be challenging and imprecise because animal studies often measure external administered doses (usually without measures of internal dose), whereas human studies measure biomarkers of internal dose (with estimates of the external administered dose being uncertain). Toxicology studies that measure internal dose metrics, including metrics that are similar to those used in human biomonitoring and those most relevant to the target tissue dose, could help address those challenges.

Broader mechanistic questions concerning how EACs might alter normal hormone function at low doses have been raised in the scientific literature (Skakkebaek et al. 2011; Vandenberg 2014; Maqbool et al. 2016). Although those questions are potentially important for risk assessment of EACs, they were deemed beyond the charge of the committee.

End-Point Concordance

The methods used to assess fetal testosterone, AGD, and hypospadias were qualitatively similar between animal and human studies, and this simplified the analysis of whether the responses seen were concordant or discordant. In addition, end-point consistency between human and animal studies provided additional confidence in biological plausibility for hazard identification. Moreover, because similar end points were evaluated in multiple studies, meta-analyses were possible for several end points of interest.

In the case of PBDE effects on cognitive function, change in IQ was a primary measure in children, and a wide array of neurobehavioral assays were used in the animal studies. Although the assays evaluate cognitive function in animals (e.g., tests of learning and memory), the types of tests used, the timing of evaluation during postnatal development, and other methodologic differences restricted the ability of the committee to synthesize the data using meta-analysis or other methods.

OTHER ISSUES

A related element of the statement of task asked the committee to consider adverse outcome pathways (AOPs) and high-throughput data in its analysis. The committee found that using AOPs and high-throughput data was difficult because molecular initiating events involved in the phthalate reproductive effects remain unclear, and steroidogenic assays that are used in current high-throughput assay systems (e.g., ToxCast) often rely on human adrenal cell lines of unknown mechanistic relevance for phthalate toxicity. In the case of the PBDEs, several potential mechanisms have been proposed (Costa et al. 2014); however, none have been conclusively linked to the neurobehavioral outcomes evaluated by the committee. Although high-throughput data were not helpful in the committee’s analyses of phthalates and PBDE, such data could be used for priority setting and other uses. For example, chemicals that exhibit endocrine activity in a high-throughput assay might be given a higher priority for future testing. That approach is consistent with the EPA’s Endocrine Disruptor Screening Program. High-throughput data might be used to differentiate chemicals on the basis of bioactivity potency to set priorities for chemical testing. High-throughput data might also be used to support read-across methods for chemicals that have few human or animal data. In addition, the use of reverse toxicokinetic methods to support in vitro–to–in vivo extrapolations that can be used to convert an in vitro concentration into an estimated serum concentration will be

important (Wetmore et al. 2012). The committee anticipates that methods will emerge to support the use of in vitro data streams in systematic reviews.

The committee also considered whether animal toxicity studies could predict the low-dose effects seen with phthalates or PBDEs in people. The committee found that, although animal data could help identify hazards associated with phthalates and PBDE exposure, they were unable to predict exposures at which effects occurred in people. Indeed, differences in exposure between animal studies and those observed in the general human population spanned several orders of magnitude. Strategies to help resolve apparent discrepancies often rest on an improved understanding of health effects seen in people, revisions to animal-testing guidelines that help improve their predictive value, consideration of animal studies that include additional end points, and efforts to use mechanistic and pharmacokinetic data to bridge these seemingly disparate data streams (NRC 2007, 2009). The recent National Academies report Using 21st Century Science to Improve Risk-Related Evaluations (NASEM 2017) emphasized the need to align environmental and test-system exposures and develop the models and methods necessary to do so. Thus, new test methods, models, and approaches will need to evolve to address the apparent discrepancies.

REFERENCES

Howdeshell, K.L., A.K. Hotchkiss, and L.E. Gray, Jr. 2017. Cumulative effects of antiandrogenic chemical mixtures and their relevance to human health risk assessment. Int. J. Hyg. Environ. Health 220(2 Pt. A):179-188.

Hsieh, M.H., M.L. Eisenberg, A.B. Hittelman, J.M. Wilson, G.E. Tasian, and L.S. Baskin. 2012. Caucasian male infants and boys with hypospadias exhibit reduced anogenital distance. Hum. Reprod. 27(6):1577-1580.

IOM (Institute of Medicine). 2011. Finding What Works in Health Care: Standards for Systematic Review. Washington, DC: The National Academies Press.

IPCS (International Programme on Chemical Safety). 2004. IPCS Risk Assessment Terminology. Part 1: IPCS/OECD Key Generic Terms Used in Chemical Hazard/Risk Assessment. Part 2: IPCS Glossary of Key Exposure Assessment Terminology. Geneva: World Health Organization [online]. Available: http://www.inchem.org/documents/harmproj/harmproj/harmproj1.pdf [accessed November 11, 2015].

Ito, Y., H. Yokota, R. Wang, O. Yamanoshita, G. Ichihara, H. Wang, Y. Kurata, K. Takagi, and T. Nakajima. 2005. Species differences in the metabolism of di(2-ethylhexyl) phthalate (DEHP) in several organs of mice, rats, and marmosets. Arch. Toxicol. 79(3):147-154.

Jain, V.G., and A.K. Singal. 2013. Shorter anogenital distance correlates with undescended testis: A detailed genital anthropometric analysis in human newborns. Hum. Reprod. 28(9):2343-2349.

Johnson, K.J., N.E. Heger, and K. Boekelheide. 2012. Of mice and men (and rats): Phthalate-induced fetal testis endocrine disruption is species-dependent. Toxicol. Sci. 129(2):235-248.

Jonnalagadda, S.R., P. Goyal, and M.D. Huffman. 2015. Automating data extraction in systematic reviews: A systematic review. Syst. Rev. 4:78.

Kerlin, R., B. Bolon, J. Burkhardt, S. Francke, P. Greaves, V. Meador, and J. Popp. 2016. Scientific and Regulatory Policy Committee: Recommended (“best”) practices for determining, communicating, and using adverse effect data from nonclinical studies. Toxicol. Pathol. 44(2):147-162.

Kessler, W., W. Numtip, K. Grote, G.A. Csanády, I. Chahoud, and J.G. Filser. 2004. Blood burden of di(2ethylhexyl) phthalate and its primary metabolite mono(2-ethylhexyl) phthalate in pregnant and nonpregnant rats and marmosets. Toxicol. Appl. Pharmacol. 195(2):142-153.

Khangura, S., J. Polisena, T.J. Clifford, K. Farrah, and C. Kamel. 2014. Rapid review: An emerging approach to evidence synthesis in health technology assessment. Int. J. Technol. Assess. Health Care 30(1):20-27.

Kurata, Y., F. Makinodan, N. Shimamura, and M. Katoh. 2012. Metabolism of di (2-ethylhexyl) phthalate (DEHP): Comparative study in juvenile and fetal marmosets and rats. J. Toxicol. Sci. 37(1):33-49.

Lagarde, F., C. Beausoleil, S.M. Belcher, L.P. Belzunces, C. Emond, M. Guerbet, and C. Rousselle. 2015. Nonmonotonic dose-response relationships and endocrine disruptors: A qualitative method of assessment. Environ. Health 14(1):13.

Lam, J., B.P. Lanphear, D. Bellinger, D.A. Axelrad, J. McPartland, P. Sutton, L. Davidson, N. Daniels, S. Sen, and T.J. Woodruff. In press. Developmental PBDE exposure and IQ/ADHD in childhood: a systematic review and meta-analysis. Environ. Health Perspect.

Maqbool, F., S. Mostafalou, H. Bahadar, and M. Abdollahi. 2016. Review of endocrine disorders associated with environmental toxicants and possible involved mechanisms. Life Sci. 145:265-273.

McKinnell, C., R.T. Mitchell, M. Walker, K. Morris, C.J.H. Kelnar, W.H. Wallace, and R.M. Sharpe. 2009. Effect of fetal or neonatal exposure to monobutyl phthalate (MBP) on testicular development and function in the marmoset. Hum. Reprod. 24(9):2244-2254.

Mendiola, J., R.W. Stahlhut, N. Jorgensen, F. Liu, and S.H. Swan. 2011. Shorter anogenital distance predicts poorer semen quality in young men in Rochester, New York. Environ. Health Perspect. 119(7):958-963.

Mendiola, J., M. Melgarejo, M. Monino-Garcia, A. Cutillas-Tolin, J.A. Noguera-Velasco, and A.M. Torres-Cantero. 2015. Is anogenital distance associated with semen quality in male partners of subfertile couples? Andrology 3(4):672-676.

NASEM (National Academies of Sciences, Engineering, and Medicine). 2017. Using 21st Century Science to Improve Risk-Related Evaluations. Washington, DC: The National Academies Press.

NRC (National Research Council). 2007. Toxicity Testing in the 21st Century: A Vision and a Strategy. Washington, DC: The National Academies Press.

NRC. 2009. Science and Decisions: Advancing Risk Assessment. Washington, DC: The National Academies Press.

NRC. 2014a. Review of the Environmental Protection Agency’s State-of-the Science Evaluation of Nonmontonic Dose-Response Relationships as They Apply to Endocrine Disruptors. Washington, DC: The National Academies Press.

NRC. 2014b. Review of EPA’s Integrated Risk Information System (IRIS) Process. Washington, DC: The National Academies Press.

NTP (National Toxicology Program). 2015. Handbook for Conducting a Literature-based Health Assessment Using OHAT Approach for Systematic Review and Evidence Integration. Office of Health Assessment and Translation, Division of the National Toxicology Program, National Institute of Environmental Health Sciences. January 9, 2015 [online]. Available: http://ntp.niehs.nih.gov/ntp/ohat/pubs/handbookjan2015_508.pdf [accessed September 21, 2015].

Palazzi, X., J.E. Burkhardt, H. Caplain, V. Dellarco, P. Fant, J.R. Foster, S. Francke, P. Germann, S. Gröters, T. Harada, J. Harleman, K. Inui, W. Kaufmann, B. Lenz, H. Nagai, G. Pohlmeyer-Esch, A. Schulte, M. Skydsgaard, L. Tomlinson, C.E. Wood, and M. Yoshida. 2016. Characterizing “adversity” of pathology findings in nonclinical toxicity studies: Results from the 4th ESTP International Expert Workshop. Toxicol. Pathol. 44(6):810-824.

Pandiri, A.R., R.L. Kerlin, P.C. Mann, N.E. Everds, A.K. Sharma, L.P. Myers, and T.J. Steinbach. 2017. Is it adverse, nonadverse, adaptive, or artifact? Toxicol. Pathol. 45(1):238-247.

Polisena, J., C. Garritty, C.A. Umscheid, C. Kamel, K. Samra, J. Smith, and A. Vosilla. 2015. Rapid Review Summit: An overview and initiation of a research agenda. Syst. Rev. 4:137.

Rooney, A.A., A.L. Boyles, M.S. Wolfe, J.R. Bucher, and K.A. Thayer. 2014. Systematic review and evidence integration for literature-based environmental health science assessments. Environ. Health Perspect. 122(7):711-718.

Schlomer, B., B. Breyer, H. Copp, L. Baskin, and M. DiSandro. 2014. Do adult men with untreated hypospadias have adverse outcomes? A pilot study using a social media advertised survey. J. Pediatr. Urol. 10(4):672-679.

Skakkebaek, N.E., J. Toppari, O. Söder, C.M. Gordon, S. Divall, and M. Draznin. 2011. The exposure of fetuses and children to endocrine disrupting chemicals: A European Society for Paediatric Endocrinology (ESPE) and Pediatric Endocrine Society (PES) call to action statement. J. Clin. Endocrinol. Metab. 96(10):3056-3058.

Thankamony, A., N. Lek, D. Carroll, M. Williams, D.B. Dunger, C.L. Acerini, K.K. Ong, and I.A. Hughes. 2014. Anogenital distance and penile length in infants with hypospadias or cryptorchidism: Comparison with normative data. Environ. Health Perspect. 122(2):207-211.

Thankamony, A., V. Pasterski, K.K. Ong, C.L. Acerini, and I.A. Hughes. 2016. Anogenital distance as a marker of androgen exposure in humans. Andrology 4(4):616-625.

Tricco, A.C., J. Antony, W. Zarin, L. Strifler, M. Ghassemi, J. Ivory, L. Perrier, B. Hutton, D. Moher, and S.E. Straus. 2015. A scoping review of rapid review methods. BMC Med. 13:224.

Tsafnat, G., P. Glasziou, M.K. Choong, A. Dunn, F. Galgani, and E. Coiera. 2014. Systematic review automation technologies. Syst. Rev. 3:74.

Vandenberg, L.N. 2014. Low-dose effects of hormones and endocrine disruptors. Vitam. Horm. 94:129-165.

Watkins, M.W., and L.G. Smith. 2013. Long-term stability of the Wechsler Intelligence Scale for Children–Fourth Edition. Psychol. Assess. 25(2):477-483.

Wetmore, B.A., J.F. Wambaugh, S.S. Ferguson, M.A. Sochaski, D.M. Rotroff, K. Freeman, H.J. Clewell III, D.J. Dix, M.E. Andersen, K.A. Houck, B. Allen, R.S. Judson, R. Singh, R.J. Kavlock, A.M. Richard, and R.S. Thomas. 2012. Integration of dosimetry, exposure, and high-throughput screening data in chemical toxicity assessment. Toxicol. Sci. 125(1):157-174.