Skip to main content

Currently Skimming:

4 Study Evaluation
Pages 52-63

The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.


From page 52...
... , study evaluation in the handbook is intentionally broadened to include the additional study evaluation domains of "sensitivity" and "reporting quality." It is not standard practice to include the concepts of sensitivity or reporting quality as part of the evaluation of individual studies included in systematic reviews of human research, although these domains are sometimes part of study evaluation for systematic reviews of animal studies (SYRCLE [Hooijmans et al., 2018)
From page 53...
... The 2014 report noted that EPA could augment risk of bias considerations with additional "quality assessment items" that are relevant to a particular systematic review if they are empirically based, as may be the case in the handbook with consideration of sensitivity and reporting quality. The handbook's use of study evaluation ratings to exclude studies from further consideration is only partially concordant with the 2014 report, and recommended practice for using study evaluation to exclude studies prior to data extraction has evolved since the 2014 report.
From page 54...
... The first area relates to the description of how studies evaluated to have certain confidence ratings will be used in future steps of the IRIS assessment. The second, more substantive, area relates to a lack of clear delineations among the separate concepts of risk of bias, reporting quality, study sensitivity, and other standard concepts in systematic review.
From page 55...
... Standard procedures for systematic review, which have a longer history in human clinical studies than in animal toxicological studies, evaluate studies on the basis of risk of bias and internal validity (Higgins and Thomas, 2019)
From page 56...
... Low sensitivity in both the animal toxicological and human epidemiological studies is generally described as "bias toward the null," which might arise due to certain types of exposure measurement error, and also in terms that relate directly to study precision, such as sample size, which would represent an issue distinct from bias. The committee viewed this notion of sensitivity as a less established construct than others used in systematic review, noting that it actually touches on several more established concepts, including (but not limited to)
From page 57...
... The systematic review infrastructure offers the opportunity to consider such studies in relation to the broad and refined PECO statements when judging relevance for the risk assessment, rather than use their features to determine sensitivity or study quality. For evaluation of human epidemiological studies, the handbook briefly notes the concept of defining "an ‘ideal' design (i.e., a study design with no risk of bias and high sensitivity)
From page 58...
... Finally, some aspects of sensitivity discussed in the animal toxicological studies were viewed as overlapping with notions of reporting quality, possibly due to their inherently subjective nature. For example, considerations described as "outcome measures and results display" within the sensitivity domain include prompting questions such as "Does the level of detail allow for an informed interpretation of the results" and listed general considerations include "use of unreliable methods," which would both point toward considerations that could be considered in the reporting quality evaluation.
From page 59...
... Should EPA maintain study evaluation concepts that are currently described as sensitivity considerations and cannot be captured with more established terminology for systematic review, the use of a word other than "sensitivity" to describe these necessary aspects would avoid confusion with several other technical and colloquial meanings of the word "sensitivity." Reporting Quality The final quality assessment item considered in Chapter 6 of the handbook is reporting quality, which receives strikingly different discussion between the human epidemiological and animal toxicological studies. Inclusion of reporting quality as a quality assessment item is atypical in systematic review protocols but, as with sensitivity, has more recently emerged in systematic reviews of animal toxicological studies (SYRCLE [Hooijmans et al., 2018 and NTP, 2019)
From page 60...
... A transparent and repeatable review process requires either explicit guidance for how to incorporate reporting quality into individual risk of bias domains or an explicit rationale for two separate processes for assessing reporting quality in human and animal study evaluation. In either case, explicit rationale for isolating elements of reporting quality from established systematic review concepts would aid operationalization of the study evaluation process.
From page 61...
... For example, see the 2021 National Academies report The Use of Systematic Review in EPA's Toxic Substances Control Act Risk Evaluations and the Cochrane Handbook for Systematic Reviews of Interventions. Recommendation 4.1: The handbook should not use the results of study evaluation as eligibility criteria for the systematic review.
From page 62...
... For animal studies, reporting quality is described in detail as a separate evaluation domain but with some aspects that overlap with other systematic review concepts such as risk of bias or external validity. The handbook notes that reviewers should reach out to study authors to obtain missing information in animal toxicological studies, but it provides no such reference to obtaining information from authors of human epidemiological studies.
From page 63...
... Study Evaluation Recommendation 4.5: EPA should monitor ongoing methodological development in assessing risk amid environmental co-exposures or mixtures in order to update explicit guidance on their potential roles in the evaluation of human epidemiological and animal toxicological studies.


This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More information on Chapter Skim is available.