Finding What Works in Health Care Standards for Systematic Reviews (2011) / Chapter Skim
Currently Skimming:
3 Standards for Finding and Assessing Individual Studies
3 Standards for Finding and Assessing Individual Studies
Pages 81-154
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 81... ...
The potential for bias to enter the selection process is significant and well documented. Without appropriate measures to counter the biased reporting of primary evidence from clinical trials and obser vational studies, SRs will reflect and possibly exacerbate existing distortions in the biomedical literature.
|
From page 82... ...
SRs often fail to acknowledge or address the risk of reporting biases, neglect to appraise the qual ity of individual studies included in the review, and are subject to errors during data extraction and the meta-analysis (Cooper et al., 2006; Delaney et al., 2007; Edwards et al., 2002; Golder et al., 2008; Gøtzsche et al., 2007; Horton et al., 2010; Jones et al., 2005; Lundh et al., 2009; Moher et al., 2007a; Roundtree et al., 2008; Tramer et al., 1997)
|
From page 83... ...
Moreover, crucial information from the studies is often difficult to locate because it is kept in researchers' files, government agency records, or manufacturers' proprietary records. The following overview further describes the context for the SR search process: the nature of the reporting bias in the biomedical literature; key sources of information on comparative effectiveness; and expert guidance on how to plan and conduct the search.
|
From page 84... ...
84 FINDING WHAT WORKS IN HEALTH CARE BOX 3-1 Recommended Standards for Finding and Assessing Individual Studies Standard 3.1 Conduct a comprehensive systematic search for evidence Required elements: 3.1.1 Work with a librarian or other information specialist trained in performing systematic reviews (SRs) to plan the search strategy 3.1.2 Design the search strategy to address each key research question 3.1.3 Use an independent librarian or other information special ist to peer review the search strategy 3.1.4 Search bibliographic databases 3.1.5 Search citation indexes 3.1.6 Search literature cited by eligible studies 3.1.7 Update the search at intervals appropriate to the pace of generation of new information for the research question being addressed 3.1.8 Search subject-specific databases if other databases are unlikely to provide all relevant evidence 3.1.9 Search regional bibliographic databases if other data bases are unlikely to provide all relevant evidence Standard 3.2 Take action to address potentially biased reporting of research results Required elements: 3.2.1 Search grey-literature databases, clinical trial registries, and other sources of unpublished information about studies 3.2.2 Invite researchers to clarify information related to study eligibility, study characteristics, and risk of bias 3.2.3 Invite all study sponsors to submit unpublished data, in cluding unreported outcomes, for possible inclusion in the systematic review 3.2.4 Handsearch selected journals and conference abstracts 3.2.5 Conduct a web search 3.2.6 Search for studies reported in languages other than En glish if appropriate Standard 3.3 Screen and select studies Required elements: 3.3.1 Include or exclude studies based on the protocol's pre specified criteria
|
From page 85... ...
screen titles and abstracts of all articles and then read the full text of articles identified in initial screening. 3.3.6 Taking account of the risk of bias, consider using obser vational studies to address gaps in the evidence from ran domized clinical trials on the benefits of interventions Standard 3.4 Document the search Required elements: 3.4.1 Provide a line-by-line description of the search strategy, including the date of every search for each database, web browser, etc.
|
From page 86... ...
. Ensuring an Accurate Search An analysis of SRs published in the Cochrane Database of Systematic Reviews found that 90.5 percent of the MEDLINE searches contained at least one search error (Sampson and McGowan, 2006)
|
From page 87... ...
recently surveyed individuals experienced in SR searching and identified aspects of the search process that experts agree are likely to have a large impact on the sensitivity and precision of a search: accurate translation of each research question into search concepts; correct choice of Boolean and proximity operators; absence of spelling errors; correct line numbers and combination of line numbers; accurate adaptation of the search strategy for each database; and inclusion of relevant subject headings. Then they developed practice guidelines for peer review of electronic search strategies.
|
From page 88... ...
Table 3-1 provides definitions of the types of reporting biases. Publication Bias The term publication bias refers to the likelihood that publication of research findings depends on the nature and direction of TABLE 3-1 Types of Reporting Biases Type of Reporting Bias Definition Publication bias The publication or nonpublication of research findings, depending on the nature and direction of the results Selective outcome The selective reporting of some outcomes but not reporting bias others, depending on the nature and direction of the results Time-lag bias The rapid or delayed publication of research findings, depending on the nature and direction of the results Location bias The publication of research findings in journals with different ease of access or levels of indexing in standard databases, depending on the nature and direction of results.
|
From page 89... ...
. Selective Outcome Reporting Bias To avert problems introduced by post hoc selection of study outcomes, a randomized controlled trial's (RCT's)
|
From page 90... ...
placebo to a nonsignificant difference and the nonsignificant difference between reboxetine and SSRIs to inferiority for reboxetine. For patients with adverse events and rates of withdrawals from adverse events inclusion of unpublished data changed nonsignificant difference between reboxetine and placebo to inferiority of reboxetine; while for rates of withdrawals for adverse events inclusion of unpublished data changed the nonsignificant difference between reboxetine and fluoxetine to an inferiority of fluoxetine.
|
From page 91... ...
included at least one RCT with suspected outcome reporting bias. The authors assessed the poten tial impact of the bias and found that meta-analyses omitting trials with presumed selective outcome reporting for the primary outcome could overestimate the treatment effect.
|
From page 92... ...
When the articles are not identical, the systematic review team has difficulty 4 The Morrison study excluded complementary and alternative medicine interventions.
|
From page 93... ...
Because a selective meta-analysis or pooling does not reflect the true state of research evidence, it is prone to selection bias and may even reflect what the authors want us to know, rather than the totality of knowledge. Addressing Reporting Bias Reporting bias clearly presents a fundamental obstacle to the scientific integrity of SRs on the effectiveness of healthcare inter
|
From page 94... ...
is persuasive. Without appropriate measures to counter the biased reporting of primary evidence from clinical trials and observational studies, SRs may only reflect -- and could even exacerbate -- existing distortions in the biomedical literature.
|
From page 95... ...
The following summarizes the available evidence on the utility of key data sources -- such as bibliographic databases, grey literature, trial registries, and authors or sponsors of relevant research -- primarily for searching for results from RCTs. While considerable research has been done to date on finding relevant randomized trials (Dickersin et al., 1985; Dickersin et al., 1994; McKibbon et al., 2009; Royle and Milne, 2003; Royle and Waugh, 2003)
|
From page 96... ...
, with 15,000 abstracts of systematic reviews including more than 6,000 Cochrane reviews and protocols. DARE focuses on the effects of health interventions including diagnostic and prognostic studies, reha bilitation, screening, and treatment.
|
From page 97... ...
Bibliographic Databases Unfortunately, little empirical evidence is available to guide the development of an SR bibliographic search strategy. As a result, the researcher has to scrutinize a large volume of articles to iden tify the relatively small proportion that are relevant to the research question under consideration.
|
From page 98... ...
The primary purpose of C2-SPECTR is to provide support for the Campbell Collaboration systematic reviews (SRs) , but the registry is open to the public.
|
From page 99... ...
Contributors to the Cochrane Collaboration have handsearched literally thousands of journals and conference abstracts to identify controlled clinical trials and studies that may be eligible for Cochrane reviews (Dickersin et al., 2002a)
|
From page 100... ...
. Thus, the search for trial data should include trial registries (ClinicalTrials.gov, Clinical Study Results, Current Controlled Trials, and WHO International Clinical Trials Registry)
|
From page 101... ...
The act mandates that sponsors of any ongoing clinical trial involving a drug, biological product, or device approved for marketing by the FDA, not only register the trial,9 but also submit data on the trial's research protocol and study results (including adverse events) .10 As of October 2010, 2,300 results records are available.
|
From page 102... ...
conducted a study of all efficacy trials found in approved NDAs for new molecular entities from 2001 to 2002 and all published clinical trials corresponding to trials within those NDAs. The authors found that trials in NDAs with favorable primary outcomes were nearly five times more likely to be published than trials with unfavorable primary outcomes.
|
From page 103... ...
examined FDA submissions for 12 antidepressants, and identified 74 clinical trials, of which 31 percent had not been reported. The researchers compared FDA review data of each drug's effects with the published trial data.
|
From page 104... ...
. For example, in a study assessing outcome reporting bias in Cochrane SRs, Kirkham and colleagues (2010)
|
From page 105... ...
to update SR searches at the time of peer review.15 CRD and the Cochrane Collaboration recommend that the search be updated before the final analysis but do not specify an exact time period (CRD, 2009; Higgins et al., 2008)
|
From page 106... ...
The median survival of a review without any signal that an update was needed was 5.5 years. RECOMMENDED STANDARDS FOR THE SEARCH PROCESS The committee recommends the following standards and elements of performance for identifying the body of evidence for an SR: Standard 3.1 -- Conduct a comprehensive systematic search for evidence Required elements: 3.1.1 Work with a librarian or other information specialist trained in performing systematic reviews to plan the search strategy 3.1.2 Design the search strategy to address each key research question 3.1.3 Use an independent librarian or other information specialist to peer review the search strategy 3.1.4 Search bibliographic databases 3.1.5 Search citation indexes 3.1.6 Search literature cited by eligible studies 3.1.7 Update the search at intervals appropriate to the pace of generation of new information for the research question being addressed
|
From page 107... ...
Thus, the search should include sources of unpublished data, including grey-literature databases, trial regis tries, and FDA submissions such as NDAs. The search to identify a body of evidence on comparative effectiveness must be systematic, prespecified, and include an array of information sources that can provide both published and unpublished research data.
|
From page 108... ...
The review team should consider what information sources are appropriate given the topic of the review and review those sources. Conference abstracts and proceedings will rarely provide useful unpublished data but they may alert the reviewer to otherwise unpublished trials.
|
From page 109... ...
. The SR research protocol should make explicit which studies to include or exclude TABLE 3-3 Expert Suggestions for Screening Publications and Extracting Data from Eligible Studies AHRQ CRD Cochrane Use two or more members of the review team, √ √ √ working independently, to screen studies √ Train screeners Use two or more researchers, working √ √ independently, to extract data from each study Use standard data extraction forms developed √ √ √ for the specific systematic review √ √ √ Pilot-test the data extraction forms and process NOTE: See Appendix E for further details on guidance on screening and extracting data from AHRQ, CRD, and the Cochrane Collaboration.
|
From page 110... ...
examined 139 metaanalyses related to critical care medicine in journals or the Cochrane Database of Systematic Reviews. They found that a substantial proportion of the papers did not address potential biases in the selection of studies; 14 of the 36 Cochrane reviews (39 percent)
|
From page 111... ...
? Category Selection criteria Population Children ages 2–12 who are diagnosed with a ASD and children under age 2 at risk for diagnosis of a ASD Interventions Treatment modalities aimed at modifying the core symp toms of ASD Study settings Developed nations/regions including the United States, Canada, United Kingdom, Western Europe, Japan, Aus tralia, New Zealand, Israel, or South America Time period 1980–present Outcomes Short- and long-term outcomes, harms, and quality of life related to treatment for core symptoms • Controlled trials, prospective trials with historical con Study design trols, prospective or retrospective cohort studies, and medium to large case series.
|
From page 112... ...
The Cochrane Collaboration recommends that screeners be trained by pilot testing the eligibility criteria on a sample of studies and assessing reliability (Higgins and Deeks, 2008) , and certain Cochrane groups require that screeners take the Cochrane online training for handsearchers and pass a test on identification of clinical trials before they become involved (Cochrane Collaboration, 2010b)
|
From page 113... ...
Deciding to extend eligibility of study designs to observational studies represents a fundamental challenge because the suitability of observational studies for assessment of effectiveness depends heav ily on a number of clinical and contextual factors. The likelihood of selection bias, recall bias, and other biases are so high in certain clinical situations that no observational study could address the question with an acceptable risk of bias (Norris et al., 2010)
|
From page 114... ...
RECOMMENDED STANDARDS FOR SCREENING AND SELECTING STUDIES The committee recommends the following standards for screening and selecting studies for an SR: Standard 3.3 -- Screen and select studies Required elements: 3.3.1 Include or exclude studies based on the protocol's prespecified criteria 3.3.2 Use observational studies in addition to random ized clinical trials to evaluate harms of interventions 3.3.3 Use two or more members of the review team, working independently, to screen and select studies 16 See Chapter 5 for a complete review of SR reporting issues.
|
From page 115... ...
read all full-text articles identified in the search or (2) screen titles and abstracts of all articles and then read the full-text of articles identified in initial screening 3.3.6 Taking account of the risk of bias, consider us ing observational studies to address gaps in the evidence from randomized clinical trials on the benefits of interventions
|
From page 116... ...
The commit tee recommends the above standards and performance elements to address the pervasive problems of bias, errors, and inadequate documentation of the study selection process in SRs. While the evidence base for these standards is sparse, these common-sense standards draw from the expert guidance of AHRQ, CRD, and the Cochrane Collaboration.
|
From page 117... ...
Data extraction refers to the process that researchers use to collect and transcribe the data from each individual study. Which data are extracted depends on the research question, types of data that are available, and whether metaanalysis is appropriate.17 Box 3-6 lists the types of data that are often collected.
|
From page 118... ...
Date of data extraction 3. Identification features of the study: • Record number (to uniquely identify study)
|
From page 119... ...
: • Number of participants enrolled • Number of participants included in the analysis • Number of withdrawals and exclusions lost to follow-up • Summary outcome data, e.g., dichotomous (number of events, number of participants) , continuous (mean and standard deviation)
|
From page 120... ...
The only known effective means of reducing data extraction errors is to have at least two individuals independently extract data (Buscemi et al., 2006)
|
From page 121... ...
. Addressing Duplicate Publication Duplicate publication is another form of reporting bias with the potential to distort the findings of an SR.
|
From page 122... ...
For other types of data, one individual could extract the data while the second individual independently checks for ac curacy and completeness. Establish a fair procedure for resolving discrepancies; do not simply give final decision-making power to the senior reviewer 3.5.2 Link publications from the same study to avoid in cluding data from the same study more than once 3.5.3 Use standard data extraction forms developed for the specific systematic review 3.5.4 Pilot-test the data extraction forms and process Rationale Quality assurance (e.g., double data extraction)
|
From page 123... ...
For example, in considering selection bias, the reviewer would note whether the study uses random assignment of participants to treatments and concealment of allocation,18 because studies that employ these measures are less susceptible to selection bias than those that do not. The reviewer would also note whether there were baseline 18 Allocation concealment is a method used to prevent selection bias in clinical trials by concealing the allocation sequence from those assigning participants to interven tion groups.
|
From page 124... ...
• Incomplete outcome data Performance bias: Differences Maintain comparable • Blinding of participants, • Comparable intensity of services and in treatment or care given to conditions throughout study personnel, and outcome assessors cointerventions in the compared groups comparison groups during the period • Other measures as appropriate study affects observed results Detection bias: Differences among Use valid, reliable measures • Blinding of participants, • Measures to ensure equal and accurate groups in how outcomes are of outcome and assess them healthcare providers. and ascertainment of exposures and assessed is associated with outcome in the same manner for all outcome assessors outcomes across groups groups being compared Reporting biases: Differences Measure and report all • Selective outcome reporting between planned and reported preplanned outcomes • Selective analysis reporting results is associated with nature and direction of findings SOURCE: Higgins and Altman (2008)
|
From page 125... ...
By examining features of each study's design and conduct, systematic reviewers arrive at a judgment about the level of confidence one may place in each study, that is, the extent to which the study results can be believed. Assessing internal validity is concerned primarily (but not exclusively)
|
From page 126... ...
Some systematic review teams use an abbreviated form such as PICO or PICOS.
|
From page 127... ...
Large differences between de- Cardiovascular clinical trials used to inform Medicare Demographic characteristics: mographics of study popula- coverage enrolled patients who were significantly age, sex, race, and ethnicity tion and community patients younger (60.1 vs. 74.7 years)
|
From page 128... ...
. Incentives given to study participants Older versions of an interven- Only one of 23 trials comparing coronary artery bypass Specific product and features tion no longer in common use surgery with percutaneous coronary angioplasty used for rapidly changing technology the type of drug-eluting stent that is currently used in practice (Bravata et al., 2007)
|
From page 129... ...
. Outcomes Composite outcomes that Cardiovascular trials frequently use composite outcomes Effects of intervention on most mix outcomes of different that mix outcomes of varying importance to patients important benefits and harms, significance (Ferreira-Gonzalez et al., 2007)
|
From page 130... ...
TABLE 3-5 Continued 130 Condition That May Limit Feature That Should Be Characteristic Applicability Example Abstracted into Evidence Tables Setting Standards of care differ mark- Studies conducted in China and Russia examined the Geographic setting edly from setting of interest effectiveness of self-breast exams on reducing breast can cer mortality, but these countries do not routinely have concurrent mammogram screening as is available in the United States (Humphrey et al., 2002)
|
From page 131... ...
A similar tool for observational studies has yet to be developed and validated. As alternatives to "risk of bias," many systematic reviewers and organizations that develop practice guidelines use terms such as "study quality," "methodological quality," "study limitations," or "internal validity" to describe the critical appraisal of individual studies.
|
From page 132... ...
A similar tool for observational studies has yet to be developed and validated. Risk of Bias in Randomized Controlled Trials As a general rule, randomized trials, without question, have more protections against bias than observational studies and are less likely to produce biased or misleading results.
|
From page 133... ...
. The implication is that systematic review teams should always assess the details of each study's design to determine how potential biases associated with the study design may have influenced the observed results, because ignoring the possibility could be hazardous (Light and Pillemer, 1984)
|
From page 134... ...
The discrepancy was most dramatic when randomized trials were compared with observational studies of the same intervention (Chalmers, 1982; DerSimonian and Laird, 1986; Glass and Smith, 1979; Hoaglin et al., 1982; Miller et al., 1989; Wortman and Yeaton, 1983)
|
From page 135... ...
Thus, it is generally acknowledged that "adjustment" in the analysis cannot be viewed as a substitute for a study design that minimizes this bias. While selection bias is a widely recognized concern, observational studies are also particularly subject to detection bias, performance bias, and information biases.
|
From page 136... ...
with baseline information about the groups, and consider the magnitude and direction of bias, if any, in order to make a judgment about whether the goal of similar groups at inception of the study was met. Although the existence and consequences of these biases are widely acknowledged, tools to assess the risk of bias in observational studies of comparative effectiveness are poorly developed (Deeks et al., 2003)
|
From page 137... ...
Criteria for case-control studies, systematic reviews, and diagnostic accuracy stud ies are omitted. SOURCE: Harris et al.
|
From page 138... ...
Cochrane Database of Systematic Reviews 2005(1)
|
From page 139... ...
2007. Creating and synthesizing evidence with decision makers in mind: Integrating evidence from clinical trials and other study designs.
|
From page 140... ...
2009. Not only randomized controlled trials, but also case series should be considered in systematic reviews of rapidly developing technologies.
|
From page 141... ...
2009. Systematic reviews: CRD's guidance for undertaking reviews in health care.
|
From page 142... ...
2007. The quality of reports of critical care meta-analyses in the Cochrane Database of Systematic Reviews: An independent appraisal.
|
From page 143... ...
2002. Iden tification of randomized controlled trials in systematic reviews: Accuracy and reliability of screening records.
|
From page 144... ...
2009. A framework for updating systematic reviews.
|
From page 145... ...
In Cochrane handbook for systematic reviews of interventions, edited by J
|
From page 146... ...
2005. High prevalence but low impact of data extraction and reporting errors were found in Cochrane systematic reviews.
|
From page 147... ...
2010. The impact of outcome reporting bias in randomised con trolled trials on a cohort of systematic reviews.
|
From page 148... ...
2003. The inclusion of reports of randomised trials published in languages other than English in systematic reviews.
|
From page 149... ...
2005. Assess ment of methodological quality of primary studies by systematic reviews: results of the metaquality cross sectional study.
|
From page 150... ...
2008. Disagreement in pri mary study selection between systematic reviews on negative pressure wound therapy.
|
From page 151... ...
2008. Poor reporting of search strategy and conflict of interest in over 250 narrative and systematic reviews of two biologic agents in arthritis: A systematic review.
|
From page 152... ...
In Cochrane handbook for systematic reviews of interventions, edited by J
|
From page 153... ...
A review of published randomized controlled trials in major medical journals. Clinical Trials 1(4)
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.