Improving the Quality of Systematic Reviews: Discussion, Conclusions, and Recommendations1
Abstract: The committee recommends that sponsors of systematic reviews (SRs) of comparative effectiveness research (CER) should adopt appropriate standards for the design, conduct, and reporting of SRs and require adherence to the standards as a condition for funding. The newly created Patient-Centered Outcomes Research Institute and agencies of the U.S. Department of Health and Human Services should collaborate to improve the science and environment for SRs of CER. Although the recommended SR standards presented in this report are based on the best available evidence and current practice of respected organizations, many of the standards should be considered provisional pending more methods research. This chapter presents a framework for improving the quality of the science underpinning SRs in several broad categories: involving the right people, methods for conducting reviews, methods for grading and synthesizing evidence, and methods for communicating and using results.
Systematic reviews (SRs) should be at the center of programs developing a coordinated approach to comparative effectiveness
research (CER), both for setting priorities among individual CER studies and for appropriately focusing studies during their design. The committee recognizes that fully implementing all of the SR standards proposed in this report will be costly, resource intensive, and time consuming. Further, as previous chapters make clear, the evidence base supporting many elements of SRs is incomplete and, for some steps, nonexistent. Finally, the committee is fully aware that there is little direct evidence linking high-quality SRs to clinical guidance that then leads to improved health. Nonetheless, designing and conducting new comparative effectiveness studies without first being fully informed about the state of the evidence from an SR risks even higher costs and waste by conducting studies that are poorly designed or redundant. Research organizations such as the Agency for Healthcare Research and Quality (AHRQ) Effective Health Care Program, Centre for Reviews and Dissemination (CRD) (University of York), and the Cochrane Collaboration have published standards, but none of these are generally accepted and consistently applied during planning, conducting, reporting, and peer reviewing of SRs. Furthermore, the environment supporting development of a robust SR enterprise in the United States lacks both adequate funding and coordination; many organizations conduct SRs, but do not typically work together. Thus the committee concludes that improving the quality of SRs will require advancing not only the science supporting the steps in the SR process and linking SRs to improved health, but also providing a more supportive environment for the conduct of SRs. In this chapter the committee outlines some of the principal issues that must be addressed in both of these domains.
Throughout the chapter and in its final recommendations, the committee refers to the newly established Patient-Centered Outcomes Research Institute (PCORI) and, in particular, its Methodology Committee, as a potentially appropriate organization to provide comprehensive oversight and coordination of the development of the science and to promote the environment for SRs in support of CER in the United States. The committee views PCORI as an unusually timely development—albeit untested—that should help advance the field of SRs as an essential component of its overall mission, building on the strengths of well-established programs in the United States (e.g., AHRQ, National Institutes of Health [NIH]) and internationally (e.g., Cochrane Collaboration, National Institute for Health and Clinical Excellence in the United Kingdom), that either produce or rely on SRs for policy purposes. Nonetheless, while the committee views PCORI as relevant and promising, PCORI is by no means the only way to achieve the stated aims. Other agencies, working
individually, are able to contribute to advancing the field as well. However, the committee believes that collaborative relationships among agencies, both public and private, would be most effective at contributing to progress. Furthermore, the committee recognizes that U.S. developments are only part of a substantial international effort focused on how best to conduct SRs, an effort that in some countries is advanced and highly sophisticated. Given the potential for duplication of efforts, the need to ensure that gaps in the information base are appropriately addressed, and the need for efficient use of limited available resources, the coordination across multiple organizations within the United States and throughout the world will have clear benefits and should be viewed as essential.
IMPROVING THE SCIENCE OF SYSTEMATIC REVIEWS
Establishing a process for ongoing development of the research agenda in SRs must be an important part of the path forward. Although the committee believes the recommended standards and elements of performance presented in this report are founded on the best available evidence and current practice of respected organizations, all SR standards should be considered provisional pending additional experience and research on SR methods. The committee recognizes that each of its recommended standards could be examined in appropriately designed research, with the expectation that some items would be validated, some discarded, and some added. Future research to develop methods that promote efficiency and scientific rigor is especially important. A detailed description of research that might be conducted on each step, however, is outside the committee’s scope of work and would require substantial time and resources. We also note that some of the needed work may be more appropriately categorized as program development and evaluation than research.
The committee promotes the goal of developing a coordinated approach to improving the science of SR, embedded in a program of innovation, implementation, and evaluation that improves the quality of SRs overall. PCORI is an appropriate organization to provide comprehensive oversight and coordination of the development of the science of SRs in support of CER, in cooperation with agencies of the U.S. Department of Health and Human Services (HHS). It could, also function as an important U.S. collaborator with international organizations similarly focused (e.g., Cochrane Collaboration, Campbell Collaboration, and CRD). Among other goals, such a coordinated program would support a description of methods currently
employed, methodological research, and comparative studies of alternative approaches, working with international partners to efficiently advance the research agenda. By supporting innovation, the incorporation of a feedback loop into design and reporting of trials and observational studies, and the appropriate and intentional (not accidental or wasteful) replication of methodological research and SRs of methods, PCORI will contribute to ensuring that standards are evidence based. SR methods research will also help to identify gaps in the literature and, through the application of the findings of empirical, “meta-epidemiologic” approaches (i.e., investigations of how particular features of study design or study populations relate to the validity and applicability of primary studies), will provide information about how well standards are being applied.
In this section, the committee proposes a framework for improving the quality of the science underpinning SRs in several broad categories: strategies for involving the right people, methods for conducting the SR, methods for synthesizing and evaluating evidence, and methods for communicating and using results.
Strategies for Involving the Right People2
Successful execution and effective use of an SR is a collaborative activity requiring a wide range of experience and expertise among the contributors (the review team). The committee believes that involving people with sufficient experience in each step of the SR process has not received enough attention. While noting that a typical review will require people with certain expertise in specific steps, the committee resisted proposing a standard that a particular menu of experts and stakeholders must be a part of every SR, regardless of topic and purpose. On the other hand, the committee believes that current practice, particularly among groups with modest resources, probably underestimates and undervalues the need for certain kinds of expertise, with the result that SRs vary enormously in quality and credibility. In contrast to authors carrying out the traditional literature review, review teams need formal education and training, which should include hands-on experience and mentoring. There is also wide variability in the involvement of consumers and other users and stakeholders in SRs. Finally, the committee recognizes that
See Chapter 2 for a discussion of the individuals who should be included on the review team and the importance of involving users and stakeholders in the SR process.
depth of experience in participating in SRs varies among individuals in a given field, so that the mere presence of an individual with general expertise in a relevant domain does not ensure that the issues will be covered adequately in the review. As an example, not all biostatisticians are fluent in methods of SRs, even though they may be experts in other areas. Similarly, generalist librarians and other information specialists may require special training or experience in conducting SRs, including special knowledge of bibliographic database-specific search terms, to design and execute the search strategy appropriately.
Little descriptive information is available about how the issues of personnel and expertise are handled in various SR enterprises, and the evidence base comparing different approaches is inadequate. For example, we believe comparative studies of models involving consumers and patients are needed. As another example, research on the effects of conflict of interest and bias is provocative, but the topic needs to be addressed more systematically using sophisticated research methods. The committee recognizes that performing such research will present challenges, beginning with defining appropriate outcome measures in these methodologic investigations.
Methods for Conducting the Systematic Review
Developing a review protocol, locating and screening studies, and collecting and appraising the data (the subject of Chapters 2 and 3) are many specific steps along the pathway to completing an SR. Some steps, such as the use of different databases and sensitive search filters to identify relevant literature, are supported with empirical data, but many other steps have not been examined in research. The committee believes an entity such as PCORI should systematically support research that examines key steps in the methods involved in conducting an SR. The committee’s criteria (Table 1-2) might be a useful framework to identify topics for further research. For example, how do alternative approaches to some individual steps affect the scientific rigor and efficiency of SR? In addition, we have data on how particular steps in a review are potentially influenced by bias (e.g., reporting biases), but not on whether the bias is of concern in an individual review. The challenge will be not only to identify topics that can be researched, but also to set priorities among them. For example, those standards that have a substantial effect on cost (e.g., dual extraction) might be initially considered higher priority.
Methods for Synthesizing and Evaluating Evidence3
Quantitative synthesis of empirical data is a highly developed and active topic of research. Research ranges from the theoretical (with emphasis on the statistics) to comparisons of different modeling approaches. Despite ongoing research in the field, many outstanding questions remain, particularly related to the synthesis of complex multivariate data structures. The committee recommends a range of approaches to answering these questions, including theoretical, empirical, and simulation studies as appropriate.
Qualitative synthesis (i.e., a narrative description of available evidence without drawing conclusions based on statistical inference) has received less attention in research than quantitative synthesis, although the committee recommends that it be part of all SRs. Failing to perform a qualitative synthesis is problematic because the evidence available to answer specific SR questions often does not lend itself to quantitative methods. There is no empirical research to guide synthesis when a qualitative synthesis is the only approach possible. Even when a quantitative review is conducted, we need to understand what perspectives and judgments should be considered in undertaking qualitative synthesis that require authors to be reflective, critical, and as objective as possible in their presentation and interpretation of the data. Because an important goal of qualitative synthesis is communication to users and stakeholders, research in this area might also focus on effectiveness of communication, or perceived objectivity.
Furthermore, although a formal approach to assessing the quality of a body of evidence is recommended, there is little, if any, research testing the reliability and validity of existing approaches to evidence assessment (e.g., the Grading of Recommendations Assessment, Development, Evaluation, or GRADE model). Careful consideration of how to define validity for an SR (e.g., defining a reference standard) will be an important challenge in this research. Finally, the field clearly needs to develop a common lexicon and set of symbols for summarizing the quality of evidence, a process that will need to be coordinated among groups using SRs to develop clinical guidance where there is further variation in lexicon and symbols.
The committee believes that PCORI and its Methodology Committee should invest in research on quantitative and qualitative syntheses and grading of the body of evidence for SRs. This work
See Chapter 4 for discussions on qualitative synthesis, quantitative synthesis, and evaluating the quality of a body of evidence.
should be done in close collaboration with other groups commissioning and doing SRs, including the U.S. Public Health Service (e.g., AHRQ, NIH, U.S. Preventive Services Task Force, and the Community Guide), professional organizations and associations, and existing international organizations (e.g., the Cochrane Collaboration).
Methods for Communicating and Using the Results4
The committee placed high value on public availability of, and transparency in reporting, SRs, but was not able to cite specific research supporting a particular format. Research on how to most effectively communicate the results of an SR to various users (e.g., clinicians, clinical guidelines panels, consumers, healthcare organizations, payers, both public and private, etc.) is limited, and more would be useful. The committee also notes that in current practice, the process of conducting some SRs is often formally separated from processes in which they are actually used. Although appropriate objectivity and freedom from undue influence need to be maintained, the committee believes that research examining the utility of connecting the SR with its intended users (e.g., clinical guidelines groups, practicing clinicians, and patients), as well as effectiveness and impact of current collaborative efforts, would be timely.
IMPROVING THE ENVIRONMENT FOR SUPPORTING SYSTEMATIC REVIEWS
Developing the science of SRs is not enough to address all the issues that the committee identified as important to improving the quality of SRs to inform CER. A number of environmental factors will critically influence whether the quality of SRs can be improved. Some are best described as infrastructure (e.g., training, registries), but others have to do with SRs as required elements for a culture aimed at improving CER overall.
The committee believes there is a need for greater collaboration among multiple stakeholder groups, including PCORI, government agencies (especially AHRQ and NIH), the Cochrane Collaboration, medical professional societies, researchers, healthcare delivery organizations, patient interest groups, and others. Such multidisciplinary and multiorganizational collaborations have the potential to improve the rigor, transparency, and efficiency of SRs; encourage standardization of methods and processes; set priorities
See Chapter 5 for a discussion on preparing final reports of SRs.
for selection of clinical topics of interest to clinicians and patients; reduce unintentional duplication of efforts; provide a shared funding source for the generation of high-quality evidence reviews; more effectively manage conflicts of interest; and facilitate implementation of reviews. Developing effective collaborations, however, requires a transformation in current thinking and structural approaches to conducting SRs. The importance of including international collaborators in discussions of priorities for methodologic research, in particular, cannot be overstated; there is deep expertise and effective leadership in the SR field inside and outside U.S. borders.
The committee also underscores that its recommended standards and elements of performance for publicly funded SRs are provisional, subject to change as the science of SRs advances and lessons are learned from applying the standards in practice. A mechanism is needed to monitor the progress of the science and update the standards periodically to reflect current best practice.
As in the preceding section on developing the science, the committee found that dividing issues into four general categories was a useful way to organize our conclusions: (1) strategies for involving the right people; (2) methods for conducting SRs; (3) methods for synthesizing and evaluating evidence; and (4) methods for communicating and using results.
Strategies for Involving the Right People
The committee believes the environment must be improved to allow and encourage people with sufficient training and experience to be engaged in an SR. Training and professional development must be well established, supported, and recognized by the research community before aspiring researchers will feel secure in choosing careers in SR. Rewards and promotion systems for faculty and scientists in academic institutions need to recognize that the conduct of SRs and the research on SR methods are inherently collaborative efforts. Substantive intellectual contributions to such collaborative efforts need to be recognized in meaningful ways that will attract, not discourage, participation by top scientists. Training targeted to the specific skills needed for SRs needs to be addressed in any national program supporting CER. This research is often multidisciplinary, which training curriculums must take into account. This may require innovation as many disciplines narrowly focus their pre- and post-doctoral training.
Support for the training of users and stakeholders—such as consumers, patients, clinicians, payers, representatives from the
insurance industry—in the design, conduct, and use of SRs will be essential if representatives from these groups are to contribute effectively to the choice of the clinical scenario for the review and otherwise fully participate in the conduct of a review and its dissemination. Finally, involving the right people requires providing an environment in which a transparent and robust approach to managing conflict of interest and bias is developed, implemented, and rigorously evaluated for all who participate in an SR. Key CER studies often involve proprietary interests, which involve confidentiality and legal issues. Promoting thorough, transparent analysis will require consideration of these interests, potentially including changes to rules and regulations.
Methods for Conducting Systematic Reviews and for Synthesizing and Evaluating Evidence
The science of conducting an SR, from design through review, synthesis, and evaluation, can only thrive in an environment in which all aspects are supported in a culture valuing the contribution of SRs to improvements in health care. The committee noted many specific ways in which the environment could provide such support. These include establishing a registry for SR protocols (under development by CRD at the University of York in the United Kingdom),5 providing a repository for data extracted during the conduct of SRs (being explored by the Tufts Medical Center Evidence-based Practice Center),6 publicly posting protocols and reviews, using public mechanisms to ensure timely updating of protocols and reviews, guaranteeing access to data from primary studies for use in SRs, and ensuring that SRs are a required part of planning, designing, and conducting future primary CER.
Establishing a collaborative methodologic research infrastructure will also be valuable to advancing the science of SRs. Some aspects of methodology might be amenable to rigorous study through the various organizations that fund SRs. For example, a study comparing
structures for presenting qualitative reviews might randomize SRs being performed by several organizations, measuring acceptability, efficiency, or transparency. The details of study designs are beyond the scope of this report, but the committee believes that a coordinated and collaborative approach to reviews that are already being conducted could offer rich opportunities for efficiently advancing research, particularly if this planning is done prospectively so that reviews are updated in a timely manner.
Methods for Communicating and Using the Results
The committee believes that developing an environment that supports the understanding and use of SRs is critical if the enterprise is to improve CER. Terminology should be consistent, and conventions and standards for publication uniform and well defined. When publicly funded SRs are intended to be used in support of clinical guidance, these reviews should be formally linked with guidelines committees that also meet rigorous standards. The use and usefulness of SRs commissioned as part of a guidelines process should be evaluated once the guideline is implemented, with a feedback loop into future reviews on similar topics and methods used to conduct the review.
CONCLUSIONS AND RECOMMENDATIONS
The committee explored a wide range of topics in its deliberations. The standards and elements of performance form the core of our conclusions, but the standards themselves do not indicate how the standards should be implemented, nor do the standards address issues of improving the science for SRs or improving the environment that supports the development and use of an SR enterprise. In consequence, the committee makes the following two recommendations.
Recommendation 1: Sponsors of SRs of CER should adopt standards for the design, conduct, and reporting of SRs and require adherence to the standards as a condition for funding.
SRs of CER in the United States are now commissioned and conducted by a vast array of private and public entities, some supported generously with adequate funding to meet the most exacting standards, others supported less generously with the result that compromises must be made at every step of the review. Regardless
of the level of funding, all sponsors of SRs of CER should adopt standards for the planning, conducting, and reporting of SRs to ensure that a minimal level of quality is met, and should make the adopted standards publicly available. The committee recognizes that its recommended standards are provisional, subject to change as the science of SRs advances and lessons are learned from applying the standards in real-world situations. Also, its standards and elements of performance are at the “exacting” end of the continuum, some of which are within the control of the review team whereas others are contingent on the SR sponsor’s compliance. However, high-quality reviews require adequate time and resources to reach reliable conclusions. The recommended standards are an appropriate starting point for publicly funded reviews in the United States (including PCORI, federal, state, and local funders) because of the heightened attention and potential clinical impact of major reviews sponsored by public agencies. The committee also recognizes that a range of SRs are supported by public funds derived from nonfederal sources (e.g., state public health agencies) and private sources where these standards will be seen as an aspiration rather than as a minimum bar. Application of the standards to reviews embedded within other programs that may be publicly funded (e.g., highly focused reviews conducted by individual investigators as part of research grants) also presents difficult operational issues. On the whole, however, the committee feels strongly that the standards (and their successor standards) should serve as a benchmark for all SRs of CER. They could even, for example, be used to inform other topic areas (e.g., risk assessment, epidemiologic research) where standards are also being developed. SRs that significantly deviate from the standards should clearly explain and justify the use of different methods.
Recommendation 2: PCORI and the Department of Health and Human Services (HHS) agencies (directed by the Secretary of HHS) should collaborate to improve the science and environment for SRs of CER. Primary goals of this collaboration should include
Developing training programs for researchers, users, consumers, and other stakeholders to encourage more effective and inclusive contributions to SRs of CER;
Systematically supporting research that advances the methods for designing and conducting SRs of CER;
Supporting research to improve the communication and use of SRs of CER in clinical decision making;
Developing effective coordination and collaboration between U.S. and international partners;
Developing a process to ensure that standards for SRs of CER are regularly updated to reflect current best practice; and
Using SRs to inform priorities and methods for primary CER.
This recommendation conveys the committee’s view of how best to implement its recommendations to improve the science and support the environment for SRs of CER, which is clearly in the public’s interest. PCORI is specifically named because of its statutory mandate to establish and carry out a CER research agenda. As noted above, it is charged with creating a methodology committee that will work to develop and improve the science and methods of SRs of CER and to update such standards regularly. PCORI is also required to assist the Comptroller General in reviewing and reporting on compliance with its research standards, the methods used to disseminate research findings, the types of training conducted and supported in CER, as well as the extent to which CER research findings are used by healthcare decision makers. The HHS agencies are specifically named because AHRQ, NIH, the Centers for Disease Control and Prevention, and other divisions of HHS are major funders and producers of SRs. In particular, the AHRQ Effective Health Care Program has been actively engaged in coordinating high-quality SRs and developing SR methodology. The committee assigns these groups with responsibility and accountability for coordinating and moving the agenda ahead.
The committee found compelling evidence that having high-quality SRs based on rigorous standards is a topic of international concern, and that individual colleagues, professional organizations, and publicly funded agencies in other countries make up a large proportion of the world’s expertise on the topic. Nonetheless, the committee necessarily follows the U.S. law that facilitated this report, which suggests a management approach appropriate to the U.S. environment is useful. A successful implementation of our final recommendation should result in a U.S. enterprise that participates fully and harmonizes with the international development of SRs, serving in some cases in a primary role and in others as a facilitator or participant. The new enterprise should also fully understand that this cannot be entirely scripted and managed in advance—structures and processes must allow for innovation to arise naturally from among U.S. individuals and organizations already fully engaged in the topic.