Evidence-Based Practice for Public Health Emergency Preparedness and Response (2020) / Chapter Skim
Currently Skimming:
3 An Evidence Review and Evaluation Process to Inform Public Health Emergency Preparedness and Response Decision Making
3 An Evidence Review and Evaluation Process to Inform Public Health Emergency Preparedness and Response Decision Making
Pages 71-115
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 71... ...
1 The committee defined PHEPR practices as a type of process, structure, or intervention whose implementation is intended to mitigate the adverse effects (e.g., morbidity and mortality, economic impacts) of a public health emergency.
|
From page 72... ...
As discussed in Chapter 1, knowledge regarding evidence-based practice is critically needed in PHEPR given the mandate of the PHEPR system to mitigate the health, financial, and other impacts of public health emergencies. To date, however, there has been little effort to develop a rigorous and transparent process for identifying evidencebased PHEPR practices.
|
From page 73... ...
. Evolving Methods for Evaluating Complex Health Interventions in Complex Systems As policy makers and practitioners have increasingly recognized the importance of having an evidence base to tackle complex challenges, there has been a growing movement among those who conduct systematic reviews and develop guidelines to embrace methods that take a complexity perspective and use multiple sources and types of evidence.
|
From page 74... ...
The series Complex Intervention Systematic Reviews, which was published in 2017 in the Journal of Clinical Epidemiology, resulted from an expert meeting convened by AHRQ. In 2019, WHO released the series Complex Health Interventions in Complex Systems: Concepts and Methods for Evidence-Informed Health Decisions, which was published in BMJ Global Health.
|
From page 75... ...
. Setting is not limited to geographic location, but also encompasses the sociocultural and demographic environment, as well as the characteristics of the communities and the responding entities (e.g., organizational structure, managerial experience, staff capabili 4 While there is general understanding of quantitative evidence as numerical data derived from quantitative measurements, misconceptions regarding what constitutes qualitative research and qualitative evidence are common.
|
From page 76... ...
. A considerable challenge when reviewing evidence to determine the effectiveness of PHEPR practices and implementation strategies relates to the often indirect links between the practices and primary health outcomes (e.g., morbidity and mortality)
|
From page 77... ...
To inform its methodology, the committee began by reviewing existing frameworks for evaluating different sources and types of evidence, both in health care and in other areas in which experimental clinical trials may be impossible or impractical (such as aviation safety) , to determine their potential to accommodate the diverse PHEPR evidence base and questions of interest to PHEPR stakeholders.
|
From page 78... ...
Findings from the evidence reviews are made accessible through the agency's clearinghouse to inform decision making. Individual studies are reviewed and assigned a rating for the strength of causal evidence.
|
From page 79... ...
inclusion, but instead adjusts the COE based on the quality and risk and GRADE- of bias of studies included in the analysis. GRADE also utilizes an Confidence in the Evidence to Decision framework for making transparent, evidence Evidence from based recommendations in the form of guidelines, considering Reviews of Qualitative evidence beyond that related to effect (e.g., feasibility, acceptability)
|
From page 80... ...
The committee's approach was also informed by a 2018 WHO report containing guidelines for emergency risk communication, which provided a timely example of how GRADE might be adapted and used in conjunction with GRADE-Confidence in the Evidence from Reviews of Qualitative Research (GRADE-CERQual) to evaluate evidence and develop recommendations on a wider range of PHEPR practices (WHO, 2018)
|
From page 81... ...
The development of this methodology and its application to the evaluation of evidence for four exemplar PHEPR review topics were undertaken in parallel using a highly iterative process, the steps of which are described in the sections below. APPLYING A METHODOLOGY TO REVIEW, SYNTHESIZE, AND ASSESS THE COE FOR PHEPR PRACTICES This section outlines the key elements and approaches of the methodology developed and applied by the committee for reviewing and evaluating PHEPR evidence to inform decision making (summarized in Box 3-2)
|
From page 82... ...
The overall rating for this domain is derived from the risk-of-bias assessments for all individual studies included in the body of evidence. • Indirectness -- considers whether the available evidence differs from the target of interest, including differences in population, interventions, outcome measures (e.g., use of surro gate outcomes removed in the putative causal pathway from important endpoints)
|
From page 83... ...
This approach was intended to result in a methodology that would be applicable across a range of different practices for which the evidence base would be expected to differ in nature. As a first step, the committee needed to select a set of review topics that would be illustrative of the diversity of PHEPR practices.
|
From page 84... ...
The next steps, standard practice for most systematic reviews and described in more detail in Appendix A, included the development of analytic frameworks and the identification of key questions11 for each topic area to further define the scope of the reviews; the development and execution of a comprehensive search of the peer-reviewed and gray literature; and the screening of titles, abstracts, and full-text articles by two reviewers to identify articles meeting the committee's inclusion criteria. Of note, determining the eligibility of studies required iterative discussions as the review methods, the scope of the four topics, and the outcomes used to assess effectiveness were refined over time.
|
From page 85... ...
population public health emergency R Existing controversy or conflicting evidence 4.
|
From page 86... ...
The quality assessment and synthesis of qualitative studies were conducted by a commissioned team at Wayne State University. The evaluation and synthesis of selected modeling studies were performed by a modeling expert at Stanford University, and the evaluation and synthesis of AARs and case reports were conducted by a PHEPR expert in evaluation at Columbia University.
|
From page 87... ...
assessed* assessed with CASP Synthesis of Narrative synthesis Framework synthesis Framework synthesis key findings Synthesized Findings Synthesized Findings/COE Individual Study Findings Synthesized Findings/COE Synthesized Findings COE assessed with COE assessed with GRADE GRADE-CERQual FIGURE 3-2 Classification and consolidation of studies into methodological streams.
|
From page 88... ...
Such studies were not classified as qualitative research studies, but their findings were extracted and considered separately in the qualitative evidence synthesis to affirm or question the findings of the more complete qualitative studies. AARs and case reports The committee sought to include a synthesis of AARs for two of its reviews (the EOC and Information Sharing Capability test cases)
|
From page 89... ...
. Additional detail on these tools and their use in quality assessment of individual quantitative and qualitative research studies is provided in Appendix A
|
From page 90... ...
For evaluating evidence of effectiveness, the coherence of evi dence from other streams was considered in rating the COE for each outcome. Quantitative evidence synthesis and grading For each of the test cases, the committee first assessed the body of quantitative impact studies using the GRADE approach (see Box 3-2 earlier in this chapter for a description of the GRADE assessment domains)
|
From page 91... ...
Qualitative evidence synthesis and grading For the qualitative evidence synthesis, the primary studies were uploaded into Atlas.ti (Version 8.1, Atlas.ti Scientific Software Development GmbH, Berlin, Germany) , and the key findings and supporting information from 17 Consistent with GRADE guidelines on rating the COE (Guyatt et al., 2011b)
|
From page 92... ...
were also phenomena of interest for the qualitative evidence synthesis. The Wayne State University team conducted the extraction and used the pragmatic framework synthesis method (Barnett-Page and Thomas, 2009; Pope et al., 2000)
|
From page 93... ...
For mixed-method studies that had both qualitative and quantitative portions, only the qualitative findings were coded. Step 3: Second-level coding involved a synthesis process of creating descriptive themes, where a theme was a family of descriptive codes in which codes that formed a cohesive set were grouped together.
|
From page 94... ...
for many PHEPR practices. Consequently, it is important to consider for all PHEPR evidence reviews (at the start of the process)
|
From page 95... ...
EtD frameworks to develop a novel EtD framework for the committee's use in formulating recommendations on evidence-based PHEPR practices. The PHEPR EtD framework comprised the following criteria:
|
From page 97... ...
Other types of evidence that may inform what works may be used to examine coherence with direct quantitative effectiveness evidence or may be used to inform committee judgment in the absence of direct quantitative evidence. AAR = after action report; CERQual = Confidence in the Evidence from Reviews of Qualitative Research; COE = certainty of the evidence; GRADE = Grading of Recommendations Assess ment, Development and Evaluation; PHEPR = public health emergency preparedness and response.
|
From page 98... ...
Upgraded Required very supportive mechanistic or modeling evidence or high-confidence findings COE one level from qualitative evidence synthesis. Upgraded Required a combination of supportive (or very supportive)
|
From page 99... ...
Other evidence review and guideline groups use multiple recommendation levels (e.g., strong and conditional recommendations) , and future groups conducting reviews of PHEPR practices may determine that modifications of the committee's methodology are needed to accommodate recommendations of different strengths.
|
From page 100... ...
Proportionality: Burdens imposed on individuals or groups, such as limitations on services pro vided or restrictions on personal liberties, should be necessary and commensurate with the scale of the public health emergency. As the situation evolves, it is also important to reassess regularly to ensure the continued optimal use of available resources.
|
From page 101... ...
The committee included in its key review questions for each of the evidence reviews a question related to the factors that created barriers to and facilitators of implementation of the PHEPR practice, which informed its implementation guidance. To address those review questions, evidence was synthesized using much the same process as that described above for the EtD framework.
|
From page 102... ...
, PHEPR practitioners, and PHEPR researchers; consultation with outside experts in systematic review methods and guideline development (during both public and closed session discussions) ; and iterative testing of the methodology on a small but diverse selection of PHEPR practices.
|
From page 103... ...
The committee found it was important when adapting and applying the methods and tools to its specific review questions to have input from those familiar with the subject area and the types of studies and other information available. The committee's methodology accommodated a wide range of evidence types, including evidence from RCTs, nonrandomized experimental studies, case reports, modeling studies, and descriptive surveys, as well as mechanistic evidence and parallel evidence from other fields.
|
From page 104... ...
The scoping review discussed in Chapter 2 found that modeling studies make up a substantial proportion of the evidence base for the Non-Pharmaceutical Interventions Capability (as well as the Medical Countermeasure Dispensing and Administration and Medical Materiel Management and Distribution Capabilities) , emphasizing the importance of integrating modeling evidence into PHEPR evidence reviews.
|
From page 105... ...
As discussed earlier in this chapter, for most PHEPR practices, the committee judged that it would not be conceptually appropriate to assume that an effect size existed independent of context and implementation fidelity. As others have done (Movsisyan et al., 2016; Rehfuess and Akl, 2013)
|
From page 106... ...
NOTES: Green = lowest-intensity tasks; yellow = moderate-intensity tasks; red = highest-intensity tasks. AAR = after action report; PICOTS = population, intervention, comparator, outcome, timing, setting.
|
From page 107... ...
, including such Delphi-type processes as the practitioner engagement activity the committee undertook to inform future potential PHEPR review topics (described in Box 3-5) .20 Moreover, the evidence base for PHEPR practices is continually evolving with the field.
|
From page 108... ...
Considering the requisite diversity of expertise and the need for guidelines to issue from an authoritative source with the trust of the PHEPR community and the ability to disseminate the guidelines widely, the committee concluded that a centralized approach supported by CDC is the best model for a system for ongoing evidence reviews of PHEPR practices. Importantly, CDC has extensive experience in overseeing evidence-based review processes, with notable examples including the CPSTF; the Advisory Committee on Immunization Practices (Lee et al., 2018)
|
From page 109... ...
The group should also identify and communicate key PHEPR evidence gaps in annual reports to CDC and Congress to guide future research on the effectiveness of PHEPR practices. 22 In 2019, the AHRQ budget for USPSTF was $11.6 million according to AHRQ's 2020 operating plan, available at https://www.ahrq.gov/sites/default/files/wysiwyg/cpi/about/mission/operating-plan/operating-plan-2020.pdf (accessed February 21, 2020)
|
From page 110... ...
2016. Guidance on choosing qualitative evidence synthesis methods for use in health technology assessments of complex interventions.
|
From page 111... ...
2019. Qualitative evidence synthesis for complex interventions and guideline development: Clarification of the purpose, designs and relevant methods.
|
From page 112... ...
2019. Synthesising quantitative evidence in systematic reviews of complex health interventions.
|
From page 113... ...
2013a. Complex interventions and their implications for systematic reviews: A pragmatic approach.
|
From page 114... ...
2017. Clinical epidemiology in South Africa, paper 3: Logic models help make sense of complexity in systematic reviews and health technology assessments.
|
From page 115... ...
2008. Methods for the thematic synthesis of qualitative research in systematic reviews.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.