Page 239 Cite

Suggested Citation:"Appendix E: Summary of Selected Systematic Review Practices." National Academies of Sciences, Engineering, and Medicine. 2022. Evaluating the Process to Develop the Dietary Guidelines for Americans, 2020-2025: A Midcourse Report. Washington, DC: The National Academies Press. doi: 10.17226/26406.

×

Appendix E

Summary of Selected Systematic Review Practices¹

The 2017 National Academies report recommended that the secretary of the U.S. Department of Agriculture (USDA) should ensure the Nutrition Evidence Systematic Review (NESR) (formerly the Nutrition Evidence Library [NEL]) align with best practices by enabling ongoing training, engagement with and learning from external groups on the forefront of systematic review methods, and inviting external systematic review experts to evaluate NESR’s methods and invest in technological infrastructure (see Chapter 3, Box 3-5). The committee first sought to understand various aspects of systematic reviews and practices for conducting them that are important to consider during the systematic review process. The committee gave its interpretation on the most important factors, listed under “Practice” in Table E-1 below. The committee then sought to understand how these practices are handled by private organizations and agencies that routinely use the technology of systematic reviews. These include Cochrane, the U.S. Preventative Services Taskforce (USPSTF), the Agency for Healthcare Research and Quality (AHRQ), the Office of Dietary Supplements (ODS), and the Institute of Medicine (IOM). The practices used by these organizations are summarized under the column “Committee-Identified Practices.” In addition, the committee summarized NESR systematic review practices used in the 2020 Dietary Guidelines for

___________________

¹ The title of this appendix was modified after release of a prepublication version of the report to the sponsor to clarify that these are selected examples of systematic review practices and that these are based on the committee’s identified sources and are not standardized.

Page 240 Cite

Suggested Citation:"Appendix E: Summary of Selected Systematic Review Practices." National Academies of Sciences, Engineering, and Medicine. 2022. Evaluating the Process to Develop the Dietary Guidelines for Americans, 2020-2025: A Midcourse Report. Washington, DC: The National Academies Press. doi: 10.17226/26406.

×

Americans (DGA) cycle. It is noteworthy that the 2017 National Academies report commended NESR practices, and envisioned the recommendations in Chapter 3 as a means of strengthening the process for their principles (see Box 3-1).

TABLE E-1 Systematic Review Practices

Practice	Committee-Identified Practices	NESR 2020 DGA	Implementation Impact^a
Transparency	Cochrane requires registration of full systematic review protocols through PROSPERO PROSPERO registered protocols include title, description of participants and funding, complete search plan, data extraction and coding plan, risk of bias assessment, strategy for data synthesis, and analysis subgroups	Systematic review protocol components related to question, analytical framework and search strategy (and associated changes) were published on the NESR website during the DGAC deliberations Search terms were available to the public upon request DGAC methodology manual indicates that “Any revisions to protocols that occurred during the course of the committee’s work were documented, posted online, and presented at meetings. The literature search plans (i.e., search terms) and screening results (i.e., flow chart, included and excluded articles) were added to the protocols as they were finalized”	Partially implemented

Page 241 Cite

Suggested Citation:"Appendix E: Summary of Selected Systematic Review Practices." National Academies of Sciences, Engineering, and Medicine. 2022. Evaluating the Process to Develop the Dietary Guidelines for Americans, 2020-2025: A Midcourse Report. Washington, DC: The National Academies Press. doi: 10.17226/26406.

×

Practice	Committee-Identified Practices	NESR 2020 DGA	Implementation Impact^a
	Original protocols remain publicly available and are cited in the final systematic review to optimize transparency Final reports/publications should clearly identify any changes to the original protocols and include a rationale for those changes	Data extraction, coding and analysis plan were not included in publicly available protocols during DGAC deliberations Originally approved search plan protocol documents are no longer publicly available Final protocols do not include dates of which minutes addressed the changes Detail regarding search plan protocol changes vary between systematic review reports and the Scientific Report of DGAC 2020	Likely to have minor impact on systematic review results Could have major impact on transparency and public perception of integrity

Page 242 Cite

Suggested Citation:"Appendix E: Summary of Selected Systematic Review Practices." National Academies of Sciences, Engineering, and Medicine. 2022. Evaluating the Process to Develop the Dietary Guidelines for Americans, 2020-2025: A Midcourse Report. Washington, DC: The National Academies Press. doi: 10.17226/26406.

×

Practice	Committee-Identified Practices	NESR 2020 DGA	Implementation Impact^a
Managing conflict of interest	Cochrane mandates that authors report all potential forms of COI going back at least 3 years IOM and Cochrane highlight the need to manage COI that might diminish credibility of the findings of individual participants	Guidelines for handling disclosure of COI in selection of the DGAC have been described in the 2020 DGAC Scientific Report COI is coordinated by the USDA or HHS Committee Management Officer as part of each member’s vetting process using guidance provided by the U.S. Office of Government Ethics COI are reviewed semi-annually All individuals with a substantive role in a NESR systematic review project, including NESR staff and external experts, submit a curriculum vitae and disclose potential COI	Implemented at individual participant level

Page 243 Cite

Suggested Citation:"Appendix E: Summary of Selected Systematic Review Practices." National Academies of Sciences, Engineering, and Medicine. 2022. Evaluating the Process to Develop the Dietary Guidelines for Americans, 2020-2025: A Midcourse Report. Washington, DC: The National Academies Press. doi: 10.17226/26406.

×

Practice	Committee-Identified Practices	NESR 2020 DGA	Implementation Impact^a
	IOM indicates that bias may be conscious or unconscious and influence choices made throughout the process Literature is emerging on best ways to manage institutional conflict of interest for research performing and research funding organizations Funders should have no role in Clinical Practice Guideline development	Information on managing institutional conflict of interest for research performing and research funding organizations is not available	Emerging area could have major impact on transparency and public perception of integrity

Page 244 Cite

Suggested Citation:"Appendix E: Summary of Selected Systematic Review Practices." National Academies of Sciences, Engineering, and Medicine. 2022. Evaluating the Process to Develop the Dietary Guidelines for Americans, 2020-2025: A Midcourse Report. Washington, DC: The National Academies Press. doi: 10.17226/26406.

×

Practice	Committee-Identified Practices	NESR 2020 DGA	Implementation Impact^a
Formulation of research questions and analytical framework	AHRQ and IOM rely heavily on development of analytical framework USPSTF allows public to nominate a new topic ODS takes a similar approach and specifically highlights using an iterative approach to refining review questions, analytical framework, and framing of research question (population, intervention, control and outcomes [PICO]) AHRQ and USPSTF use an expert “topic team” to consult on the analytical framework and inclusion/exclusion criteria, with all other systematic review steps being performed by a separate evidence team Cochrane focuses on developing a strong PICO framework, but relies less heavily on an analytical framework	In 2020, the federal stakeholders (USDA and HHS members of Dietary Guidelines Subcomittee) identified the broad topics and questions to address The DGAC set priorities and refined the questions in an iterative process with NESR; DGAC created analytical framework for new systematic reviews	Partially implemented (peer review addressed below,^b documenting changes in protocols addressed above)

Page 245 Cite

Suggested Citation:"Appendix E: Summary of Selected Systematic Review Practices." National Academies of Sciences, Engineering, and Medicine. 2022. Evaluating the Process to Develop the Dietary Guidelines for Americans, 2020-2025: A Midcourse Report. Washington, DC: The National Academies Press. doi: 10.17226/26406.

×

Practice	Committee-Identified Practices	NESR 2020 DGA	Implementation Impact^a
	Formulation of the research questions and PICO framework are peer reviewed by content and systematic review experts at the systematic review protocol stage Any changes made after systematic review protocol approval are transparently documented in the published full review When systematic reviews are conducted specifically to inform particular guidelines, the guideline development group and systematic review team should interact regarding the scope, approach, and output of both processes

Page 246 Cite

Suggested Citation:"Appendix E: Summary of Selected Systematic Review Practices." National Academies of Sciences, Engineering, and Medicine. 2022. Evaluating the Process to Develop the Dietary Guidelines for Americans, 2020-2025: A Midcourse Report. Washington, DC: The National Academies Press. doi: 10.17226/26406.

×

Practice	Committee-Identified Practices	NESR 2020 DGA	Implementation Impact^a
Search Strategy	Search strategies that are tailored to the various databases created by librarians trained in systematic review training Search strategies are externally peer reviewed as part of systematic review protocol review Search strategy results are captured at each stage of the search in flow diagrams	Overall search strategy formed a priori by NESR Search strategies tailored for each systematic review are created by a NESR systematic review librarian Search strategies are peer reviewed by a second internal NESR librarian but are not reviewed by an external peer review at the systematic review protocol stage Search strategy results are captured and reported at each step	Partially implemented (more detail in task 2, peer review addressed later^b)

Page 247 Cite

Suggested Citation:"Appendix E: Summary of Selected Systematic Review Practices." National Academies of Sciences, Engineering, and Medicine. 2022. Evaluating the Process to Develop the Dietary Guidelines for Americans, 2020-2025: A Midcourse Report. Washington, DC: The National Academies Press. doi: 10.17226/26406.

×

Practice	Committee-Identified Practices	NESR 2020 DGA	Implementation Impact^a
Screening	Cochrane/AHRQ/IOM rely on screening by two individuals Have well-articulated procedures for how to resolve conflict between the two reviewers Transparency in listing of excluded articles, often with reasons	Screening processes include two independent screeners Excluded articles are listed with a rationale in the final systematic review	Implemented
Data Extraction	Most leading groups have their own data extraction tools Data extraction is performed independently by two trained abstractors Process is identified to resolve differences in data extraction Data extraction parameters are included in the registered and peer-reviewed protocols Any modifications to data collection made throughout the systematic review are identified in the final protocol with rationale for change	In 2020, the NESR analysts and contractors extracted the data Extracted data were crosschecked by a second NESR analyst (DGAC, 2020) instead of having two independent abstractors Discrepancies that could not be resolved between the two analysts was taken to a third analyst or the committee for resolution Use of standardized data extraction forms (with some tailoring for specific needs for each review) and DistillerSR software ensured that common data elements were captured in a similar way across all systematic reviews	Partially implemented Not in strict alignment, but likely to have minor impact on outcome

Page 248 Cite

Suggested Citation:"Appendix E: Summary of Selected Systematic Review Practices." National Academies of Sciences, Engineering, and Medicine. 2022. Evaluating the Process to Develop the Dietary Guidelines for Americans, 2020-2025: A Midcourse Report. Washington, DC: The National Academies Press. doi: 10.17226/26406.

×

Practice	Committee-Identified Practices	NESR 2020 DGA	Implementation Impact^a
Risk of Bias	Analytical tools have been developed to effectively characterize risk of bias for specific types of research At least two assessors complete ROB Cochrane has developed risk of bias tools for both RCTs and non-RCTs including domains of consistency, directness, precision, generalizability, and publication bias Both publication and outcome reporting bias has been recently explored as a factor that may be important to address in policy-related systematic reviews, although these generally include intervention research Funding is specifically excluded as a source of bias and in validated risk of bias tools, but can be extracted as part of characteristics of the study.	NESR employed risk of bias tools commonly used by other groups with systematic review expertise including the Cochrane Risk of Bias 2.0, the ROBINS-I for non-RCTs, and a modified ROBINS-I designed for observational studies NESR does not consider publication bias. “NESR acknowledges that publication bias is important and is prevalent in nutrition research (as in other biomedical research). NESR considers potential for publication bias in the evidence synthesis process by considering the extensiveness of the search, and whether large and small studies were included in the review, in particular small studies with null findings” Source of funding is extracted and summarized in the data table	Partially implemented Slightly different than optimal, but likely to have minor impact on systematic review outcomes

Page 249 Cite

Suggested Citation:"Appendix E: Summary of Selected Systematic Review Practices." National Academies of Sciences, Engineering, and Medicine. 2022. Evaluating the Process to Develop the Dietary Guidelines for Americans, 2020-2025: A Midcourse Report. Washington, DC: The National Academies Press. doi: 10.17226/26406.

×

Practice	Committee-Identified Practices	NESR 2020 DGA	Implementation Impact^a
Evidence Synthesis	Two people working independently should assess the certainty of the body of evidence and reach a consensus view on any downgrading decisions AHRQ process includes five domains: study limitations, directness, consistency, precision, and reporting bias Cochrane uses the GRADE approach, which “involves 5 criteria to assess certainty of evidence: risk of bias, consistency of effect, imprecision, indirectness of effect, and publication bias” Use the online GRADEpro tool GRADE is the most widely used approach for summarizing confidence in effects of interventions by outcome across studies Includes justification of GRADE decisions to enhance transparency and makes the results more informative to the user	The Committee [DGAC] assigned a grade to each conclusion statement versus having two people working independently (best practice) NESR has grade informed predefined criteria, based on 5 grading elements that the committee used to evaluate and grade the strength of the evidence supporting each conclusion statement. The 5 grading elements are risk of bias, consistency, directness, precision, and generalizability of the evidence. Study design was also considered during the grading process DGAC/NESR does not use the GRADEpro tool, but rather employs its own grading rubric. “This ensures the final grade reflects consideration of all of the grading criteria and promotes consistency across systematic reviews, and allows for the Committee’s assessment of each element to be transparently documented”	Partially implemented Refinements group processes and descriptions could further enhance transparency and integrity The significant differences from GRADE is the absence of publication bias with major impact on systematic review outcomes if the NESR processes are not continually updated and adapted

Page 250 Cite

Suggested Citation:"Appendix E: Summary of Selected Systematic Review Practices." National Academies of Sciences, Engineering, and Medicine. 2022. Evaluating the Process to Develop the Dietary Guidelines for Americans, 2020-2025: A Midcourse Report. Washington, DC: The National Academies Press. doi: 10.17226/26406.

×

Practice	Committee-Identified Practices	NESR 2020 DGA	Implementation Impact^a
Creating Recommendations from Conclusion Statements	Description of formal consensus processes and how to effectively use the analytical framework to evaluate both benefits and harms and arrive at recommendations as well as the following statement	The NESR analysts drafted a description of the studies included in the systematic review to begin the process of synthesizing the evidence. This description included information about the study designs, sample sizes (i.e., baseline and analytic sample size, attrition) and subject characteristics (i.e., age, sex, race/ethnicity, socioeconomic status, and health status), the independent and dependent variables and their measurement methods, statistical adjustments, results, limitations, and funding sources The DGAC considered study design, key associations between the intervention/exposure and outcome(s) of interest in the systematic review question, along with key factors	Partially implemented

Page 251 Cite

Suggested Citation:"Appendix E: Summary of Selected Systematic Review Practices." National Academies of Sciences, Engineering, and Medicine. 2022. Evaluating the Process to Develop the Dietary Guidelines for Americans, 2020-2025: A Midcourse Report. Washington, DC: The National Academies Press. doi: 10.17226/26406.

×

Practice	Committee-Identified Practices	NESR 2020 DGA	Implementation Impact^a
	USPSTF uses a structured format for arriving at recommendations that involves six steps: assessing adequacy of evidence at the key question level, assessing adequacy of evidence at the linkage level, estimating the magnitude of benefit, harm, and net benefit and certainty of net benefit Rationale statement (summarizes benefits and harms, why outcomes are deemed important, assumptions about relationships, nature of evidence [quantity and consistency, and gaps] upholding the linkages)	addressed in grading the strength of the evidence (risk of bias, consistency, directness, precision, and generalizability) The DGAC then drafted a conclusion, which consisted of one or more summary statement(s) carefully constructed to answer the systematic review question The NESR methodology includes considerations for making decisions; however, it does not describe how the formal consensus process will be conducted within the committee The NESR approach does not address or include a process to estimate the magnitude of benefit, harm, and magnitude of net benefit	Refinements could positively impact the DGAC decision-making process if it more clearly balanced recommendation based on estimates of benefits and harms and magnitude of net benefit. May lead to different questions and topics for systematic reviews

Page 252 Cite

Suggested Citation:"Appendix E: Summary of Selected Systematic Review Practices." National Academies of Sciences, Engineering, and Medicine. 2022. Evaluating the Process to Develop the Dietary Guidelines for Americans, 2020-2025: A Midcourse Report. Washington, DC: The National Academies Press. doi: 10.17226/26406.

×

Practice	Committee-Identified Practices	NESR 2020 DGA	Implementation Impact^a
Updating Systematic Reviews	Cochrane generally expects updates within 5 years and specifies a process for updating reviews including decisions around whether the review question is still valid Cochrane concluded that “Adding new studies and new data can substantively change the findings of the review. Even where the new studies observe results consistent with the existing data, increasing the number of studies can improve precision of effect estimates, demonstrate wider applicability of the effect, or enable additional comparisons or subgroup analyses to be performed. The introduction of new review methods, such as updated risk of bias assessment tools or improved statistical analysis methods, can also change both the results and the certainty of the review’s findings.”	Decision regarding whether an existing systematic review would be used or updated was after the DGAC developed their protocol and analytical framework at the 5-year mark Existing NESR systematic reviews were compared to the new protocol developed by the DGAC to determine relevance using criteria included in the methodology manual Considerations for when and how to use existing systematic reviews are available, but not at the level of decision guides or algorithms Documentation varied between systematic review reports	Partially implemented Less detailed descriptions are provided Major improvements are possible in describing, implementing, and documenting decision-making processes for using/updating existing systematic reviews Unclear if there would be change in ROB ratings, but if there were significant differences it could impact systematic review outcomes if the they are deciding between two grades

Page 253 Cite

Suggested Citation:"Appendix E: Summary of Selected Systematic Review Practices." National Academies of Sciences, Engineering, and Medicine. 2022. Evaluating the Process to Develop the Dietary Guidelines for Americans, 2020-2025: A Midcourse Report. Washington, DC: The National Academies Press. doi: 10.17226/26406.

×

Practice	Committee-Identified Practices	NESR 2020 DGA	Implementation Impact^a
		When formal updates were performed the process included a new search, data extraction, ROB assessment, and data synthesis. Neither of the two other options for data synthesis (e.g., assessing new evidence as it related to existing conclusions or a separate synthesis of new evidence) would be considered an update of a systematic review Risk of bias assessment was not re-accomplished when existing systematic reviews were used and documentation was not presented that showed that the older tool would detect risk of bias the same as the newer tool adopted The 2020 methodology manual stated, “In addition, the complete systematic review update was documented, including details about the protocol and methodology, the full description and synthesis of the evidence, and conclusion statements and grades”

Page 254 Cite

Suggested Citation:"Appendix E: Summary of Selected Systematic Review Practices." National Academies of Sciences, Engineering, and Medicine. 2022. Evaluating the Process to Develop the Dietary Guidelines for Americans, 2020-2025: A Midcourse Report. Washington, DC: The National Academies Press. doi: 10.17226/26406.

×

Practice	Committee-Identified Practices	NESR 2020 DGA	Implementation Impact^a
Peer Review^b	The leading groups AHRQ, USPSTF, and IOM have very defined and rigorous review process involving external reviewers of various expertise (methodology, content, and consumer) Peer review occurs at both the protocol stage and for the completed review for Cochrane CPQ requires that external reviewer comments are documented and a written rationale provided for modifying (or not modifying) the review in response to reviewer comments	Search strategies were peer reviewed internally by a second librarian prior to conducting the search Each completed systematic review was then reviewed by two federal scientists (who self-identified their interests) after the DGAC draft conclusion statements were formulated and discussed at full committee meetings. NESR then reviewed comments and suggested proposed edits to the appropriate subcommittee and sent responses back to peer reviewers External peer review of full systematic review protocol was not completed prior to conducting the systematic review	Partially implemented True external peer review of the full systematic review protocol a priori could increase rigor and transparency The rigor of this current process is undetermined

Page 255 Cite

Suggested Citation:"Appendix E: Summary of Selected Systematic Review Practices." National Academies of Sciences, Engineering, and Medicine. 2022. Evaluating the Process to Develop the Dietary Guidelines for Americans, 2020-2025: A Midcourse Report. Washington, DC: The National Academies Press. doi: 10.17226/26406.

×

Practice	Committee-Identified Practices	NESR 2020 DGA	Implementation Impact^a
		Expertise of federal reviewers was not fully described and it is unknown if they included those with content expertise in systematic review methodology, content expertise and consumers Peer reviewer comments were not made available to committee, documentation is unknown

NOTES: AHRQ = Agency for Healthcare Research and Quality; COI = conflict of interest; DGAC = Dietary Guidelines Advisory Committee; HHS = U.S. Department of Health and Human Services; IOM = Institute of Medicine; NESR = Nutrition Evidence Systematic Review; PICO = population, intervention, control, outcomes; PROSPERO = Prospective Register of Systematic Reviews; ROB = risk of bias; USDA = U.S. Department of Agriculture; USPSTF = U.S. Preventive Services Task Force. The practices for the Pregnancy and Birth to 24 Months systematic reviews was not included in this table.

^a Implementation refers to alignment between the committee-identified practices and NESR practices. If the category was listed as partially implemented, the level of difference is listed in the last column across from items that were different; no annotation is made for areas that were consistent.

^b Peer review affects multiple categories of activities; however, it is only described once. SOURCES: Ayorinde et al., 2020; Cochrane, 2021; DGAC, 2020; IOM, 2011; NESR, 2021; USPSTF, 2018; VanderWeele and Ding, 2017.

REFERENCES

Ayorinde, A. A., I. Williams, R. Mannion, F. Song, M. Skrybant, R. J. Lilford, and Y. Chen. 2020. Assessment of publication bias and outcome reporting bias in systematic reviews of health services and delivery research: A metaepidemiological study. PLoS ONE 15(1):e0227580. https://doi.org/10.1371/journal.pone.0227580.

Cochrane. 2021 Cochrane Handbook for Systematic Reviews of Interventions version 6.2 (updated February 2021). www.training.cochrane.org/handbook (accessed November 10, 2021).

DGAC (Dietary Guidelines Advisory Committee). 2020. Scientific report of the 2020 Dietary Guidelines Advisory Committee: Advisory report to the Secretary of Agriculture and the Secretary of Health and Human Services. Washington, DC: Agricultural Research Service, U.S. Department of Agriculture.

Page 256 Cite

Suggested Citation:"Appendix E: Summary of Selected Systematic Review Practices." National Academies of Sciences, Engineering, and Medicine. 2022. Evaluating the Process to Develop the Dietary Guidelines for Americans, 2020-2025: A Midcourse Report. Washington, DC: The National Academies Press. doi: 10.17226/26406.

×

IOM (Institute of Medicine). 2011. Clinical practice guidelines we can trust. Washington, DC: The National Academies Press. https://doi.org/10.17226/13058.

NASEM (National Academies of Sciences, Engineering, and Medicine). 2017. Redesigning the process for establishing the Dietary Guidelines for Americans. Washington, DC: The National Academies Press. https://doi.org/10.17226/24883.

NESR (Nutrition Evidence Synthesis). 2021. Roles and responsibilities. https://nesr.usda.gov/roles-and-responsibilities (assessed October 25, 2021).

USPSTF (U.S. Preventive Services Task Force). 2017. Appendix V. Work plan/research plan template. https://www.uspreventiveservicestaskforce.org/uspstf/about-uspstf/methods-and-processes/procedure-manual/procedure-manual-appendix-v-work-planresearch-plan-template (accessed October 27, 2021).

VanderWeele, T. J., and P. Ding. 2017. Sensitivity analysis in observational research: Introducing the e-value. Annals of Internal Medicine 167(4):268–274. https://doi.org/10.7326/M16-2607.