Page 257 Cite

Suggested Citation:"Appendix F: Selected Systematic Review Methodologies 2015 Versus 2020." National Academies of Sciences, Engineering, and Medicine. 2022. Evaluating the Process to Develop the Dietary Guidelines for Americans, 2020-2025: A Midcourse Report. Washington, DC: The National Academies Press. doi: 10.17226/26406.

×

Appendix F

Selected Systematic Review Methodologies 2015 Versus 2020¹

TABLE F-1 Comparison of Selected Systematic Review Methodologies 2015 Versus 2020

	2015	2020
Formulation of research questions to be addressed in SR	“DGAC members developed the SR questions and worked with NEL staff to complete SR.” “DGAC made all substantive decisions required during the process.”	TEP used for B-24, but not other sub-comms

___________________

¹ The title of this appendix and Table F-1 were modified after release of a prepublication version of the report to the sponsor to clarify that this is not a comprehensive list of methodological comparisons.

Page 258 Cite

Suggested Citation:"Appendix F: Selected Systematic Review Methodologies 2015 Versus 2020." National Academies of Sciences, Engineering, and Medicine. 2022. Evaluating the Process to Develop the Dietary Guidelines for Americans, 2020-2025: A Midcourse Report. Washington, DC: The National Academies Press. doi: 10.17226/26406.

×

	2015	2020
SR process (overall)	6-step process: Develop sys review questions and analytic framework Search, screen Extract data and risk of bias Synthesize evidence Develop conclusion and grade evidence Identify research recommendations	6-step process very similar to 2015 process, but there is considerably more detail on each step: “Committee established their protocols before any evidence was reviewed and synthesized.” “All SR protocols were posted online to increase transparency.” “Protocols presented and discussed at full Comm meetings.” “The Comm established inclusion and exclusion criteria to provide an objective, consistent, and transparent framework.” “NESR analysts worked jointly with the Comm members to establish inclusion and exclusion criteria that were tailored to the SR question addressed.” Specific mention that population in “PICO” was “applicable to U.S. pop, including those who are healthy and/or those at risk of chronic disease.” Also specified that NESR included studies that enroll some subjects with a disease Excluded studies based solely on individuals with disease, as this would be a clinical practice guideline

Page 259 Cite

Suggested Citation:"Appendix F: Selected Systematic Review Methodologies 2015 Versus 2020." National Academies of Sciences, Engineering, and Medicine. 2022. Evaluating the Process to Develop the Dietary Guidelines for Americans, 2020-2025: A Midcourse Report. Washington, DC: The National Academies Press. doi: 10.17226/26406.

×

	2015	2020
Development of questions and analytic framework	“DGAC identified, refined, and prioritized the most relevant topics and then developed clearly focused SR questions that were appropriate in scope, reflected the state of the science, and targeted important policy relevant to public health issue(s). Once topics and systematic review questions were generated, DGAC developed an analytical framework for each topic in accordance with NEL methodology.”	“For the first time, USDA and HHS identified topics and scientific questions to be examined by the 2020 committee before establishing the committee. The departments added this step for a number of reasons, including to promote a deliberate and transparent process, identify expertise needed on the committee, help manage resources, and ensure the scientific review conducted by the committee would address federal nutrition policy and program needs.” “The committee developed an analytic framework for each systematic review question…. The analytic framework serves as the foundation for the rest of the systematic review process, and informs the inclusion/exclusion criteria and literature search strategy, data extraction and risk of bias assessments, and the strategy for synthesizing the evidence to develop and grade conclusion statements.”

Page 260 Cite

Suggested Citation:"Appendix F: Selected Systematic Review Methodologies 2015 Versus 2020." National Academies of Sciences, Engineering, and Medicine. 2022. Evaluating the Process to Develop the Dietary Guidelines for Americans, 2020-2025: A Midcourse Report. Washington, DC: The National Academies Press. doi: 10.17226/26406.

×

	2015	2020
Search and screening procedures	Search: “Searching, screening, and selecting scientific literature was an iterative process that sought to identify the most complete and relevant body of evidence to answer a SR question. This process was guided by inclusion and exclusion criteria determined a priori by DGAC. The NEL librarians created and implemented search strategies that included appropriate databases and search terms to identify literature to answer each SR question.” Screening: “The results of the literature search were screened by the NEL librarians and staff in a dual, stepwise manner, beginning with titles, followed by abstracts, and then full-text articles, to determine which articles met the criteria for inclusion in the review.” “DGAC provided direction throughout this process to ensure that the inclusion and exclusion criteria were applied appropriately and the final list of included articles was complete and captured all research available to answer a SR question. Each step of the process also was documented to ensure transparency and reproducibility.”	Search: “The committee established inclusion and exclusion criteria to provide an objective, consistent, and transparent framework for determining which articles to include in each systematic review. These criteria were developed before any studies were reviewed to guide selection of the most relevant and appropriate body of evidence for each systematic review question. Additionally, these criteria were framed to increase the utility of the systematic review to inform U.S. federal policy and programs. To minimize bias, revisions to the criteria after studies had been reviewed were discouraged. Any revisions to the criteria that occurred were documented with dates and rationales.” Screening: NESR analysts worked jointly with the committee members to establish inclusion and exclusion criteria that were tailored to the systematic review question addressed. Considering the perspectives of both NESR and the committee members helped ensure that the evidence reviewed was Applicable to the U.S. population, including those who are healthy and/or those at risk of chronic disease, Relevant to public health nutrition policies and programs, and Rigorous from a scientific perspective. “Two NESR analysts independently screened all search results.” Documentation of process also well articulated in methodology. Excluded articles are shown in tabular format that is publicly available on NESR website (full text) or archived by NESR (title or abstract stage).

Page 261 Cite

Suggested Citation:"Appendix F: Selected Systematic Review Methodologies 2015 Versus 2020." National Academies of Sciences, Engineering, and Medicine. 2022. Evaluating the Process to Develop the Dietary Guidelines for Americans, 2020-2025: A Midcourse Report. Washington, DC: The National Academies Press. doi: 10.17226/26406.

×

	2015	2020
	“The NEL established and DGAC approved standard inclusion and exclusion criteria to promote consistency across reviews and ensure that the evidence being considered in NEL SRs was most relevant to the U.S. population.”
Data extraction and ROB	Data extraction: “NEL abstractors are National Service Volunteers from across the United States with advanced degrees in nutrition or a related field who were trained to review individual research articles included in NEL systematic reviews (a list of the Volunteers is included in Appendix E-10: Dietary Guidelines Advisory Committee Report, Acknowledgments).” ROB: “The risk of bias (i.e., internal validity) for each study was assessed using the NEL Bias Assessment Tool (BAT)” NEL BAT considered 4 types of bias: Selection, performance, detection, attrition	Data extraction: Data were extracted and verified by two NESR analysts. Both analysts used data extraction form within^a DistillerSR software. “Types of data typically extracted include study design, sample size (i.e., baseline and analytic sample size, attrition) and participant characteristics (i.e., age, sex, race/ethnicity, socioeconomic status, and health status), the independent and dependent variables and their measurement methods, statistical adjustments, results, limitations, and funding sources.” ROB: RCTs: Cochrane ROB tool for randomized trials; RoB2.0, August 2016 Non-RCT: ROBINS-I Observational: RoB-NObs, which was created by NESR by modifying ROBINS-I ROB performed independently by two NESR analysts

Page 262 Cite

Suggested Citation:"Appendix F: Selected Systematic Review Methodologies 2015 Versus 2020." National Academies of Sciences, Engineering, and Medicine. 2022. Evaluating the Process to Develop the Dietary Guidelines for Americans, 2020-2025: A Midcourse Report. Washington, DC: The National Academies Press. doi: 10.17226/26406.

×

	2015	2020
Evidence synthesis	“To facilitate the DGAC’s review and analysis of the evidence, staff prepared a “Key Trends” template for each SR question. This document was customized for each question and included questions related to major trends, key observations, themes for conclusion statements and key findings. It also addressed methodological problems or limitations, magnitude of effect, generalizability of results, and research recommendations. DGAC members used the description of the evidence, along with the full data extraction grid, and full-text manuscripts to complete the “Key Trends” questions.”	“NESR analysts drafted a description of the studies included in the systematic review to begin the process of synthesizing the evidence … next the committee synthesized the evidence.” Then, “The committee provided its feedback to the NESR analysts, who drafted text that synthesized the evidence.”
Development of conclusions and grading	“DGAC used predefined criteria to evaluate and grade the strength of available evidence supporting each conclusion statement.” 5 domains: ROB Consistency Directness Precision Generalizability	The committee graded the evidence. 5 domains: ROB Consistency Directness Precision Generalizability Study design also was a critical consideration in the process of grading. NESR does not consider publication bias as a grading criterion.^b

Page 263 Cite

Suggested Citation:"Appendix F: Selected Systematic Review Methodologies 2015 Versus 2020." National Academies of Sciences, Engineering, and Medicine. 2022. Evaluating the Process to Develop the Dietary Guidelines for Americans, 2020-2025: A Midcourse Report. Washington, DC: The National Academies Press. doi: 10.17226/26406.

×

	2015	2020
Identification of research recommendations	“Based on the existing body of evidence, research gaps, and limitations, the DGAC formulated several research recommendations that could advance knowledge related to the SR question. These recommendations can be used to inform research agendas and further inform policymakers.”^c	“The committee identified and documented research gaps and methodological limitations throughout the systematic review process. These gaps and limitations were used to develop research recommendations that describe the research, data, and methodological advances that are needed to strengthen the body of evidence on a particular topic. Rationales for the necessity of additional or stronger research also may have been provided with the research recommendations.”

Page 264 Cite

Suggested Citation:"Appendix F: Selected Systematic Review Methodologies 2015 Versus 2020." National Academies of Sciences, Engineering, and Medicine. 2022. Evaluating the Process to Develop the Dietary Guidelines for Americans, 2020-2025: A Midcourse Report. Washington, DC: The National Academies Press. doi: 10.17226/26406.

×

	2015	2020
Use of existing reviews	“When committee members were aware of high-quality existing reports that addressed their question(s), they decided a priori to use existing report(s)…. A literature search was then conducted to identify other existing reports to augment the existing report(s) identified by the committee.” “In other cases, the committee was not aware of any existing reports and intended to conduct a de novo NEL SR. However, as part of the duplication assessment step of the NEL process, one or more existing SRs or MA were identified that addressed the question that led to the committee deciding to proceed using existing SRs/MA rather than complete an independent review of the primary literature.” “Finally, for some questions, the committee used existing reports as the primary source of evidence to answer a question, but chose to update one or more of those existing reports using the NEL process to identify and review studies that had been published after the completion of the literature search for the existing report(s).”	“If one or more relevant existing systematic reviews were identified, a determination was made as to whether the existing review reflected the current state of science on the topic, or whether reviewing newly published evidence would likely result in changes to the conclusion statement and/or grade, thus warranting the investment of time and resources in a full systematic review update.” This determination was based on date range of previous review, GRADE of previous review, DGAC knowledge of the field, systematic evidence scan for new information. “If one or more relevant existing reviews were determined to reflect the current state of science, the committee documented their rationale, and used the existing NESR review(s) to answer the systematic review question, carrying forward the conclusion(s) and grade(s) from the review(s). If the relevant existing review(s) were determined to be out of date, the review was updated using the methods described below.” “NESR’s systematic review methodology (described above in “Synthesize Evidence, Develop Conclusion Statements, Grade the Evidence, and Identify Research Recommendations”) for developing conclusion statements and grading the strength of the evidence was applied. In addition, the complete systematic review update was documented, including details about the protocol and methodology, the full description and synthesis of the evidence, and conclusion statements and grades.”

Page 265 Cite

Suggested Citation:"Appendix F: Selected Systematic Review Methodologies 2015 Versus 2020." National Academies of Sciences, Engineering, and Medicine. 2022. Evaluating the Process to Develop the Dietary Guidelines for Americans, 2020-2025: A Midcourse Report. Washington, DC: The National Academies Press. doi: 10.17226/26406.

×

	2015	2020
	“In cases where DGAC used an existing report with its own formally graded conclusions, the committee acknowledged the grade assigned within that existing report, and then assigned a DGAC grade that was the closest equivalent to the grade assigned in the existing report.”
Peer Review	NEL librarians peer review each other’s search strategies	Described in detail in 2020 document and was new to 2020 process. “Each systematic review was peer reviewed by two federal scientists … they were asked to self-identify their research questions of interest.” “Each reviewer was asked to provide a personal expert review.” “Process was anonymous and confidential.” “Peer review occurred after draft conclusion statements were discussed by the full Comm at meetings 4 and 5.” “Following peer review, the ARS Co-exec Secretary shared peer-review comments with the NESR lead. NESR staff then reviewed the comments, addressed editorial comments, and proposed edits to the relevant subcommittee … revisions were then made based on subcomm discussion.”

NOTE: ARS = Agricultural Research Service; DGAC = Dietary Guidelines Advisory Committee; GRADE = Grading of Recommendations Assessment, Development and Evaluation; HHS = U.S. Department of Health and Human Services; MA = meta-analysis; NEL = Nutrition Evidence Library; NEL BAT = Nutrition Evidence Library Bias Assessment Tool; NESR = Nutrition Evidence Systematic Review; PICO = population, intervention/exposure, comparator, and outcome of interest; RCT = randomized controlled trial; ROB = risk of bias; SR = systematic review; TEP = technical expert panel; USDA = U.S. Department of Agriculture.

^a Text modified after delivery of the prepublication report to the sponsor to clarify that one analyst extracted the data and the second verified the data. Also to clarify that the extraction form was within DistillerSR.

^b Text modified after delivery of the prepublication report to the sponsor to clarify that NESR does not consider publication bias as a grading criterion.

^c Text modified after delivery of the prepublication report to the sponsors to add information from the 2015 DGAC Scientific Report.

Page 266 Cite

Suggested Citation:"Appendix F: Selected Systematic Review Methodologies 2015 Versus 2020." National Academies of Sciences, Engineering, and Medicine. 2022. Evaluating the Process to Develop the Dietary Guidelines for Americans, 2020-2025: A Midcourse Report. Washington, DC: The National Academies Press. doi: 10.17226/26406.

×