Summary

In 1984, the U.S. Department of Health and Human Services (HHS) assistant secretary Edward N. Brandt commissioned the U.S. Preventive Services Task Force (USPSTF) to “improve the health of people nationwide by making evidence-based recommendations about clinical preventive services” (USPSTF, n.d.). The USPSTF was assembled as an effort to continue and expand the innovative work of the Canadian Task Force on the Periodic Health Examination (CTFPHE) within the United States (CTFPHE, 1979; McGinnis and Woolf, 1990). The CTFPHE laid out the basic elements of clinical preventive evidence assessment as they inform clinical practice guidelines (CPGs). These included comprehensively reviewing the literature; using prioritization methods for assessing individual interventions, including determining the burden of suffering and ability to measure outcomes; understanding the risks and benefits of an intervention; rating the effectiveness of interventions; grading the level of evidence for a recommendation; and defining a research agenda to address unanswered questions (CTFPHE, 1979). The USPSTF is codified in the Healthcare Research and Quality Act, which states that the USPSTF

However, despite having the authority to do so, the USPSTF does not evaluate evidence related to cost-effectiveness when issuing a grade for a topic:

The USPSTF recommendations guide clinical practice and garner broad attention, as evidenced by the coverage in the lay and medical literature of the recent change in guidance regarding aspirin use to prevent cardiovascular disease (Mahase, 2021; Rabin, 2021; USPSTF, 2021a).

The USPSTF’s process of selecting a topic for review through publishing the final recommendation statement takes 3 years on average (Borsky, 2021). The USPSTF works with one of the Evidence-based Practice Centers (EPCs) supported by the Agency for Healthcare Research and Quality (AHRQ) to develop a draft research plan, which is posted for public comment. After the USPSTF and the EPC review and revise as needed, the final plan is posted online. The EPC conducts the evidence review according to the research plan, and the USPSTF begins assessing the EPC’s draft review. The USPSTF then drafts a recommendation statement, posts it online for, and reviews any public comments, and the EPC finalizes its evidence review. The USPSTF then finalizes its recommendation statement, and both are posted online and, as of 2016, published in a journal of the JAMA Network (Bauchner et al., 2016). The USPSTF also considers previously graded topics for updates starting 2–3 years after they are first published, soliciting stakeholder input and examining the likelihood that enough new evidence has been developed that the recommendation for practice may be updated.

The USPSTF assessment of the overall evidence for a clinical preventive service requires two key determinations: assessment of the degree

___________________

¹ 106th U.S. Congress, First Session. Public Law 106-129, Section 915. Healthcare Research and Quality Act of 1999. https://www.ahrq.gov/sites/default/files/wysiwyg/policymakers/hrqa99.pdf (accessed July 20, 2021).

of certainty about the net benefit (benefits minus harms) and assessment of the magnitude of net benefit for the population for whom the service is intended. Currently, the USPSTF recommends offering or providing preventive services with grades A or B; selectively offering or providing a preventive service with a grade C; and not offering or providing a preventive service with a grade D. While the recommendations for or against a clinical service receive a grade, some topics are given I statements, indicating that the USPSTF has not found sufficient evidence to make a recommendation at all. In 2009, the USPSTF announced that it would use the term “I statement” instead of “I recommendation” to reduce confusion among clinicians about the fact that an I statement was not a recommendation (USPSTF, 2017). Topics can receive an I statement instead of a graded recommendation due to lack of evidence, a lack of good-quality evidence, or an abundance of conflicting evidence. As of publication of this report, 53 of 86 published topics have I statements either for a specific subpopulation or for recommending for or against the service at all. The USPSTF has suggested evidence gaps or research needs related to the statement topic since its first report, though a 2019 methods update formalized a Research Needs and Gaps section for each statement. Both I statements and letter grade recommendations include research needs; the details of those sections vary widely in their specificity and scope. Addressing evidence gaps is important for changing an I statement to a letter grade recommendation or, as in the case of the recent change in the draft recommendation regarding aspirin use to prevent cardiovascular disease, changing a letter grade recommendation, in this example from a C recommendation to a D for some populations (USPSTF, 2021a).

STATEMENT OF TASK

An ad hoc committee of the National Academies of Sciences, Engineering, and Medicine (the National Academies) was convened to examine current issues in identifying and filling evidence gaps in the USPSTF clinical prevention recommendations. The committee was specifically tasked with developing a taxonomy of evidence gaps. The complete Statement of Task can be found in Chapter 1. At the first meeting of the committee, David Murray, the director of the Office of Disease Prevention (ODP) at the National Institutes of Health (NIH), characterized the challenges in addressing evidence gaps in two ways: nonspecific descriptions (e.g., “high-quality studies on the potential harms of screening and treatment are needed”) and discussion of evidence gaps in various places. Expressly stated was the hope to link evidence gaps with USPSTF criteria for study inclusion and coordinated effort to describe and communicate evidence gaps so that they are easier to identify and address (Murray,

2020). Thus, AHRQ and ODP expressed a desire for a more systematic way for the USPSTF and NIH to work together to describe research needs and make them more actionable. Both AHRQ and ODP representatives stressed that the audience for the report is not limited to the USPSTF, AHRQ, or NIH—rather, it is hoped that the report would be helpful to CPG developers, research funders, and researchers.

Committee Process

The committee held four open sessions to solicit input on the Statement of Task, experiences of others in developing taxonomies, and challenges for NIH and researchers to address I statements. The committee reviewed literature on taxonomies and developed a taxonomy using current I statements as test candidates for the taxonomy. As part of its work, the committee developed a workflow to show how to use the taxonomy to develop a research agenda.

TAXONOMY

Taxonomies come in various forms, but they all share certain characteristics. The general purpose of any taxonomy is to bring order to a collection of information by organizing and systematizing it, often with a fixed vocabulary and clearly defined structure. A well-designed taxonomy makes it easier to see the big picture and to think about and work with the mass of information that it has imposed order on. The taxonomy adopted by the committee provides a systematic way of describing evidence gaps with a clear list of categories and a controlled vocabulary for consistency across evidence reviews. Furthermore, by offering a list of the different classes of evidence gaps that may play a role in analyzing a preventive service, the taxonomy and an accompanying workflow offer a road map for researchers and funders.

The USPSTF analytic framework begins with the specification of a population of interest and then examines the evidence for connections between actions (screening, preventive medications, or behavioral interventions) and outcomes (early detection, decreased morbidity or mortality, adverse effects) (see Figure S-1). Each line in the figure between an action and an outcome is associated with a question of interest: Does screening lead to an early detection of the target condition? Does screening lead to adverse effects? Do early detection and treatment lead to reduced morbidity or mortality? Do early detection and treatment lead to an intermediate outcome that may in turn be related to a decrease in morbidity or mortality? Do early detection and treatment lead to adverse effects? When the USPSTF examines the evidence concerning the effectiveness of

a particular preventive service, it uses these particular questions to guide its analysis and to frame the reporting of its results. It should also be noted that while this generic framework focuses specifically on screenings, the USPSTF uses this approach to guide its recommendations for preventive medications and behavioral counseling as well (see Chapter 4 for more varied and complex examples of the USPSTF analytic frameworks).

The committee realized that these various pieces of the analytic framework—such as the population under consideration, early detection, intermediate outcomes, effectiveness, harms—could serve as a good starting point for a taxonomy for characterizing the various research needs and other issues associated with a given preventive service. However, they determined that a taxonomy consisting only of elements from the analytic framework would be incomplete, and thus they added other components.

One set of missing elements could be classified as foundational issues, such as a lack of clarity on nomenclature or on the specificity of a screening tool. Another set of issues not touched upon in the USPSTF analytic framework involves the dissemination and implementation (D&I) of a recommended preventive service. Traditionally this has not been something that the USPSTF has concerned itself with, but the committee’s discussions led it to conclude that it was crucial to include evidence gaps related to D&I in the taxonomy.

Thus, the committee developed a taxonomy that covers three different facets of prevention-related evidence gaps: foundational issues, clinical and epidemiologic evidence as outlined in the USPSTF’s analytic framework, and D&I. The committee also wanted the taxonomy to be useful for outlining a prioritized research agenda. Thus, the committee offers a

taxonomy for use in prioritizing research gaps as well as a way to specify the types of studies needed to fill those gaps. All terms are expanded and defined in Chapter 4, which also includes two examples of applying the evidence gap taxonomy to two recent I statements issued by the USPSTF. See Figure S-2 for the basic workflow to use with the taxonomy terms; the complete workflow can be found in Chapter 4. In addition to the taxonomy itself, a workflow guiding users through using each facet of the taxonomy can be found in Chapter 4.

Per the workflow, the first step is to review the evidence concerning a particular preventive service in order to develop a recommendation statement. As described in the recommendations, the evidence base may be insufficient to answer certain questions, and in those cases, it will be necessary to gather additional information from stakeholders.

Once the review of evidence has been completed, the next step is to use the taxonomy to identify the gaps that exist in the evidence. These gaps will be placed in one of three facets: foundational issues, analytic framework evidence gaps, and D&I evidence gaps. Generally speaking, any gaps that would leave a topic with an I statement would be put into one of the first two facets; that is, those two facets cover the evidence gaps that have traditionally been considered by the USPSTF.

After identifying and characterizing any evidence gaps using the first three facets of the taxonomy, users will develop a research agenda using two additional facets. Users will prioritize evidence gaps within the topic being reviewed using criteria from the prioritization facet, clarifying which evidence gaps are most important to address, so limited research funding can be put to optimal use. Users will then use the study specification facet to outline the important aspects of the key research studies needed to address the high-priority evidence gaps. The study specification facet is an expansion of the PICOTS (population, intervention, comparison, outcome, timing, setting) framework.

Not every interested party will necessarily carry out every step in the framework. Some might, for example, choose to focus on the traditional USPSTF concerns and apply only the first two parts of the taxonomy—covering foundational issues and analytic framework issues—and not address D&I. Others, such as funders, might work with a collection of evidence gaps generated by someone else and apply their own prioritization criteria to develop a research agenda that reflects their own interests.

RECOMMENDATIONS

The committee makes eight recommendations in three categories: using the taxonomy, fostering clinical prevention research, and advancing the work of the USPSTF. The committee’s rationale follows each recommendation.

Using the Taxonomy

The committee has provided a taxonomy for use in systematically describing evidence gaps in three facets: foundational research, the analytic framework used by the USPSTF, and D&I. The committee also outlines two additional facets to guide the process for developing a research agenda: prioritizing evidence gaps and outlining key research studies.

The committee envisions this as the shared responsibility of the USPSTF with assistance from AHRQ and ODP and other stakeholders, including but not limited to the USPSTF partner organizations and other research funders. The committee developed Recommendations 1 and 2 with the work of the USPSTF in mind but hopes that other clinical prevention guideline developers find this useful and adopt it. Recommendation 3 is directed at AHRQ and NIH to maintain and update the taxonomy.

Recommendation 1: For each of its recommendation statements, the U.S. Preventive Services Task Force should use the Clinical Prevention Research Taxonomy to identify and describe evidence gaps.

The USPSTF will likely and understandably be most inclined to focus their efforts on the analytic framework evidence gaps, as it is already a part of their process. These evidence gaps emerge from the evidence review provided by an EPC and the USPSTF deliberations about the grade recommendation (AHRQ, 2018). The USPSTF may also find the foundational evidence gaps facet useful, particularly because some recommendation and I statement Research Needs and Gaps sections name evidence gaps of that type, though not by name. Framing the research needs around the taxonomy terms will impart consistency in the USPSTF efforts to describe research needs. The committee envisions that AHRQ will direct EPCs to conduct reviews using terminology consistent with the taxonomy.

Despite the fact that the USPSTF has not had D&I as part of its remit, it recognizes the importance of D&I to clinical prevention research. Additionally, AHRQ has a robust portfolio on implementation research. Recently, USPSTF members and AHRQ staff wrote, “Although the use of implementation research is not within the scope of the USPSTF’s deliberations, it notes that there are critical questions about how to best implement recommended clinical preventive services in primary care practices. Additional implementation and translational research will increase the value of the USPSTF’s work and would be helpful in its deliberations” (Mabry-Hernandez et al., 2018, p. S98).

Because the committee was not tasked with recommending how the USPSTF determines its recommendations, it does not intend that D&I considerations would influence the letter grade recommendations issued by the USPSTF. However, it is clear from AHRQ’s and ODP’s request to the National Academies and the committee in open session that the committee was encouraged to address the clinical prevention community writ large and thus has proposed a D&I facet as part of the larger taxonomy of evidence gaps in clinical prevention (NASEM, 2020).

There are several ways the work related to D&I could be accomplished. The committee hopes that eventually the USPSTF will assume

responsibility for D&I work; however, the committee understands that the USPSTF carries a large workload and that this requires time and resources to be fully implemented. In the short term, partner organizations and staff at AHRQ and NIH could provide pivotal support and input. D&I evidence gaps or questions might be similar for certain types of interventions. Attention to D&I might start with an effort to outline common themes or challenges to facilitate the work on specific I statements.

Recommendation 2: For each recommendation and I statement, the U.S. Preventive Services Task Force (USPSTF) should indicate high-priority evidence gaps. Staff from the Office of Disease Prevention (ODP) in the National Institutes of Health (NIH) and from the Agency for Healthcare Research and Quality (AHRQ) should broadly outline research that could address those priority gaps and be available to help funders, whether NIH or others, develop a research agenda to address them. The USPSTF and staff from ODP and AHRQ should consult partner organizations and stakeholders as needed.

The committee was urged to facilitate prioritizing the evidence gaps. In order to make those priorities evident and actionable by funders and researchers, the committee outlines two steps in the development of a research agenda: setting priorities across all evidence gaps and outlining key studies needed to address each high-priority evidence gap.

Given the demands on USPSTF members, the committee suggests that the development of a research agenda could be done by NIH and AHRQ staff in consultation with USPSTF members and stakeholders. If there are concerns about this process holding up issuance of a USPSTF recommendation statement, the research agenda could be published separately and subsequent to publication of the Recommendation statement itself. USPSTF members, however, are best suited to prioritize evidence gaps, especially regarding foundational research and analytic framework gaps. A structured approach to prioritization would be an important enhancement of the committee’s work. Until that time, those prioritizing the evidence gaps can use the prioritization facet qualitatively or selectively. The USPSTF might initially focus on centrality or equity.

One of the frustrations identified by stakeholders is that the USPSTF research needs have not usually been specific enough with regard to some key elements of study design, such as whether a randomized controlled trial is required. As described in Chapter 4, the committee provides a taxonomy of study specifications based on an enhancement of the PICOTS framework for use by whomever completes the research agenda. The

enhancements include aggregability, design considerations, and potential funders and funding mechanisms. The committee urges that the research agenda not be overly prescriptive, so as not to limit researcher creativity or funders’ expectations of research proposals.

Recommendation 3: The National Institutes of Health and the Agency for Healthcare Research and Quality should make the taxonomy accessible on their websites and integrate the taxonomy terms and phrases in their relevant publications, including but not limited to U.S. Preventive Services Task Force recommendation statements, funding announcements, and grant and contract awards.

Consistent use of these metadata will provide a common language to stakeholders and encourage wider adoption of the taxonomy. Applying the taxonomy terms in publications linking USPSTF recommendations or I statements with funding opportunity announcements will be crucial for making evident the impact of the research agenda laid out by the USPSTF. In order to ensure relevance and usability, the taxonomy and workflow will need to be maintained, updated, and evaluated.

Fostering Clinical Prevention Research

The committee was asked to propose ways for prevention research funders and guideline developers to enhance their partnership to accelerate research to close important gaps in prevention. A robust and relevant clinical prevention research agenda requires funding, funding requests that are clear and clearly linked to USPSTF research needs, and appropriate mechanisms for and oversight of funded research. As such, the committee provides recommendations regarding the funding and commissioning of clinical prevention research.

Recommendation 4: Funders, in particular the National Institutes of Health and the Patient-Centered Outcomes Research Institute, should set aside funding to address high-priority evidence gaps identified by the U.S. Preventive Services Task Force.

In order to act in a timely manner on a research agenda for clinical prevention developed by the USPSTF, AHRQ staff and ODP and other NIH staff, EPCs, and other stakeholders, funding, especially dedicated funding, will be needed. NIH and stakeholders can work with Congress and HHS leadership to identify appropriate funding levels and secure suf-

ficient funding for this area of need. Many research questions may require trans-institute collaboration, and these should be actively encouraged.

The committee acknowledges that calls for increased funding for prevention research have been made by others in the past and not always successfully. However, NIH’s budget is approximately $41.7 billion, and in 2016 only 20 percent of that was, at best, spent on prevention research, broadly defined (Murray, 2017; NIH, 2020). The amount devoted to clinical prevention research of direct relevance to the USPSTF is undoubtedly much less.

Recommendation 5: The National Institutes of Health staff developing funding announcements for research related to high-priority evidence gaps should consult with the Office of Disease Prevention and the Agency for Healthcare Research and Quality to ensure appropriateness of support mechanisms and fidelity to the research specifications and should include a link to the research needs document on the U.S. Preventive Services Task Force website.

Program announcements, notices of special interest, requests for applications or proposals, and research opportunity announcements related to high-priority research needs of importance to the USPSTF should be as explicit as possible to ensure usefulness to the USPSTF and other guideline bodies. AHRQ and ODP staff could serve as an important resource in the crafting of the requests and notices. Reviewers of applications and proposals could be encouraged to study the research needs specification from the USPSTF. Study sections dedicated to these applications could ensure that study section members are oriented to the needs of the USPSTF.

Recommendation 6: The National Institutes of Health and other funders addressing high-priority clinical prevention evidence gaps identified by the U.S. Preventive Services Task Force should use funding mechanisms and processes that can assure that research is conducted expediently, efficiently, and with fidelity to the specified research needs, rather than waiting for such research needs to be addressed predominantly through investigator-initiated grant mechanisms.

A contract, or other directive funding mechanism increases the likelihood that studies are of sufficient design to allow for aggregating studies; to demonstrate an effect, if the intervention indeed is effective; and to meet the methodologic rigor for high-quality studies identified in USPSTF evidence syntheses. Consultation with the awardees regularly

could ensure that the research is being conducted in a timely manner with fidelity to the research needs specification. Additionally, the committee encourages NIH and other funders to explore developing creative new programs that can be brought to bear on closing evidence gaps in clinical prevention. Expanding and establishing trans-NIH prevention interests into networks for clinical prevention research could also contribute to closing high-priority evidence gaps identified by the USPSTF by establishing an infrastructure to conduct prevention research. Such a network would complement the Centers for Disease Control and Prevention’s Prevention Research Centers.

Advancing the Work of the USPSTF

The committee understands the immensity of the workload of the USPSTF, the rigor with which the members adhere to the advice of methodologists, and the importance of consistency of approach across time. The committee makes recommendations intended to enhance and advance the work of the USPSTF in order for them to provide as many definitive recommendations for practice as possible and to ensure that the recommendations lead to improved health of the target population.

Recommendation 7: The Agency for Healthcare Research and Quality should fund research in how guideline developers assess evidence and issue clinical practice guidelines, including how the decision of “insufficient evidence” is made and how a committee decides which research gaps are the key.

Most evidence-based CPGs include a statement of the level of the evidence and a separate statement on the strength of the recommendation. An evidence-based guideline may be explicit in terms of the methods for evaluating the level of evidence but not explicit in how the guideline committee uses many factors to determine not only the strength of the recommendation but the recommendation itself.

Different expert groups looking at the same topic can come up with different guideline statements. Sometimes this is because they look at different data (an accidentally incomplete literature review or a review that is purposefully limited by language, years of publication, study type). However, even if two guideline panels look at the same evidence there will often be differences in the guideline statements informed by differences in the dynamics of the guideline committee, different biases, and differences in the rules used for the guideline.

It is important to better understand the process of group decision making by a guideline panel. At times, how the committees works may

be more important than the facts discussed. This means there are many questions on what determines which data the committee chooses to look at, how it evaluates the data, how it balances the different elements of the evidence, how the dynamics of the committee affect the final assessment/guideline statement, and so on. These questions, if addressed, can improve guideline fidelity and quality. Increasing the knowledge of the processes of group decision making may lead to methods for improving this task.

Recommendation 8: The Agency for Healthcare Research and Quality should work with relevant government agencies and key stakeholders to evaluate how effectively U.S. Preventive Services Task Force recommendations are implemented in real-world settings to identify and address gaps in achieving the intended benefits.

Simply making a recommendation may not have the intended impact on clinical care and population health. Evaluation of programs suggested by a recommendation can occur for both process and health outcomes. These evaluations ask if outcomes improve after the guideline is disseminated or implemented. For example, do colorectal cancer screenings increase after the guideline is published (process outcome); does colorectal cancer death decrease after the guideline is published (clinical outcome); or are some groups disproportionately benefiting after a recommendation while others lag behind (process and clinical outcome) (Davidson et al., 2021)?

Many questions are of course addressed and provisionally answered in the initial efficacy studies, but as these programs are promulgated, implemented, and disseminated over time, questions related to effectiveness should arise. A primary question is how a program’s “real-world” outcomes compare to those of the initial trials. The outcomes may change because of training (practitioners in real-world settings may be trained and incentivized differently than those in a trial), and there may not be sufficient fiscal and skilled personnel resources available. This may lead to a program that is implemented differently than the program that was evaluated in the efficacy studies. In addition, underlying technologies and protocols may change, as may the target populations or the disease natural histories. All of these factors may influence the effectiveness of the disseminated program. AHRQ, working with relevant government agencies and key stakeholders, can develop programs to improve the evaluation of the effect of USPSTF CPGs.

COMMITTEE VISION

The committee developed a vision as to how the taxonomy and larger report recommendations would serve AHRQ’s and ODP’s goals to address evidence gaps in clinical prevention research. The vision encompasses activities, outputs, short-term outcomes, and long-term outcomes (see Figure S-3 and Chapter 5 for more detail). The committee approached the taxonomy and recommendations as an important component of a much larger effort for AHRQ, ODP, and USPSTF partners and other stakeholders to improve clinical prevention research and practice. They may find success in this endeavor not only by implementing this committee’s taxonomy and recommendations but also by adapting the committee’s vision to improve clinical prevention research and practice in the United States. The committee acknowledges that to do so will increase the burden on AHRQ and NIH staff and the USPSTF members. Resources will be needed to accomplish this.

REFERENCES

This page intentionally left blank.