National Academies Press: OpenBook

Closing Evidence Gaps in Clinical Prevention (2022)

Chapter: 5 Recommendations

« Previous: 4 Taxonomy and Workflow
Suggested Citation:"5 Recommendations." National Academies of Sciences, Engineering, and Medicine. 2022. Closing Evidence Gaps in Clinical Prevention. Washington, DC: The National Academies Press. doi: 10.17226/26351.
×

5

Recommendations

The committee makes eight recommendations in three categories: using the taxonomy developed in Chapter 4, fostering clinical prevention research, and advancing the work of the U.S. Preventive Services Task Force (USPSTF). The rationale for each recommendation follows the recommendation text itself. The chapter and report end with the committee’s vision.

USING THE TAXONOMY

Chapter 4 describes a taxonomy for use in systematically describing evidence gaps in three facets: foundational research, the analytic framework used by the USPSTF, and dissemination and implementation (D&I). The committee also outlines two additional facets to guide the process for developing a research agenda: prioritizing evidence gaps and outlining key research studies. Embedded throughout this process is the opportunity to elevate evidence gaps pertaining to recommendations that promote health equity. The committee envisions identifying and prioritizing evidence gaps as the responsibility of the USPSTF with assistance from staff from the Agency for Healthcare Research and Quality (AHRQ) and the National Institutes of Health’s (NIH’s) Office of Disease Prevention (ODP) and other stakeholders, including but not limited to the USPSTF partner organizations. The committee developed Recommendations 1 and 2 with the work of the USPSTF in mind but hopes that other prevention guideline developers find this useful and adopt it. The committee

Suggested Citation:"5 Recommendations." National Academies of Sciences, Engineering, and Medicine. 2022. Closing Evidence Gaps in Clinical Prevention. Washington, DC: The National Academies Press. doi: 10.17226/26351.
×

understands that successfully implementing these recommendations will take time, staff, training, and other resources, and encourages AHRQ and NIH to begin planning their implementation as soon as possible. Recommendation 3 is directed at AHRQ and NIH to maintain and update the taxonomy.

Characterizing Evidence Gaps

Recommendation 1: For each of its recommendation statements, the U.S. Preventive Services Task Force should use the Clinical Prevention Research Taxonomy to identify and describe evidence gaps.

The USPSTF will likely and understandably be most inclined to focus their efforts on the analytic framework evidence gaps, as it is already a part of their process. These evidence gaps emerge from the evidence review provided by an Evidence-based Practice Center (EPC) and the USPSTF deliberations (AHRQ, 2018). As noted in Chapters 2 and 4, each USPSTF recommendation statement includes a section titled “Research Needs and Gaps,” and the text of these sections has grown more specific in recent years. Framing the research needs around the terms within the taxonomy will impart consistency in the USPSTF efforts to describe research needs. The committee notes that while it is important for the USPSTF to articulate what evidence gaps exist, at least one I statement mentioned research that is not needed (USPSTF, 2018a). That information is likely very helpful to researchers and to funders. Although this recommendation does not refer to EPCs, it is nonetheless envisioned that AHRQ will direct EPCs to conduct reviews using terminology consistent with the taxonomy.

The USPSTF can also use the taxonomy to identify foundational research gaps. As described in Chapters 2 and 4, the basis for some I statements rests on the inability of the EPC and the USPSTF to use the analytic framework due to a lack of common understanding among researchers and clinicians regarding definitions of the condition, accepted and standardized diagnoses and treatments, or simple mechanistic information about how and when a condition leads to adverse health outcomes. As an example, the research need section in the USPSTF statement about screening for child maltreatment states, “Further research to determine effective methods for clinicians to identify children at increased risk should be a priority” (USPSTF, 2018b). Other recommendations lack evidence on groups disproportionately affected by the condition or on the specifics of the preventive service.

Suggested Citation:"5 Recommendations." National Academies of Sciences, Engineering, and Medicine. 2022. Closing Evidence Gaps in Clinical Prevention. Washington, DC: The National Academies Press. doi: 10.17226/26351.
×

Dissemination and Implementation Gaps

AHRQ has a robust implementation portfolio. While the USPSTF has not had D&I of recommendations as its remit, it has recognized the importance of D&I research. The first annual report to Congress issued in 2011 identified “Gaps in Implementation Research” as follows (USPSTF, 2011, p. 13):

  1. How do primary care professionals incorporate new evidence to change their practice?
  2. What are the most effective strategies to assist primary care professionals in the translation of evidence-based clinical preventive services into practice?
  3. How can primary care professionals share evidence with their patients to empower patients and families to make health care decisions about prevention?
  4. How can health information technology, including electronic health records and personal health records, be utilized to increase the number of Americans receiving recommended clinical preventive services?
  5. How can the USPSTF continue to improve its work to better meet the needs of primary care professionals and their patients?

More recently, AHRQ staff and the USPSTF members wrote,

Although the use of implementation research is not within the scope of the USPSTF’s deliberations, it notes that there are critical questions about how to best implement recommended clinical preventive services in primary care practices. Additional implementation and translational research will increase the value of the USPSTF’s work and would be helpful in its deliberations. (Mabry-Hernandez et al., 2018)

Because the committee was not tasked with recommending how the USPSTF determines its recommendations, it does not intend that dissemination and implementation considerations would influence the letter grade recommendations issued by the USPSTF. However, it is clear from the request from AHRQ and ODP to the National Academies of Sciences, Engineering, and Medicine and from the discussions with the committee in open sessions, that the committee was encouraged to address clinical prevention research needs writ large (NASEM, 2020). Ignoring dissemination and implementation issues could mean that important preventive services are not deployed.

There is not currently a body tasked with ensuring that the USPSTF recommendations are adopted in practice, although the requirement in

Suggested Citation:"5 Recommendations." National Academies of Sciences, Engineering, and Medicine. 2022. Closing Evidence Gaps in Clinical Prevention. Washington, DC: The National Academies Press. doi: 10.17226/26351.
×

the Patient Protection and Affordable Care Act that A and B recommendations be covered at no cost to patients is an important tool (KFF, 2015). The Guide to Community Preventive Services, discussed in Chapter 2, deals with some of the same topics as the USPSTF, focusing on population-based interventions, but their remit is much broader than the clinical preventive services the USPSTF addresses. Additionally, Murad et al. (2021) recently proposed the consideration of contextual and implementation evidence as part of a five-prong approach to supplementing systematic review findings when evidence is insufficient for decision making. The authors explain, “Evidence on other factors that may affect the decision may be helpful to decision makers, such as patient values, equity, resources, acceptability, and feasibility” (Murad et al., 2021). For example, the USPSTF updated its lung cancer screening recommendation to lower pack years and age for initiating screening (USPSTF, 2021c). This revised recommendation may promote health equity by identifying Black men who are at greater risk for lung cancer relative to white men (Doubeni et al., 2021). Dissemination and implementation approaches could also address known inequities in screening access, screening quality, time to treatment, and treatment quality (Doubeni et al., 2021).

There are several ways the work related to the D&I of the USPSTF recommendations could be accomplished. The committee hopes that eventually the USPSTF will assume responsibility for the D&I work, agreeing with the USPSTF members and AHRQ staff who acknowledge that it will be helpful in their work (Mabry-Hernandez et al., 2018). However, the committee understands that the USPSTF carries a large workload and that this requires time and resources to be fully implemented. In the short term, partner organizations and staff at AHRQ and ODP could provide pivotal support and input. Other important partners for working on D&I include AHRQ staff who do not already provide scientific and technical support to USPSTF; the Centers for Disease Control and Prevention (CDC), particularly the Community Guide and the Prevention Research Center networks; NIH staff tasked with implementation research; the Patient-Centered Outcomes Research Institutes; and representatives of professional provider organizations, such as the American Academy of Family Physicians or the American Academy of Pediatrics.

As with foundational and analytic framework evidence gaps, addressing D&I will involve information gathering, identification of evidence gaps, and a research agenda addressing priority gaps. For the evidence review, AHRQ could broaden the request to the EPC doing the evidence review for the recommendation statement or engage a different group. The evidence review and identification of evidence gaps could range from an extremely thorough process and report, similar to the evidence reviews that underpin the current USPSTF recommendations, to a more

Suggested Citation:"5 Recommendations." National Academies of Sciences, Engineering, and Medicine. 2022. Closing Evidence Gaps in Clinical Prevention. Washington, DC: The National Academies Press. doi: 10.17226/26351.
×

top-level summary of key issues derived from workshops with stakeholders, for example. EPCs could comment on the D&I challenges, particularly those that affect health equity, with input from their review author teams and stakeholders. Regardless of who performs the evidence review and identifies the D&I evidence gaps, the committee has provided a taxonomy for use. The committee envisions that the USPSTF will be sufficiently supported and resourced to be involved in and informed by this activity. Nevertheless, the committee acknowledges that this work will take time, staff, and other resources.

For some recommendation topics, there will be scant information on dissemination and implementation. For others, such as recommendations that address an expanded age range for an already recommended intervention (as with lung cancer screening), or an intervention already widely used in practice despite the recommendation grade, there could be a great deal of existing information and expertise available from the provider community. D&I issues might also be similar for certain types of interventions. For example, the implementation issues for administering brief questionnaires are likely to be similar, regardless of the specific topic. In addition, implementation issues around preventive services requiring costly equipment will likely have features in common, and referring patients based on screening results to community resources will likely have common or overlapping D&I issues as well. Those interested in describing gaps related to dissemination and implementation work might start with an effort to outline common themes or challenges in implementation to facilitate the work on specific I statements.

Emerging technological advances for discrete steps in the systematic review process, such as artificial intelligence, can revolutionize the efficiency of systematic review production and updating (Lau, 2019). For example, living systematic reviews, which incorporate findings from research studies as they become available, are also emerging as a strategy (Elliott et al., 2017; McDonagh et al., 2021). As another example, a machine learning-based routine has been developed to distinguish between randomized controlled trials (RCTs), quasi-randomized trials, and non-randomized studies (Marshall et al., 2018). Others can extract PICO (population, intervention, comparison, outcomes) information, study design and results, and risk of bias information (Jonnalagadda et al., 2015). These methods could potentially facilitate a less burdensome review of D&I evidence.

It is important to recognize the iterative nature of the assessment of the analytic framework evidence and the D&I evidence, even with I statements. Postponing a review of information on dissemination and implementation until a letter grade recommendation of A or B is made by the USPSTF presents potential problems. Should the D&I evidence

Suggested Citation:"5 Recommendations." National Academies of Sciences, Engineering, and Medicine. 2022. Closing Evidence Gaps in Clinical Prevention. Washington, DC: The National Academies Press. doi: 10.17226/26351.
×

review suggest that the barriers to adoption and spread of the intervention are great, the USPSTF and others might not prioritize filling analytic framework gaps. Alternatively, D&I research might suggest insufficient evidence for a complex intervention, yet sufficient evidence for a simpler intervention. The USPSTF and others may use this information when evaluating the intervention (a less effective intervention that can be implemented easily may be preferred over a more effective intervention that cannot be implemented). Additionally, appropriately targeted research on dissemination and implementation, if not funded or conducted until after an intervention receives a letter grade of A or B, will lag, and the needs of practitioners and patients will suffer.

Given the demands on the USPSTF members and on the EPCs, a full articulation by the USPSTF of D&I evidence gaps might not be possible, particularly at the time of the publication of the recommendation or I statement. When these evidence gaps are articulated, regardless by whom, the USPSTF recommendation statement page could link to the information, even if it is not authored by the USPSTF.

The committee’s task was primarily focused on the USPSTF; however, the committee hopes that other guideline developers working in prevention will consider using the taxonomy provided. Consistency in framing research questions about a specific intervention could lead to synergies in research agendas and research funding and thereby advance health equity. AHRQ and ODP can post the taxonomy prominently on their websites so that others can use it.

Developing a Research Agenda

Recommendation 2: For each recommendation and I statement, the U.S. Preventive Services Task Force (USPSTF) should indicate high-priority evidence gaps. Staff from the Office of Disease Prevention (ODP) in the National Institutes of Health and from the Agency for Healthcare Research and Quality (AHRQ) should broadly outline research that could address those priority gaps and be available to help funders, whether NIH or others, develop a research agenda to address them. The USPSTF and staff from ODP and AHRQ should consult partner organizations and stakeholders as needed.

The committee was asked to provide a taxonomy of evidence gaps and has done so; however, as described in Chapters 1 and 4, the committee was also urged to facilitate prioritizing the evidence gaps. In order to make those priorities evident and actionable by funders and researchers, the committee outlines two steps in the development of a research agenda: identifying priority gaps related to a specific recommendation or

Suggested Citation:"5 Recommendations." National Academies of Sciences, Engineering, and Medicine. 2022. Closing Evidence Gaps in Clinical Prevention. Washington, DC: The National Academies Press. doi: 10.17226/26351.
×

I statement and broadly outlining research to address priority evidence gaps.

Prioritization

The committee acknowledges that formal prioritization exercises can be difficult and burdensome and might require additional resources to support this activity. Nevertheless, the committee believes that any description of evidence gaps in foundational issues or the analytic framework should include a discussion of the priorities. Two I statements in the last 5 years reference a research need as a “priority,” so this is not completely anathema to the USPSTF (2017, 2018b). The USPSTF members are best suited to prioritize evidence gaps, especially regarding foundational research and analytic framework gaps.

As above regarding the USPSTF role in reviewing D&I evidence, the committee suggests that, given the demands on the USPSTF members, identifying priority gaps and outlining research to fill those gaps could be done by AHRQ staff and ODP and other NIH staff in consultation with the USPSTF members and stakeholders. If there are concerns about this process holding up issuance of a USPSTF recommendation statement, this could be published separately and subsequent to publication of the recommendation statement itself.

A structured method for setting priorities would be an important enhancement of the committee’s approach but is beyond the remit and expertise of the committee. However, AHRQ and NIH could embark on this project. Examples of such structured approaches to prioritization include rating, ranking, voting, multi-criteria decision analysis, analytic hierarchy process; and value of information. Iterative dialogue among stakeholders will be important in this process, regardless of the approach chosen. Until such methods are tested and deployed, those setting USPSTF-related research priorities can use the prioritization criteria terms in the committee’s taxonomy qualitatively or perhaps selectively.

The USPSTF might initially focus on “centrality,” rather than prioritizing using all of the criteria proposed. The USPSTF has recently committed to “advancing its methods to better identify when systemic racism contributes to health inequities and to include evidence-based strategies that will reverse the negative effects of systemic racism on preventable disease” (Doubeni et al., 2021, p. 628). As such, the committee anticipates that the USPSTF members might also consider health equity as a prioritization criterion. The committee further urges that whoever takes on the responsibility for prioritizing dissemination and implementation evidence gaps think broadly about the stakeholders it involves, to include practitioners who disproportionately care for patients with the condition, and to include patients themselves.

Suggested Citation:"5 Recommendations." National Academies of Sciences, Engineering, and Medicine. 2022. Closing Evidence Gaps in Clinical Prevention. Washington, DC: The National Academies Press. doi: 10.17226/26351.
×

It should be emphasized that prioritized evidence gaps may not and should not only emerge from I statements. The USPSTF includes research needs for all populations singled out in recommendation statements, not just I statements. Research needs for a specific subgroup might be prioritized as more important to address than the research needs that apply to a different subgroup within the same recommendation statement. For example, the USPSTF recommends chlamydia and gonorrhea screening for women with a B grade, while there is an I statement for screening for men (USPSTF, 2021a). Current evidence gaps note that the prevalence of disease is high among American Indian/Alaska Native, Black, Hispanic/Latino, and Native Hawaiian/Pacific Islander persons. When developing a research agenda for this recommendation statement, groups like AHRQ, NIH, and the USPSTF will need to consider how to prioritize research addressing racial and ethnic health disparities compared with the dearth of evidence regarding screening men in general for chlamydia or gonorrhea.

The committee has not addressed the important issue of prioritizing across recommendation statements. The prioritization facet would be useful for that as well. As the AHRQ, NIH, the USPSTF, and stakeholders gain experience prioritizing within a recommendation statement, they could indicate priorities across topics. The annual reports to Congress would be a natural place for such efforts to emerge.

The prioritization taxonomy is useful for many purposes and might result in different priority rankings when applied by different stakeholders. The USPSTF should use it to signal to funders the evidence gaps it most needs addressed to improve its ability to offer recommendations for clinical practice. Funders might come to different priorities using the same criteria because of differences in focus or mission. For example, a funder who has prioritized advancing equity in their mission might choose to invest in those evidence gaps for which evidence is missing regarding disadvantaged populations. A funder specifically interested in advancing women’s health might choose to invest in evidence gaps that focus on women as a subpopulation about whom more information is needed.

Outlining Research

One of the frustrations identified by stakeholders is that the USPSTF research needs have not usually been specific with regard to some key elements of study design, such as whether an RCT is required. As described in Chapter 4, the committee provides a taxonomy of study specifications, an elaboration of the standard PICOTS (population, intervention, comparison, outcome, timing, setting) framework, which should be considered by whomever completes the research agenda. The enhancements of

Suggested Citation:"5 Recommendations." National Academies of Sciences, Engineering, and Medicine. 2022. Closing Evidence Gaps in Clinical Prevention. Washington, DC: The National Academies Press. doi: 10.17226/26351.
×

the PICOTS framework include aggregability, design considerations, and potential funders and funding mechanisms. In particular, the committee urges ODP and AHRQ staff, when outlining research to address the prioritized gaps, to consider these three enhancements. Aggregability is important for building a robust dataset for the USPSTF to consider in the future. NIH staff in the Institutes and Centers (ICs) interested in supporting research addressing USPSTF gaps could also consider whether the newly funded studies can be pooled with existing evidence.

Detailed design considerations could be left up to those submitting research proposals, but if the USPSTF or others outlining the research agenda think an RCT is or is not necessary, they should so state. Since 2018, the USPSTF has specifically called for an RCT in four I statement research need sections (USPSTF, 2018c, 2019, 2021b,d). Another I statement calls out the need for research that would allow for better pooling of new data (USPSTF, 2020). The importance of and rationale for specifying potential funding mechanisms are addressed in Recommendation 6.

The committee urges that each item in the specification facet be considered, but that the research agenda not be overly prescriptive, so as not to limit researcher creativity or funders’ expectations of research proposals. Appendix B includes a discussion of study designs that AHRQ and ODP staff, funders, and researchers can consider when outlining studies for a USPSTF-focused research agenda.

Institutionalizing the Use of the Taxonomy

Recommendation 3: The National Institutes of Health and the Agency for Healthcare Research and Quality should make the taxonomy accessible on their websites and integrate the taxonomy terms and phrases in their relevant publications, including but not limited to the U.S. Preventive Services Task Force recommendation statements, funding announcements, and grant and contract awards.

The goal is to make the taxonomy and workflow accessible to the clinical prevention community and the public at large. Consistent use of these metadata will provide a common language to stakeholders and encourage wider adoption of the taxonomy. Applying the taxonomy terms in publications linking USPSTF recommendations or I statements with funding opportunity announcements will be crucial for making evident the impact of the research agenda laid out by the USPSTF.

In order to ensure relevance and usability, the taxonomy and workflow will need to be maintained, updated, and evaluated. Maintenance of the taxonomy includes logistics such as documentation, formatting, and

Suggested Citation:"5 Recommendations." National Academies of Sciences, Engineering, and Medicine. 2022. Closing Evidence Gaps in Clinical Prevention. Washington, DC: The National Academies Press. doi: 10.17226/26351.
×

sharing with other stakeholders. Updating and evaluating the taxonomy and workflow require subject-matter expertise to assess and modify terms to reflect the needs and experiences of the users. It is not unusual for taxonomies to undergo regular updates. AHRQ and ODP staff are well placed to advise on this. Encouraging wide use of the taxonomy means that the taxonomy must be shared; thus, AHRQ and ODP are encouraged to post the taxonomy on their websites.

FOSTERING CLINICAL PREVENTION RESEARCH

The committee was asked to propose ways for prevention research funders and guideline developers to enhance their partnership to accelerate research to close important gaps in prevention. A robust and relevant research agenda requires funding, funding requests that are clear and clearly linked to USPSTF research needs, and appropriate mechanisms for and oversight of funded research. As such, the committee provides recommendations regarding the funding and commissioning of clinical prevention research.

Recommendation 4: Funders, in particular the National Institutes of Health and the Patient-Centered Outcomes Research Institute, should set aside funding to address high-priority evidence gaps identified by the U.S. Preventive Services Task Force.

In order to act in a timely manner on a research agenda for clinical prevention developed by the USPSTF, AHRQ staff and ODP and other NIH staff, EPCs, and other stakeholders, dedicated funding will be needed. NIH and stakeholders can work with Congress and the U.S. Department of Health and Human Services leadership to identify appropriate funding levels and secure sufficient funding for this area of need. This funding might be identified and managed at the IC level or at the level of the Office of the Director, or both. Many research questions may require trans-institute collaboration, and these should be actively encouraged. Managing large, directive research, such as called for in Recommendation 4, requires program staff with specialized expertise and the remit to focus on this important task.

The committee acknowledges that calls for increased funding for prevention research have been made by others in the past and not always successfully. However, as described in Chapter 3, NIH’s budget, while never large enough to satisfy all stakeholders, is approximately $41.7 billion and in 2016 only 20 percent of that was, at best, spent on prevention research, broadly defined (Murray, 2017; NIH, 2020). The amount

Suggested Citation:"5 Recommendations." National Academies of Sciences, Engineering, and Medicine. 2022. Closing Evidence Gaps in Clinical Prevention. Washington, DC: The National Academies Press. doi: 10.17226/26351.
×

devoted to clinical prevention research of direct relevance to the USPSTF is undoubtedly much less.

Recommendation 5: The National Institutes of Health staff developing funding announcements for research related to high-priority evidence gaps should consult with the Office of Disease Prevention and the Agency for Healthcare Research and Quality to ensure appropriateness of support mechanisms and fidelity to the research specifications and should include a link to the research needs document on the U.S. Preventive Services Task Force website.

Program announcements; notices of special interest; requests for applications or proposals; and research opportunity announcements related to high-priority research needs of importance to the USPSTF should be as explicit as possible to ensure usefulness to the USPSTF and other guideline developers. AHRQ and ODP staff could serve as an important resource in the crafting of the requests and notices. Reviewers of applications and proposals could be encouraged to study the research needs specification from the USPSTF. Study sections dedicated to these applications could ensure that study section members are oriented to the needs of the USPSTF. As discussed in Chapter 2, although much of the evidence used to establish a letter grade recommendation uses RCT designs, not all important studies do. Given the expense of RCTs, when the research agenda suggests other designs, the funding announcements should so note.

Recommendation 6: The National Institutes of Health and other funders addressing high-priority clinical prevention evidence gaps identified by the U.S. Preventive Services Task Force should use funding mechanisms and processes that can assure that research is conducted expediently, efficiently, and with fidelity to the specified research needs, rather than waiting for such research needs to be addressed predominantly through investigator-initiated grant mechanisms.

The committee encourages NIH and other funders to explore developing creative new programs that can be brought to bear on closing evidence gaps in clinical prevention. For example, NIH’s Clinical and Translational Science Awards (CTSA) program aims to improve the efficiency, quality, and impact of turning research observations into interventions. Their history of supporting multi-disciplinary teams working toward very specific goals is the sort of program that could be used to help close the USPSTF priority evidence gaps. NIH supports a federally funded research and

Suggested Citation:"5 Recommendations." National Academies of Sciences, Engineering, and Medicine. 2022. Closing Evidence Gaps in Clinical Prevention. Washington, DC: The National Academies Press. doi: 10.17226/26351.
×

development center. This very directed and focused mechanism might be used to build out contracted clinical prevention research. Expanding and establishing trans-NIH prevention interests into networks for clinical prevention research could also contribute to closing high-priority evidence gaps identified by the USPSTF by establishing an infrastructure to conduct prevention research. Activities would include dataset curation, survey researchers, modelers, and clinical trial infrastructure that would be poised to respond to calls to fill evidence gaps. Such a network would complement CDC’s Prevention Research Centers, discussed in Chapter 3. As a complement to its work described in Chapter 3 regarding I statement surveys, ODP could serve as the consultant to and coordinator of these directed research efforts.

As discussed in Chapter 3, a recent analysis of research that was used to upgrade a USPSTF recommendation from an I Statement to an A or B recommendation indicates that contract mechanisms were key to resolving three I statements (Klabunde et al., 2021). A contract, or other directive funding mechanism (e.g., Other Transaction Authority Cooperative Agreements), can increase the likelihood that studies are of sufficient design; to allow for aggregating studies; to demonstrate an effect, if the intervention indeed is effective; and to meet the methodologic rigor for high-quality studies identified in the USPSTF evidence syntheses. Regular consultation with the awardees could ensure that the research is being conducted in a timely manner with fidelity to the research needs specification.

As discussed in Chapter 3, NIH has programs that are specifically designed to stimulate technological innovation. Programs such as Small Business Innovation Research (SBIR) or Small Business Technology Transfer (SBTT) could encourage private enterprise to engage in appropriate clinical prevention research. ODP and AHRQ staff could undertake outreach activities to such firms when a high-priority research need involves the development of new technologies.

ADVANCING THE WORK OF THE U.S. PREVENTIVE SERVICES TASK FORCE

The committee understands the immensity of the workload of the USPSTF, the rigor with which the members adhere to the advice of methodologists, and the importance of consistency of approach across time. The committee’s recommendations are based on a combination of evidence explained in Chapter 2 and familiarity with the work of the USPSTF and other guideline developers. As such, the committee makes recommendations intended to enhance and advance the work of the USPSTF and other guideline developers in order for them to provide as many

Suggested Citation:"5 Recommendations." National Academies of Sciences, Engineering, and Medicine. 2022. Closing Evidence Gaps in Clinical Prevention. Washington, DC: The National Academies Press. doi: 10.17226/26351.
×

definitive recommendations for practice as possible, and to ensure that the recommendations lead to improved health of the target population.

Recommendation 7: The Agency for Healthcare Research and Quality should fund research in how guideline developers assess evidence and issue clinical practice guidelines, including how the decision of “insufficient evidence” is made and how a committee decides which research gaps are the key.

Most evidence-based clinical practice guidelines (CPGs) include a statement on the level of the evidence and a separate statement on the strength of the recommendation. The USPSTF combines these two attributes into a “Level of Certainty.” Simply stating a Level of Certainty, or even separately stating a level of evidence and strength of the recommendation, obscures much of the decision making that occurs when translating the evidence into a recommendation. In other words, an evidence-based guideline may be explicit in terms of the methods for evaluating the level of evidence, but not explicit in how the guideline developer uses many factors to determine not only the strength of the recommendation but also the recommendation itself.

This is important because different groups looking at the same topic can issue different guidelines for practice. Sometimes this is because they look at different data (an accidentally incomplete literature review or a review that is purposefully limited by language, years of publication, study type, etc.). Regardless, if two guideline developers look at the same body of evidence, CPGs will still vary, as they are also informed by differences in the dynamics of the guideline developers, different biases among or between the guideline developers, and differences in considerations used to determine the guideline. These considerations might include

  • the types of outcomes (e.g., primary outcomes of a trial versus primary and secondary, including side effects),
  • the balance of benefits and harms,
  • opportunities for shared decision making (including recognizing cultural differences), or
  • the incorporation of elements of equity.

Different guideline developers arriving at different recommendations can lead to several problems. For one, there is the confusion for the clinicians and patients as to which guideline should be followed. That two guideline developers arrive at a different recommendation raises the questions, Which is right?, Is either right?, Might they both be right? In addition, CPGs with differing recommendations can lead to a question-

Suggested Citation:"5 Recommendations." National Academies of Sciences, Engineering, and Medicine. 2022. Closing Evidence Gaps in Clinical Prevention. Washington, DC: The National Academies Press. doi: 10.17226/26351.
×

ing of the process and its validity: Should any guideline be followed? The confusion might also lead payers to wonder which, if any, guideline practice to reimburse.

In addition, there is the question of evidence threshold. How much evidence is needed for a topic to receive a letter grade and become a CPG? Again, this is subjective and will vary not only between CPG panels but on different guideline statements from one panel. This is because many factors (evidence, benefit–harm balance, group dynamics, bias, tradition, and much more) all interplay in this decision.

There are many potential solutions for these problems. One solution is for guideline developers to be even more explicit in their rationales. For example, Rosenfeld et al. (2013) proposed an explicit profile that should accompany each recommendation. The authors proposed that several factors play into the final “strength of recommendation” decision, including the quality of the evidence, the benefits, the risks/harms/costs, the benefit–harm assessment, value judgments, intentional vagueness, role of patient preferences, and others. This level of detail helps make the decision-making process of guideline developers more explicit, and could potentially clarify why two guideline developers might arrive at different conclusions regarding the same topic. These factors are not always clearly defined. For example, one of the factors identified by Rosenfield et al. (2013) is “Risks, harms, cost.” A component of “Risk, harms, cost” might include how each of those are counted in the future. In addition to these factors that supplement the evidence base during guideline development, guideline developers might also consider advancing health equity, achieving successful dissemination and implementation, and improving public health.

This complexity leads to another potential solution. In addition to being more explicit in the thought process supporting a guideline, it is important to better understand the process of group decision making by guideline developers. Balancing the large number of factors and data relevant for a given CPG may be difficult for an individual, and this is made more difficult when each individual brings their own expertise, experiences, and conscious and unconscious biases to a group discussion. At times, more influential than the evidence reviewed might be how the committee works. This means there are many questions on what determines which data the committee chooses to look at, how it evaluates the data, how it balances the different elements of the evidence base, how the dynamics of the committee affect the final assessment. These questions are not evidence gaps, but they are important questions that, if addressed, can improve guideline fidelity and quality. Increasing the knowledge of the processes of group decision making may lead to methods for improving this task.

Suggested Citation:"5 Recommendations." National Academies of Sciences, Engineering, and Medicine. 2022. Closing Evidence Gaps in Clinical Prevention. Washington, DC: The National Academies Press. doi: 10.17226/26351.
×

Recommendation 8: The Agency for Healthcare Research and Quality should work with relevant government agencies and key stakeholders to evaluate how effectively U.S. Preventive Services Task Force recommendations are implemented in real-world settings to identify and address gaps in achieving the intended benefits.

Simply making a recommendation may not have the intended impact on clinical care and population health, and in fact, may lead to widening health disparities—particularly if there are underlying differences in access and quality of screening and care. Evaluation of programs suggested by a recommendation can occur for both process and health outcomes. These evaluations ask if outcomes improve after the guideline is disseminated or implemented. For example, do colorectal cancer screenings increase after the guideline is published (process outcome); does colorectal cancer death decrease after the guideline is published (clinical outcome); or are some groups disproportionately benefiting after a recommendation while others lag behind (process and clinical outcome) (Davidson et al., 2021)?

Program evaluation has been defined as the systematic definition of the merit, worth, and significance of a program, project, or policy (Adams and Neville, 2020). The methods, models, and theories of evaluation science are diverse (Frye and Hemmer, 2012), but the abiding questions are those of determining (in the case of a preventive or other clinical intervention program that has been recommended) the program’s benefits, its harms, who it benefits, and what it costs. Many questions are addressed and provisionally answered in the initial efficacy studies, but as these programs are promulgated, implemented, and disseminated over time, questions related to effectiveness should arise. A primary question is how a program’s “real-world” outcomes compare to those of the initial trials. The outcomes may change because of training (practitioners in real-world settings may be trained and incentivized differently than those in a trial) and there may not be sufficient fiscal and skilled personnel resources available. This may lead to a program that is implemented differently than the program that was evaluated in the efficacy studies. In addition, underlying technologies and protocols may change, as may the target populations or the disease natural histories. All these factors may influence the effectiveness of the program’s implementation.

Some important evaluation research is being conducted. For example, in a study of colorectal cancer in 21 European countries, divergent mortality rates are related to the extent of screening program implementation (Cardoso et al., 2021). Staff from CDC and NIH collaborate on studies using data from national surveys to analyze temporal trends in the use of cancer screening tests (Hall et al., 2018). In another example, in an

Suggested Citation:"5 Recommendations." National Academies of Sciences, Engineering, and Medicine. 2022. Closing Evidence Gaps in Clinical Prevention. Washington, DC: The National Academies Press. doi: 10.17226/26351.
×

evaluation of a meta-analysis and new RCTs, it was concluded that low-dose computed tomography for lung cancer screening does continue to improve survivorship in high-risk persons (Manuel et al., 2021), as reported in the original USPSTF guideline.

Still, despite the professed need for periodic evaluation programs of all important and commonly applied interventions, evaluation challenges remain. In the context of evaluating federal programs, Orr suggested three important barriers: availability of research and evaluation resources, resistance to rigorous evaluation, and convincing policy makers to use evaluation results (NASEM, 2017). These barriers, while important, are addressable, and AHRQ working with relevant government agencies and key stakeholders can develop programs to improve the evaluation of the effect of USPSTF guidelines.

COMMITTEE VISION

This committee approached the taxonomy and recommendations as an important component of a much larger effort for AHRQ, ODP, and USPSTF partners and other stakeholders to improve clinical prevention research and thereby practice. They may find success in this endeavor not only by implementing this committee’s taxonomy and recommendations but also by adapting the committee’s vision to improve clinical prevention research and practice in the United States. The committee acknowledges that to do so will increase the burden on ODP and NIH staff and the USPSTF members. Resources will be needed to accomplish this. The committee’s vision (see Figure 5-1) comprises activities, outputs, and short-term and long-term outcomes.

Suggested Citation:"5 Recommendations." National Academies of Sciences, Engineering, and Medicine. 2022. Closing Evidence Gaps in Clinical Prevention. Washington, DC: The National Academies Press. doi: 10.17226/26351.
×
Image
FIGURE 5-1 The committee vision for improving clinical prevention research.
NOTE: AHRQ = Agency for Healthcare Research and Quality; EPC = Evidence-based Practice Center; NIH = National Institutes of Health; ODP = Office of Disease Prevention; PCORI = Patient-Centered Outcomes Research Institute; USPSTF = U.S. Preventive Services Task Force.
Suggested Citation:"5 Recommendations." National Academies of Sciences, Engineering, and Medicine. 2022. Closing Evidence Gaps in Clinical Prevention. Washington, DC: The National Academies Press. doi: 10.17226/26351.
×

REFERENCES

Adams, J., and S. Neville. 2020. Program evaluation for health professionals: What it is, what it isn’t and how to do it. International Journal of Qualitative Methods 19.

AHRQ (Agency for Healthcare Research and Quality). 2018. U.S. Preventive Services Task Force procedures for evidence-based practice centers. https://www.ahrq.gov/sites/default/files/wysiwyg/funding/contracts/epc-vi/2-uspstf_procedures_guide_for_epcs.pdf (accessed November 3, 2021).

Cardoso, R., F. Guo., T. Heisser, M. Hackl, P. Ihle, H. De Schutter, N. Van Damme, Z. Valerianova, T. Atanasov, O. Májek, J. Mužík, M. C. Nilbert, A. J. Tybjerg, K. Innos, M. Mägi, N. Malila, A. M. Bouvier, V. Bouvier, G. Launoy, et al. 2021. Colorectal cancer incidence, mortality, and stage distribution in European countries in the colorectal cancer screening era: An international population-based study. The Lancet Oncology 22(7):P1002–P1013.

Davidson, K. W., A. H. Krist, C. Tseng, M. Simon, C. A. Doubeni, A. R. Kemper, M. Kubik, Q. Ngo-Metzger, J. Mills, and A. Borsky. 2021. Incorporation of social risk in US Preventive Services Task Force recommendations and identification of key challenges for primary care. JAMA 326(14):1410–1415.

Doubeni, C. A., M. Simon, and A. H. Krist. 2021. Addressing systemic racism through clinical preventive service recommendations from the US Preventive Services Task Force. JAMA 325(7):627–628.

Elliott, J. H., A. Synnot, T. Turner, M. Simmonds, E. A. Akl, S. McDonald, G. Salanti, J. Meerpohl, H. MacLehose, J. Hilton, D. Tovey, I. Shemilt, J. Thomas, and Living Systematic Review Network. 2017. Living systematic review: 1. Introduction—The why, what, when, and how. Journal of Clinical Epidemiology 91:23–30.

Frye, A. W., and P. A. Hemmer. 2012. Program evaluation models and related theories: AMEE guide no. 67. Medical Teacher 34(5):e288–e299.

Hall, I. J., F. K. Tangka, S. A. Sabatino, T. D. Thompson, B. I. Graubard, and N. Breen. 2018. Patterns and trends in cancer screening in the United States. Preventing Chronic Disease 15:170465.

Jonnalagadda, S. R., P. Goyal, and M. D. Huffman. 2015. Automating data extraction in systematic reviews: A systematic review. Systematic Reviews 4(78).

KFF (Kaiser Family Foundation). 2015. Preventive services covered by private health plans under the Affordable Care Act. https://www.kff.org/health-reform/fact-sheet/preventive-services-covered-by-private-health-plans (accessed July 19, 2021).

Klabunde, C. N., E. M. Ellis, J. Villani, E. Neilson, K. Schwartz, E. A. Vogt, and Q. Ngo-Metzger. 2021. Characteristics of scientific evidence informing changed U.S. Preventive Services Task Force insufficient evidence statements. American Journal of Preventive Medicine. https://www.sciencedirect.com/science/article/abs/pii/S0749379721004578 (accessed November 29, 2021).

Lau, J. 2019. Editorial: Systematic review automation thematic series. Systematic Reviews 8:70.

Mabry-Hernandez, I. R., S. J. Curry, W. R. Phillips, F. A. Garcia, K. W. Davidson, J. W. Epling, Jr., Q. Ngo-Metzger, and A. S. Bierman. 2018. U.S. Preventive Services Task Force priorities for prevention research. American Journal of Preventive Medicine 54(1 Suppl 1) S95–S103.

Manuel, L., L. S. Fong, T. Ly, and G. Meredith. 2021. Does lung cancer screening with low-dose computerized tomography improve survival? Interactive Cardiovascular and Thoracic Surgery ivab154.

Marshall, I. J., A. Noel-Storr, J. Kuiper, J. Thomas, and B. C. Wallace. 2018. Machine learning for identifying randomized controlled trials: An evaluation and practitioner’s guide. Research Synthesis Methods 9(4):602–614.

Suggested Citation:"5 Recommendations." National Academies of Sciences, Engineering, and Medicine. 2022. Closing Evidence Gaps in Clinical Prevention. Washington, DC: The National Academies Press. doi: 10.17226/26351.
×

McDonagh, M. S., J. Wagner, A. Y. Ahmed, B. Morasco, D. Kansagara and R. Chou. 2021. Living systematic review on cannabis and other plant-based treatments for chronic pain—quarterly progress report. Rockville, MD: Agency for Healthcare Research and Quality. https://doi.org/10.23970/AHRQEPCCER250.

Murad, M. H., S. M. Chang, C. Fiordalisi, J. S. Lin, T. J. Wilt, A. Tsou, B. Leas, S. Siddique, A. R. Zullo, E. Balk, C. M. Rutter, K. Robinson, C. Coleman, O. Costa, E. Stoeger, M. Helfand, and M. Viswanathan. 2021. Improving the utility of evidence synthesis for decision makers in the face of insufficient evidence. AHRQ Methods for Effective Health Care. https://www.ncbi.nlm.nih.gov/books/NBK569735 (accessed October 28, 2021).

Murray, D. M. 2017. Prevention research at the National Institutes of Health. Public Health Reports 132(5):535–538.

NASEM (National Academies of Sciences, Engineering, and Medicine). 2017. Principles and practices for federal program evaluation: Proceedings of a workshop. Washington, DC: The National Academies Press. https://doi.org/10.17226/24831.

NASEM. 2020. Addressing evidence gaps in clinical prevention: Meeting 1. 2nd Session. December 15, 2020, by the Committee on Addressing Evidence Gaps in Clinical Prevention Recommendations. https://www.nationalacademies.org/event/12-15-2020/addressing-evidence-gaps-in-clinical-prevention-committee-meeting-1-session-2 (accessed October 28, 2021).

NIH (National Institutes of Health). 2020. What we do: Budget. https://www.nih.gov/about-nih/what-we-do/budget (accessed August 10, 2021).

Rosenfeld, R. M., R. N. Shiffman, and P. Robertson. 2013. Clinical practice guideline development manual, third edition: A quality-driven approach for translating evidence into action. Otolaryngology—Head and Neck Surgery 148:S1–S55.

USPSTF (U.S. Preventive Services Task Force). 2011. First annual report to Congress on high-priority evidence gaps for clinical preventive services. https://www.uspreventiveservicestaskforce.org/uspstf/about-uspstf/reports-congress/first-annual-report-congresshigh-priority-evidence-gaps-clinical-preventive-services (accessed August 27, 2021).

USPSTF. 2017. Celiac disease: Screening. https://www.uspreventiveservicestaskforce.org/uspstf/recommendation/celiac-disease-screening (accessed August 17, 2021).

USPSTF. 2018a. Cardiovascular disease: Risk assessment with nontraditional risk factors. https://www.uspreventiveservicestaskforce.org/uspstf/recommendation/cardiovascular-disease-screening-using-nontraditional-risk-assessment (accessed August 17, 2021).

USPSTF. 2018b. Child maltreatment: Interventions. https://www.uspreventiveservicestaskforce.org/uspstf/recommendation/child-maltreatment-primary-care-interventions (accessed August 17, 2021).

USPSTF. 2018c. Intimate partner violence, elder abuse, and abuse of vulnerable adults: Screening. https://www.uspreventiveservicestaskforce.org/uspstf/recommendation/intimate-partner-violence-and-abuse-of-elderly-and-vulnerable-adults-screening (accessed August 17, 2021).

USPSTF. 2019. Abdominal aortic aneurysm: Screening. https://www.uspreventiveservicestaskforce.org/uspstf/recommendation/abdominal-aortic-aneurysm-screening (accessed August 17, 2021).

USPSTF. 2020. Illicit drug use in children, adolescents, and young adults: Primary care-based interventions. https://www.uspreventiveservicestaskforce.org/uspstf/recommendation/drug-use-illicit-primary-care-interventions-for-children-and-adolescents (accessed August 17, 2021).

USPSTF. 2021a. Chlamydia and gonnorhea: Screening. https://www.uspreventiveservicestaskforce.org/uspstf/recommendation/chlamydia-and-gonorrhea-screening (accessed November 3, 2021).

Suggested Citation:"5 Recommendations." National Academies of Sciences, Engineering, and Medicine. 2022. Closing Evidence Gaps in Clinical Prevention. Washington, DC: The National Academies Press. doi: 10.17226/26351.
×

USPSTF. 2021b. Gestational diabetes: Screening. https://www.uspreventiveservicestaskforce.org/uspstf/recommendation/gestational-diabetes-screening (accessed August 17, 2021).

USPSTF. 2021c. Lung cancer: Screening. https://www.uspreventiveservicestaskforce.org/uspstf/recommendation/lung-cancer-screening (accessed November 3, 2021).

USPSTF. 2021d. Tobacco smoking cessation in adults, including pregnant persons: Interventions. https://www.uspreventiveservicestaskforce.org/uspstf/recommendation/tobacco-use-in-adults-and-pregnant-women-counseling-and-interventions (accessed August 17, 2021).

Suggested Citation:"5 Recommendations." National Academies of Sciences, Engineering, and Medicine. 2022. Closing Evidence Gaps in Clinical Prevention. Washington, DC: The National Academies Press. doi: 10.17226/26351.
×
Page 113
Suggested Citation:"5 Recommendations." National Academies of Sciences, Engineering, and Medicine. 2022. Closing Evidence Gaps in Clinical Prevention. Washington, DC: The National Academies Press. doi: 10.17226/26351.
×
Page 114
Suggested Citation:"5 Recommendations." National Academies of Sciences, Engineering, and Medicine. 2022. Closing Evidence Gaps in Clinical Prevention. Washington, DC: The National Academies Press. doi: 10.17226/26351.
×
Page 115
Suggested Citation:"5 Recommendations." National Academies of Sciences, Engineering, and Medicine. 2022. Closing Evidence Gaps in Clinical Prevention. Washington, DC: The National Academies Press. doi: 10.17226/26351.
×
Page 116
Suggested Citation:"5 Recommendations." National Academies of Sciences, Engineering, and Medicine. 2022. Closing Evidence Gaps in Clinical Prevention. Washington, DC: The National Academies Press. doi: 10.17226/26351.
×
Page 117
Suggested Citation:"5 Recommendations." National Academies of Sciences, Engineering, and Medicine. 2022. Closing Evidence Gaps in Clinical Prevention. Washington, DC: The National Academies Press. doi: 10.17226/26351.
×
Page 118
Suggested Citation:"5 Recommendations." National Academies of Sciences, Engineering, and Medicine. 2022. Closing Evidence Gaps in Clinical Prevention. Washington, DC: The National Academies Press. doi: 10.17226/26351.
×
Page 119
Suggested Citation:"5 Recommendations." National Academies of Sciences, Engineering, and Medicine. 2022. Closing Evidence Gaps in Clinical Prevention. Washington, DC: The National Academies Press. doi: 10.17226/26351.
×
Page 120
Suggested Citation:"5 Recommendations." National Academies of Sciences, Engineering, and Medicine. 2022. Closing Evidence Gaps in Clinical Prevention. Washington, DC: The National Academies Press. doi: 10.17226/26351.
×
Page 121
Suggested Citation:"5 Recommendations." National Academies of Sciences, Engineering, and Medicine. 2022. Closing Evidence Gaps in Clinical Prevention. Washington, DC: The National Academies Press. doi: 10.17226/26351.
×
Page 122
Suggested Citation:"5 Recommendations." National Academies of Sciences, Engineering, and Medicine. 2022. Closing Evidence Gaps in Clinical Prevention. Washington, DC: The National Academies Press. doi: 10.17226/26351.
×
Page 123
Suggested Citation:"5 Recommendations." National Academies of Sciences, Engineering, and Medicine. 2022. Closing Evidence Gaps in Clinical Prevention. Washington, DC: The National Academies Press. doi: 10.17226/26351.
×
Page 124
Suggested Citation:"5 Recommendations." National Academies of Sciences, Engineering, and Medicine. 2022. Closing Evidence Gaps in Clinical Prevention. Washington, DC: The National Academies Press. doi: 10.17226/26351.
×
Page 125
Suggested Citation:"5 Recommendations." National Academies of Sciences, Engineering, and Medicine. 2022. Closing Evidence Gaps in Clinical Prevention. Washington, DC: The National Academies Press. doi: 10.17226/26351.
×
Page 126
Suggested Citation:"5 Recommendations." National Academies of Sciences, Engineering, and Medicine. 2022. Closing Evidence Gaps in Clinical Prevention. Washington, DC: The National Academies Press. doi: 10.17226/26351.
×
Page 127
Suggested Citation:"5 Recommendations." National Academies of Sciences, Engineering, and Medicine. 2022. Closing Evidence Gaps in Clinical Prevention. Washington, DC: The National Academies Press. doi: 10.17226/26351.
×
Page 128
Suggested Citation:"5 Recommendations." National Academies of Sciences, Engineering, and Medicine. 2022. Closing Evidence Gaps in Clinical Prevention. Washington, DC: The National Academies Press. doi: 10.17226/26351.
×
Page 129
Suggested Citation:"5 Recommendations." National Academies of Sciences, Engineering, and Medicine. 2022. Closing Evidence Gaps in Clinical Prevention. Washington, DC: The National Academies Press. doi: 10.17226/26351.
×
Page 130
Suggested Citation:"5 Recommendations." National Academies of Sciences, Engineering, and Medicine. 2022. Closing Evidence Gaps in Clinical Prevention. Washington, DC: The National Academies Press. doi: 10.17226/26351.
×
Page 131
Suggested Citation:"5 Recommendations." National Academies of Sciences, Engineering, and Medicine. 2022. Closing Evidence Gaps in Clinical Prevention. Washington, DC: The National Academies Press. doi: 10.17226/26351.
×
Page 132
Next: Appendix A: Topics with "I Statements" from the U.S. Preventive Services Task Force »
Closing Evidence Gaps in Clinical Prevention Get This Book
×
Buy Paperback | $55.00 Buy Ebook | $44.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

Closing Evidence Gaps in Clinical Prevention, a new consensus study report from the National Academies of Sciences, Engineering, and Medicine's Board on Population Health and Public Health Practice, evaluates evidence gaps in clinical prevention recommendations described by the United States Preventive Services Task Force and other clinical practice guideline developers and presents a taxonomy of these evidence gaps for future use. This report aims to improve the coordination of efforts to describe and communicate priority evidence gaps among funders and researchers. It also proposes new opportunities for collaboration among researchers, funders, and guideline developers to accelerate research that could close evidence gaps.

The authoring committee has also developed an interactive graphic that can be used as a workflow diagram for implementing the taxonomy. This workflow walks users through reviewing evidence, characterizing evidence gaps using relevant taxonomies, and developing a research agenda. Click here to view and engage with the interactive graphic.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!