Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
4 The Criteria and Process for Setting Priorities Abstract: This chapter describes the prioritization criteria and process the committee used in its review of the nominated research topics. The com- mittee considered the balance of the portfolio of nominated research topics across research areas, populations to be studied, types of interventions, and methodologies. In establishing prioritization and portfolio criteria, the committee had the benefit of examining methods and criteria used in several past priority-setting exercises. The committee developed two types of criteria: (1) condition-level criteria that relate to the significance of specific conditions or diseases for the population as a whole or certain age groups, and (2) priority topic-level criteria that include the appropriateness of the nominated research topics for CER, information gaps, variability in care, and gaps in translation. The committee used three rounds of vot- ing to narrow the list of nominated topics to a manageable, high priority portfolio. Based on this experience, the committee made recommendations for future priority-setting projects. Prioritization of CER topics should be a sustained and continuous process that requires the prioritizing body to make regular reports to the Secretary of Health and Human Services, in- volves the public in a transparent process, and is informed by robust topic briefs and background information. Introduction The previous chapter described the committeeâs method of obtaining the nominated topics from stakeholders and the public, while this chapter describes the prioritization criteria and process the committee used in its 77
78 INITIAL NATIONAL PRIORITIES FOR CER review of the nominated topics. The first half of the chapter outlines the various types of prioritizing criteria. The committee developed the concept of portfolio criteria that were intended to ensure that the final priority comparative effectiveness research (CER) topics reflect a balance of CER questions across research area (e.g., geriatrics, neurology, psychiatry), study populations (e.g., women, children, elderly), type of intervention (e.g., surgical, pharmaceutical), and study methodology (e.g., randomized trial, observational study). The committee also developed two sets of criteria for committee members to use in evaluating the specific nominated priority topics: (1) condition-level criteria, including data on burden of disease and variability in care, and (2) priority topic-level criteria, including the appro- priateness of the nominated research topics for CER, existing information gaps, variability in care, and gaps in translation. The second section of the chapter includes an overview of the Institute of Medicineâs (IOMâs) previous recommendations, criteria, and procedures for setting research priorities, as well as the criteria and methodologies used by other prioritization projects. The committee relied on these past IOM and external reports to develop the criteria it used in the voting process that established the final priority CER topics. The committeeâs voting process that was used to narrow the list of nominated topics to a manageable, high priority portfolio is also described in detail. The chapter concludes with a description of the lessons learned from the committeeâs priority-setting exercise and provides recommendations for future prioritization processes in CER. The final results of the committeeâs deliberations and voting are presented in Chapter 5. portfolio considerations As described in Chapter 3, the committee developed the majority of its priority CER topics from the public input gathered through the web-based questionnaire. The questionnaire required the respondents to provide de- tailed information about each of their nominated priority topics. Respon- dents were required to identify the primary area of study of their nominated topic from among the 32 disease classifications, patient conditions, and systems of care categories provided. Research areas included categories such as oncology and hematology, geriatrics disorders, neurologic disorders, and so on. Respondents were also asked to identify the proposed population to be studied, ranging from categories based on gender, age, race, special popu- lations, and rare diseases, as well as the proposed comparators, such as alternative and complementary treatments, behavior intervention, devices, and pharmacological therapy. In addition, the respondents were requested to identify a proposed methodology for their nominated research topic.
THE CRITERIA AND PROCESS FOR SETTING PRIORITIES 79 TABLE 4-1â Portfolio and Priorities Criteria Priority Topic-Level Portfolio Criteria Condition-Level Criteria Criteria â¢ Research area â¢ Prevalence â¢ Appropriateness of â¢ Population to be studied â¢ Mortality topic for CER â¢ Interventions â¢ Morbidity â¢ Information gaps and â¢ Proposed methodology â¢ Cost duplication â¢ Variability â¢ Gaps in translation Potential methodologies included database review, prospective observa- tional study, randomized clinical trial, or systematic review. As part of the process of prioritizing the nominated research topics, the committee, with the financial support provided by the Agency for Health- care Research and Quality (AHRQ), created a portfolio of CER topics that was balanced and diverse across the characteristics previously described and listed in Table 4-1 (see next section for discussion of criteria for prior- ity setting). The distribution of the nominated research topics according to the portfolioâs characteristics was provided to the committee throughout each step in the voting process. The committee weighed the balance of the portfolio in its deliberations and selection of the final priority CER topics. The committee selected the following four characteristics for balancing the portfolio after careful consideration of the goals of CER. The commit- tee determined that it is important to have a portfolio balanced in terms of research areas because it did not want to study only those diseases and conditions with the greatest effects on the health of the U.S. population. It determined that it is also important to study rare diseases and conditions that disproportionately and seriously affect subgroups of the population, partly because the scientific opportunity may be greatest for some of these conditions. Similarly, the committee determined that it is essential for the portfolio to be balanced across populations to be studied. The priority CER topics should include populations and subpopulations representing minor- ity, racial and ethnic groups, all genders, and different age groups ranging from infancy to the elderly. The committee determined that the portfolio should also include a diversity of interventions. Traditionally, much of CER has focused on head- to-head comparisons of pharmaceutical treatments. However, the commit- tee saw great value in extending the concept of comparison to include a variety of interventions, including tests to screen for or monitor diseases (e.g., imaging for cancer or during normal pregnancy), surgical techniques (e.g., closed vs. open procedures), and therapeutic alternatives (e.g., medi- cal therapy vs. surgical vs. radiotherapy for prostate cancer). This diversity
80 INITIAL NATIONAL PRIORITIES FOR CER helps ensure that the portfolio covers the entire continuum of health care, including screening and prevention, treatment of acute health problems, chronic health problems, and palliative and end-of-life care. Additionally, CER that examines different means of delivering health care was considered to be an important determinant of quality and was incorporated into the options for intervention. The committee also decided that the portfolio should include a diversity of methodologies. This is especially important in light of the committeeâs charge to develop a portfolio of topics that will lead to an appropriate expenditure of the $400 million for CER in the American Recovery and Re- investment Act (ARRA) of 2009 (P.L. 111-5). The different methodologies vary widely in terms of resource requirements, time lines, and types of in- frastructure necessary to conduct the research. Varying these methodologies in the portfolio ensures that some relatively inexpensive and easy results will be generated early on, and within the scope of ARRA. Performance of randomized controlled trials or prospective observational trials will have to extend well beyond the 2-year focus of ARRA. Criteria Chosen for priority setting In addition to the portfolio criteria intended to assess the balance of all the priority CER topics, the committee also concluded that criteria were needed to evaluate the individual nominated research topics. The committee reviewed the criteria used for priority setting by other projects and the two IOM reports discussed below, but it developed its own set of criteria that applied specifically to setting priorities for CER. After careful consideration, the committee recognized two levels of criteria to assess the nominated CER research topics: (1) condition-level criteria, and (2) priority topic-level criteria. Condition-Level Criteria The condition-level criteria focus on burden of disease indicators that could be readily obtained by the committee. The committee recognized the importance of selecting priority topics that affected a large portion of the population (prevalence), were the leading causes of death and disease (mortality and morbidity), imposed serious costs on patients, families, payers, and society (costs), and had the greatest differences in treatment used by practitioners (variability). The committee would have also liked to include work loss due to disability as a criterion, but it could not find data to support that criterion within the time frame of the study. Data on the following criteria were provided to the committee for consideration in the prioritizing and voting process:
THE CRITERIA AND PROCESS FOR SETTING PRIORITIES 81 â¢ Prevalence: The number or percentage of people with a specified condition in the United States at a given time (Appendix C, Table C-1). â¢ Mortality: The number or percentage of deaths due to a specified condition in the United States in a given time period (Appendix C, Table C-2). â¢ Morbidity: The extent of illness, injury, or disability in a defined population (Appendix C, Table C-3). â¢ Cost: The total treatment expenses for selected conditions (Appen- dix C, Table C-4). â¢ Variability: A measure of the dispersion of data. In the context of Appendix C, Table C-5, it refers to the pattern of variation in admissions for specific procedures among hospital referral regions. In the context of Appendix C, Table C-6, it refers to the pattern of variation in admission for treatment of conditions among hospital referral regions. Priority Topic-Level Criteria The committee made a distinction between condition-level criteria and the criteria that could be applied to individual topics involving those con- ditions (Table 4-1). For example, as shown in Appendix C, the prevalence data provided to the committee identified the top 20 conditions for adults and the top five for children. From this information, committee members might determine that a particular condition targeted in a proposed research topic is widespread and so it may be of importance to study; however, they had little or no information on other aspects of the topic such as whether particular procedures, clinical decisions, or delivery models were also prevalent or appropriate for CER. The specific priority topic-level crite- ria considered by the committee were intended to help assess the particular questions identified in the nominated research topics, not just the conditions and diseases. The priority topic-level criteria used by the committee include the following: â¢ Appropriateness of topic for CER o tility for decision makingâDoes the proposed topic include U populations previously excluded from trials, clinically meaning- ful comparisons, or patient-important outcomes rather than markers or intermediate outcomes? Does it involve direct, head- to-head comparisons to inform the decisions of daily practice? Is it patient-centered so that it tailors the test or treatment to the specific characteristics of the patient? Will it enhance clinical
82 INITIAL NATIONAL PRIORITIES FOR CER practice decision making by patients and physicians in everyday circumstances and help policy makers? o isks associated with careâThere may be risks as well as benefits R associated with particular treatments/methods of care either in current practice or the comparator that should be considered. â¢ Information gaps and duplication. CER should address gaps left by existing research. Research gaps for selected topics can be identified by Cochrane or Evidence-based Practice Center (EPC) systematic reviews on the subject, which compare âwhat we need to knowâ to make good decisions with âwhat we know nowâ from existing studies. The description of gaps should follow the logic of the pre- viously outlined criteria for utility in decision making: o or suggested comparative effectiveness systematic reviews, are F there recent EPC or Cochrane reviews on the same subject? o or suggested CER studies or trials, do recent EPC or Cochrane F reviews on the same subject identify the suggested research as needed research (evidence gaps)? o ave previous studies ignored patients with comorbidities? Does H the proposed study explicitly include them? o ave previous studies ignored patients from special popula- H tions? Does the proposed study include them? o ave previous studies made meaningful comparisons? Does the H proposed study include head-to-head comparisons? o ave previous studies fully explored benefits and harms? H o the proposed topic redundant with current research? Data Is derived from a check of the short list against www.clinicaltrials. gov and answers to the above questions. o Variability â¢ The data requested from Dartmouth indicate rates of hos- pitalizations and common procedures with high variability across the country. â¢ Health care deliveryâDoes the proposed study address the effectiveness of different strategies for delivering the inter- vention? Strategies may include organization characteristics, work patterns, or work processes. â¢ Gaps in translation o oving from research to practiceâHas CER been conducted on M the topic and recommendations made, but with limited impact on practice? Will the proposed study be likely to improve the implementation of the recommendations? Or identify improved strategies for research translation?
THE CRITERIA AND PROCESS FOR SETTING PRIORITIES 83 Data collection to aid Topic selection The committee was unable to collect priority topic-level data relating to the significance of the 1,268 nominated research topics to be assessed initially, and instead, focused mainly on data to support criteria at the condition level. The committee selected individual data tables to aid in the voting pro- cess. The Medical Expenditure Panel Survey (MEPS) (AHRQ, 2009) pro- vided proxy indicators for information on prevalence, morbidity and cost, and the National Vital Statistics Report on mortality (Kung et al., 2008). The Dartmouth Institute for Health Policy and Clinical Practice analyzed clinical practice data according to variation in treatment for medical condi- tions and surgical procedures at the IOMâs request (Wennberg, 2009). Data sources were chosen based on their year of production (with preference given to the most recent reports), representativeness of the whole popula- tion, and ability to provide age stratification. Data pertaining to knowledge gaps (areas of scientific uncertainty in terms of treatment strategies for the population as a whole or for subpopu- lations) and funding gaps (areas with a dearth of recent or existing research studies) were feasible to use after the first round of voting on the 1,268 nominated topics had narrowed to the list of 145. For the second round of voting, the committee was provided several proxy indicators for knowledge and information gaps, including the most recent systematic reviews as well as the funding source and number of recent and ongoing clinical trials. The AHRQ Effective Health Care Programâs issue briefs and the National Institutes of Healthâs registry of privately and publicly supported clinical trials in the United States and abroad supplied data on perceived knowledge gaps remaining to be addressed. Many of the second-round research topic nominations for CER were not covered in the AHRQ issue briefs, and committee members were informed of the unevenness of the supporting data across topics. The committee used the criteria and data tables as guided in voting instructions and as summarized in cover sheets for each of the 32 broad research areas. The cover sheets for each research area indicated whether specified conditions within that research area were among the top-ranked conditions by each condition-level criterion (as ranked in the tables in Appendix C), and if the topic was listed by other national priority-setting projects. For instance, in the cardiovascular and peripheral vascular disease category, hypertension and hyperlipidemia were 2 among the top 20 most prevalent diseases across all conditions for all age groups, and hypertension along with heart conditions were 2 of the top 12 diseases according to the morbidity proxy table, number of events, for all ages. Cardiovascular dis- ease including stroke and hypertension were listed among the top priorities
84 INITIAL NATIONAL PRIORITIES FOR CER of AHRQâs Effective Health Care Program (Whitlock et al., 2009), Healthy People 2010 (HHS, 2000), and the Cochrane Collaboration (Doyle et al., 2005). The cover sheets were designed to be a quick reference to highlight more specific subcategories or conditions within each broad research area. An example of a cover sheet for cardiovascular and peripheral vascular disease is presented in Appendix D. All 32 cover sheets are available at www.iom.edu/cerpriorities. The voting instructions for the committee listed and, if necessary, de- fined the condition-level and priority topic-level criteria for the committee to consider. However, there was no explicit direction on how to weigh criteria, except that members should decide based on their own expertise and preferences. The selected criteria provided the committee with a framework for their voting decisions. However, individual committee member expertise played an important role in the decision-making process. Consequently, although committee members were instructed to take into account quantitative data such as prevalence, morbidity, and cost where such data were available, the voting process had subjective elements in terms of how each member selected their top priorities. Lessons From previous Priority-setting processes IOM Reports For this project, the most relevant IOM reports concerning priority set- ting, Priority Areas for National Action (IOM, 2003) and Knowing What Works in Health Care (IOM, 2008), provided the initial basis for the meth- odology and criteria used by this committee in setting priorities for CER. In Priority Areas for National Action, AHRQ tasked an earlier IOM committee with recommending a list of 20 priority conditions whose im- provement would help the nation achieve significant advances in health care quality over the next 5 years, and then with establishing a process and set of criteria for determining those priorities. The study committee chose a framework, initially designed to assess care across the lifespan, developed by the Foundation for Accountability (FACCT, which closed its opera- tions in 2004) to organize all potential high priority conditions (Markle Foundation, 2008). This framework consisted of four âdomains of careâ: preventive, acute, chronic, and palliative; the committee also considered system-wide interventions that would cut across these domains (FAACT, 1997). The goal was to create a final portfolio of priorities for quality improvement that touched on the full continuum of care. Members of that committee nominated most of the topics with the aid of stakeholder input
THE CRITERIA AND PROCESS FOR SETTING PRIORITIES 85 through presentations at a public workshop and a review of other relevant priority lists (IOM, 2003). That committee then applied three sets of criteria to the suggested top- ics. The criteria reflected potential impact (disease burden variables), im- provability (the likelihood the priority would address one of the six quality aims in the 2001 Quality Chasm report), and inclusiveness (relevance to a broad range of patients, conditions, and health care settings). The com- mittee ranked the topic suggestions within their respective domains of care categories and then determined the complete list, which is shown in Table 4-2 (IOM, 2003). The committee that produced Knowing What Works in Health Care provided further research and recommendations on the topic of setting pri- orities. In a detailed assessment of methods used by other organizations to identify topics for systematic reviews, the committee found little guidance for designing an optimal priority-setting process. However, it did establish several principles for future priority-setting committees (IOM, 2008): â¢ The process should be open, transparent, efficient, and timely. â¢ should consider how evidence-based practice could help reduce It burden of disease. â¢ should include cost considerations in the decision-making process. It â¢ membership should include people with a broad base of inter- Its ests and expertise to minimize bias and conflicts of interest. Knowing What Works in Health Care also identified the most common criteria used in other priority-setting efforts (IOM, 2008): â¢ Burden of disease (prevalence, disability, mortality, morbidity, etc.) â¢ Public controversy (uncertainty around the topic and supporting data) â¢ Cost â¢ Potential impact â¢ New evidence that might change previous conclusions â¢ Existence of an evidence gap â¢ Unexplained variation in the use of healthcare services â The list of organizations includes the Agency for Healthcare Research and Quality, Blue Cross and Blue Shield Association Technology Evaluation Center, Centers for Medicare & Medicaid, the Cochrane Collaboration, Drug Effectiveness Review Project, Medicare Evidence Development and Coverage Advisory Committee, the National Institute for Health and Clini- cal Excellence, National Institutes of Health Office of Medical Applications of Research, and the U.S. Preventive Services Task Force.
86 INITIAL NATIONAL PRIORITIES FOR CER TABLE 4-2â Criteria and Priorities for Quality Improvement Criteria Priority List Impactâdisease burden (disability, â¢ Asthma mortality, and economic costs affecting â¢ Cancer screening that is evidence-based patients, families, communities and â¢ Care coordination society) â¢ Children with special health care needs â¢ Diabetes Improvabilityâthe likelihood that â¢ End of life with advanced organ system systemic changes in health system could failure improve priorities in the six quality aims â¢ Frailty associated with old age listed in the Quality Chasm report â¢ Hypertension â¢ Immunization Inclusivenessâequity (across ages, races, â¢ Ischemic heart disease gender, and socioeconomic status), â¢ Major depression representativeness (across spectrum of â¢ Medication management healthcare conditions); reach (across â¢ Nosocomial infections spectrum of healthcare settings and â¢ Pain control in advanced cancer providers); also, later included four care â¢ Pregnancy and childbirth domains (preventive, acute, chronic, and â¢ Self-management/health literacy palliative), or crosscutting â¢ Severe and persistent mental illness â¢ Stroke â¢ Tobacco dependence treatment in adults â¢ Obesity SOURCE: IOM (2003). External Priority-Setting Initiatives In addition to the two IOM reports, the CER committee reviewed the following external priority-setting initiatives to select the condition-level criteria and priority topic-level criteria: AHRQâs Effective Health Care Program, which identified topics for comparative effectiveness systematic reviews (Whitlock et al., 2009); Healthy People 2010, an alliance of na- tional and state public health agencies that developed a list of leading health indicators and priority focus areas to set a prevention agenda for the nation (HHS, 2000); the Cochrane Collaboration, which identified global priorities for Cochrane systematic reviews of public health topics (Doyle et al., 2005); the World Health Organization (WHO) Advisory Committee, which reviewed the literature on priority setting for health care guidelines, recommendations, and technology assessments (Oxman et al., 2006); and the National Quality Forum (NQF), which convened the National Priorities Partnership (NPP) to set national priorities and goals for performance im- provement efforts in potentially fruitful areas (NPP, 2008); among others. These external priority-setting programs identified additional criteria, including appropriateness of a topic for CER research, feasibility of study design, potential for change, and potential risk from inaction (Whitlock et
THE CRITERIA AND PROCESS FOR SETTING PRIORITIES 87 al., 2009); research that is not duplicative (Doyle et al., 2005); priorities that are likely to improve quality of life and reduce health disparities (HHS, 2000); priorities that address the major challenges of eliminating harm and removing waste (NPP, 2008); and interventions that would likely require systems change (Oxman et al., 2006). Criteria chosen by the priority-setting organizations are summarized in Table 4-3. As described above, the com- mittee incorporated the appropriateness of the topic for CER and informa- tion gaps and duplication into the priority topic-level criteria. The Committee on Comparative Effectiveness Research Prioritization also considered the methodology used by the other priority-setting groups to arrive at its final priority CER topics including the following: â¢ The creation of a specific taskforce or committee to oversee and ultimately vote on the priority questions â¢ invitation for stakeholders to submit comments and priority An agendas via written or oral testimony to committee members â¢ The establishment of explicit priority criteria (and gathering of data sources/information relevant to criteria) on which the committee members were to base their decisions â¢ process to refine submitted questions and gain feedback on the A revisions â¢ recommendation to continuously evaluate and improve upon the A inherently dynamic and subjective priority-setting process TABLE 4-3â A Variety of Priority-Setting Initiatives and Their Selected Criteria Name Study/ Priority List Source Criteria Identifying, Selecting AHRQ â¢ Appropriatenessâapplies to and Refining Topics Medicare and/or Medicaid for Comparative populations, HHS priority Effectiveness Systematic condition Reviews: AHRQ and â¢ Importanceâdisease burden, Effective Health Care cost, strong stakeholder support, Program uncertainty or controversy surrounding issue â¢ Desirability of New Research/Duplication â¢ Feasibility â¢ Potential Valueâpotential for significant health and economic impact, change, and risk of inaction continued
88 INITIAL NATIONAL PRIORITIES FOR CER TABLE 4-3â Continud Name Study/ Priority List Source Criteria Healthy People 2010 U.S. Department of â¢ Goal 1: Increase Quality and Health and Human Years of Healthy LifeâLife Services (HHS) expectancy, quality of life: looked at global assessments, healthy days, and years of healthy life â¢ Goal 2: Eliminate Health DisparitiesâIn terms of gender, race and ethnicity, income and education, disability, geographic location, and sexual orientation â¢ Leading Health Indicators considered when choosing focus areasâphysical activity, overweight and obesity, tobacco use, substance abuse, responsible sexual behavior, mental health, injury and violence, environmental quality, immunization, access to health care National Priorities and NPP/NQF â¢ Eliminating harm Goals: Aligning Our â¢ Eradicating disparities Efforts to Transform â¢ Reducing disease burden Americaâs Healthcare â¢ Removing waste Improving the Use of WHO â¢ Problems associated with a high Research Evidence in Advisory Committee burden of illnessâin low- and Guideline Development: on Health Research middle-income countries, or new 2 Priority Setting and emerging diseases â¢ existing guidelines or No recommendations of good quality â¢ Feasibility of developing recommendationsâthat will improve health outcomes, reduce inequities or reduce unnecessary costs if they are implemented â¢ Implementation is feasibleâwill not exhaustively use available resources, and barriers to change are not likely to be so high they cannot be overcome â¢ Interventions that will likely require systems changes â¢ Interventions where there might be a conflict in choices between individual and societal perspectives
THE CRITERIA AND PROCESS FOR SETTING PRIORITIES 89 TABLE 4-3â Continued Name Study/ Priority List Source Criteria Global Priority The â¢ Burden of disease, magnitude of Setting for Cochrane Cochrane Health problem, urgency Systematic Reviews of Promotion and Public â¢ Importance to developing Health Promotion and Health Field countries Public Health Research â¢ Avoidance of duplication â¢ Opportunity for action Recommendations for Healthy People 2020 â¢ The overall burden of the risk the Framework and (HHS) factor or disease Format of Healthy â¢ The extent the burden may be People 2010 preventable or reducible â¢ Cost-effectiveness of alternate opportunities â¢ The net health benefit â¢ The synergy of different interventions that target the same disease â¢ The likely timeframe to observe the impact â¢ The potential to reduce health inequities among populations â¢ The willingness of public health, private organizations, and other collaborating entities to address a particular health problem and to accept accountability for convening multisectoral stakeholders to effect changes in these areas. SOURCES: Doyle et al. (2005); HHS (2000, 2008); NPP (2008); Oxman et al. (2006); ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ Whitlock et al. (2009). Several groups used a two-step process to arrive at their ultimate list. For example the AHRQ program first decided on broad priority areas and then selected specific research questions that fit in those areas (Whitlock et al., 2009). Similarly, the Cochrane Collaboration first chose a list of eight broad priority topics and then formulated a longer list of more specific review priorities (Doyle et al., 2005). The committee incorporated the lessons learned by the IOM and ex- ternal priority-setting projects whenever possible, including maintaining a transparent and systematic process, involving stakeholder input, and using many of their suggested criteria. The committee also recognized that each
90 INITIAL NATIONAL PRIORITIES FOR CER priority-setting enterprise has its own unique needs, and so it developed its own priority-setting process and criteria to meet the needs specified in the ARRA legislation. voting procedures The committee used three rounds of voting, illustrated in Figure 4-1, to establish the CER priority topics listed in Chapter 5. Each round was conducted using a web-based data entry system. The first round included the 1,268 unique nominated research topics submitted by the public and screened by the committee for nonresponsive and verbatim duplicates (as discussed in Chapter 3). In this round, the committee was divided into five groups and given 3 days to vote. Each group voted on the nominated re- search topics categorized into several of the 32 unique research areas, with each group voting on approximately 20 percent of the total nominated research topics, and each committee member voting independently. This design was intended to ensure that the leading nominated research topics from each of the 32 research areas were likely to be retained in the next round of voting, thus preserving the balance of the portfolio. Committee members were given points equal to the number of nomi- nated research topics assigned to their group. They were required to allocate all of their points, and they could give as many as 10 percent of those points to any one topic. This design allowed committee members to indicate the strength of their preference for specific topics. The score for each nominated topic was tallied as a percentage of allocatable points received, because the five groups had an unequal distribution of topics and, thus, of points. After the initial voting, raw data distributions of scores were reviewed by the committee without knowledge of the topics or portfolio distribution. Of the 1,268 that were voted on, 200 topics received at least 1 percent of the available points and included 60 percent of the topics that received at least one point by any committee member. The committee concluded that the top 200 nominated topics represented a natural statistical break for the second round of voting. Three clinicians on the committee reviewed these 200 topics again for duplications. Any topic that all three clinicians agreed was a duplicate was removed from consideration for the second round of voting. Fifty-five top- ics were consolidated in this process. The committee assessed the remaining 145 topics against the portfolio criteria and determined that the portfolio was sufficiently balanced across research areasâonly 3 of 32 designated research areas were eliminated (see Table 5-1 in the next chapter). In the second round of voting, committee members received the scores from the first round of voting for each of the nominated topics, as well as expanded information on research and funding gaps related to those topics.
THE CRITERIA AND PROCESS FOR SETTING PRIORITIES 91 Stakeholder questionnaire open to public March 6-27, 2009 2,606 recommended CER topics received from 1,758 respondents Initial screen April 2-6, 2009 (removal of nonresponsive topics and verbatim duplicative topics) Round 1 Voting = 1,268 Nominated Topics Nominated topics voted on by five voting groups organized by clinical areas April 10-13, 2009 Round 1 Results = 200 Nominated Topics Substantive duplicate topics removed Round 2 Voting = 145 Nominated Topics Committee Deliberation (April 14-16, 2009) In-depth discussion and review, consolidation of list to 129 topics, discussion of overall portfolio considerations, and nomination of 26 new topics to fill gaps in research areas resulting in Round 3 Voting = 155 Nominated Topics Committee voted April 19-20, 2009 Round 3 Results = Final 100 Priority Topics FIGURE 4-1 Voting process and selection of priority topics. Figure 4-1 R01511 vector, editable
92 INITIAL NATIONAL PRIORITIES FOR CER Unlike the first round of voting, in the second round all committee members voted on all 145 remaining nominated topics. Each committee member was allocated 145 total points and could give a maximum of 14 points to any individual topic. At a 3-day retreat, the entire committee reviewed and discussed the nominated topics in priority order following the second round voting. The leading topics were discussed in detail to clarify the topics. The discussion also allowed committee members to share their opinions and expertise about the individual nominated topics. Through the discussion process, the committee combined 16 of the 145 nominated topics and expanded several of them beyond the scope of their original condition or population. For example, topics addressing mental health issues or obesity in adults were extended to include children and adolescents. Health care delivery topics examining care of a single chronic disease were expanded to cover multiple chronic diseases. In general, the committee decided that it was useful to broaden the nominated research topics at this stage because agencies spend- ing the CER funds will later be issuing their own more detailed requests for proposals as part of the grant applications process, and researchers will define the questions further when applying for those funds. At the retreat, the committee also reached consensus on topics to fill out or eliminate gaps in the portfolio representation. A total of 26 topics were nominated by the committee. These topics were incorporated into the 129 remaining submitted topics without distinguishing them, providing a total of 155 unique nominated research topics for consideration in the third round of voting. Committee members were allocated 300 total points for voting, with a maximum of 30 points allowed for any particular topic. Web-based voting took place over a day and a half. The raw scores were reviewed by the committee, and the distribution of the scores provided a natural cutoff at 100 topics, 18 of which had been proposed by the com- mittee. The top 100 topics all received a mean of at least 1.0 points. Topics that fell below this threshold received zero scores from at least 60 percent of the committee members. The final results of the voting process are pre- sented in Chapter 5. Lessons learned from the Current Prioritization Process and Committee recommendations The IOM committee developed several recommendations to set future priorities based on the experience of this project. Recommendation 1: Prioritization of CER topics should be a sustained and continuous process, recognizing the dynamic state of disease, inter- ventions, and public concern.
THE CRITERIA AND PROCESS FOR SETTING PRIORITIES 93 Health care is dynamic; new diseases and health needs can arise sud- denly and other health problems might become insignificant when a treat- ment is found. As new evidence is produced and gaps in evidence are diminished, CER will need to go in new directions. New scientific tools and techniques may open opportunities for CER where none previously existed. A continuous process is necessary to update funding priorities as conditions change and the impact of previous CER becomes evident. The criteria used in the prioritization process may also need to be evaluated on a regular basis. The prioritizing body may consider additional criteria, such as evaluating levels of uncertainty and potential for future funding from various stakeholders. Recommendation 2: Public (including consumers, patients, and care- givers) participation in the priority-setting process is imperative to provide transparency in the process and input to delineating research questions. An ongoing process of CER prioritization will need to engage the public more completely. Efficiency requires that representative stakeholder perspectives be engaged at the most critical time points rather than every possible step in the research process. With respect to prioritizing topics, there are two key stages. The first is in setting criteria for choosing topics and balancing the overall portfolio. While the committee has set forth cri- teria here, these criteria should be revisited to ensure that they reflect the publicâs goals and values. Recommendation 3: Consideration of CER topics requires the devel- opment of robust, consistent topic briefs providing background infor- mation, current practice, nd research status of the condition and its a interventions. âTopic nomination developmentâ is the second critical stage in the prioritization process for public input (Whitlock et al., 2009). Many po- tentially important nominations delivered through the web-based question- naire would have benefited from further development before voting so that the voters had a better idea of what motivated the nomination and the nominator and so that other contributors had a chance to expand on the patients, interventions, comparators, and outcomes to be considered by the research. In the future, the topic brief preparation process should be an interactive one in which the prioritizing body gains the perspective of the nominator and other stakeholders to better convey the context and main points of the nomination to the voters. The process should allow sufficient time to develop robust, consistent
94 INITIAL NATIONAL PRIORITIES FOR CER topic briefs for use in voting in order to âlevel the playing field.â The process should provide background information on the condition, address current practice and policy, and document existing research in order to al- low explicit consideration of each topic against pre-specified prioritization criteria. Recommendation 4: Regular reporting of the activities and recommen- dations of the prioritizing body is necessary to evaluate the portfolioâs distribution, its impact for discovery, and its translation into clinical care in order to provide a process for continuous quality improvement. The committee believes that increased transparency of the overall pro- cess and documentation of decision making for each topic would allow improved public participation and allow the public to revise and resubmit rejected research suggestions for future consideration. This type of transpar- ency is needed for any ongoing process to be responsive to public concerns and interests and to enhance its legitimacy. Thus, the prioritization process should produce regular reports evaluating its portfolio of potential and selected topics for CER against a variety of criteria, including type of ser- vice domain, clinical domain, population characteristics, and other policy priorities such as addressing vulnerable populations and health disparities. A rolling evaluation of theÂ selection and prioritization processes, as well as the return on investment of prior CER research by application throughout the health system should be incorporated in the prioritization process to ensure quality improvement. Ultimately, any prioritization process for a CER Program will be evaluated by the impact of the funded research on improving health decision making, health outcomes, and reducing unneces- sary variation in health care. References AHRQ (Agency for Healthcare Research and Quality). 2009. Medical Expenditure Panel Survey. http://www.meps.ahrq.gov/mepsweb/ (accessed March 10, 2009). Doyle, J., E. Waters, D. Yach, D. McQueen, A. De Francisco, T. Stewart, P. Reddy, A. M. Gulmezoglu, G. Galea, and A. Portela. 2005. Global priority setting for Cochrane sys- tematic reviews of health promotion and public health research. Journal of Epidemiology and Community Health 59:193-197. FAACT. 1997. The FACCT consumer information framework: Comparative information for better health care decisions http://www.facct.org/information.html (no longer accessible). HHS (Department of Health and Human Services). 2000. Healthy People 2010: Understand- ing and improving health. U.S. Government Printing Office. http://purl.access.gpo.gov/ GPO/LPS4217 (accessed April 3, 2009).
THE CRITERIA AND PROCESS FOR SETTING PRIORITIES 95 âââ. 2008. Recommendations for the framework and format of Healthy People 2020. http://www.healthypeople.gov/HP2020/advisory/PhaseI/sec5.htm#_Toc211942927 (ac- cessed June 12, 2009). IOM (Institute of Medicine). 2003. Priority areas for national action: Transforming health care quality. Edited by K. Adams and J. Corrigan. Washington, DC: The National Academies Press. âââ. 2008. Knowing what works in health care: A roadmap for the nation. Edited by J. Eden, B. Wheatley, B. J. McNeil and H. Sox. Washington, DC: The National Academies Press. Kung, H.-C., D. L. Hoyert, J. Xu, S. L. Murphy, and Division of Vital Statistics. 2008. Deaths: Final data for 2005. National vital statistics reports National Center for Health Statistics. Markle Foundation. 2008. FACCT legacy documents http://www.markle.org/archives/facct/ (accessed May 12, 2009). NPP (National Priorities Partnership). 2008. National priorities and goals. Washington, DC: National Quality Forum. Oxman, A., H. Schunemann, and A. Fretheim. 2006. Improving the use of research evidence in guideline development: 2. Priority setting. Health Research Policy and Systems 4(1):14. Wennberg, J. E. 2009 (unpublished). Recommendations to the Institute of Medicine on com- parative effectiveness research priorities. Submitted in response to a request from the Institute of Medicine Committee on Comparative Effectiveness Research Prioritization. The Dartmouth Institute for Health Policy and Clinical Practice. Whitlock, E. P., S. A. Lopez, S. Chang, M. Helfand, M. Eder, and N. Floyd. 2009. Identi- fying, selecting, and refining topics for comparative effectiveness systematic reviews: AHRQ and the effective health care program. http://effectivehealthcare.ahrq.gov/repFiles/ 20090427IdenttifyingTopics.pdf (accessed June 5, 2009).