Over the centuries, from Edward Jenner to Bill Gates, as our scientific understanding of diseases has increased, so has the focus on prioritizing new vaccines to help achieve better health. Despite the expanding interest in and support toward improving global health, constraints inherent to vaccine development and delivery present decision makers with difficult choices. Given the lack of effective tools and models to assist the decision-making process, renewed attention is needed to improve the approaches available for priority setting and for guiding investment decisions.
Prioritizing vaccines—“arranging in the order of relative importance”—is a time-and resource-intensive process requiring diverse considerations. Examples of such considerations include the emergence and reemergence of disease threats, limits in the progress of research related to the disease in question, technological feasibility, economic and other resource constraints, possibilities for enhancing vaccine administration methods, and other broader objectives. Decision makers involved in setting priorities come from different constituencies with different perspectives. Therefore, it becomes vitally important to develop not only a practical approach that provides a common language to assist decision making but also a flexible tool that embraces a wide spectrum of inputs and perspectives in efforts to advance vaccine development.
This report, Ranking Vaccines: A Prioritization Framework, describes a decision-support model and the blueprint of accompanying software being developed to help prioritize vaccines. The consensus study that produced this report is being carried out in two phases. Phase I work, described in this report, provides the conceptual underpinning of Strategic Multi-Attribute Ranking Tool for Vaccines, or SMART Vaccines. SMART Vaccines Beta, developed by the committee in Phase I, is not available for public use. SMART Vaccines 1.0 is expected to be released at the end of Phase II, when it will be fully operational and capable of guiding discus-
sions about prioritizing the development and introduction of potential new vaccines. In the committee’s view, a “new vaccine” (or “vaccine candidate”) can refer not only to a completely novel vaccine but also to an existing vaccine given improvements to some of its features, including innovations in its production or delivery methods.
The audience and potential users of SMART Vaccines include those institutions funding and carrying out basic biomedical research, private firms involved in vaccine production, philanthropic foundations with a strong interest in vaccination and global health programs, international health organizations, and high-level decision makers, such as ministers for health, commerce, and finance or senior administrators.
The committee’s charge
Phase I of the study was supported by the National Vaccine Program Office of the U.S. Department of Health and Human Services. The Phase I statement of task is presented in Box S-1. Phase II of the study is oriented toward expanding and enhancing the capabilities of the model and transforming SMART Vaccines Beta to SMART Vaccines 1.0.
This report describes the committee’s approach toward demonstrating a proof of concept using three hypothetical vaccine candidates that have not yet been developed. The committee included a broad range of attributes that represent the various perspectives relating to vaccine development and impact. Some of the data for these attributes are readily available (such as population characteristics), while other data are estimated by the user (e.g., qualitative attributes of the vaccines) or through expert opinion (e.g., disease burden or cold-chain requirements).
Because the data inputs in this report were not intended to be precise, readers should not take any output of SMART Vaccines Beta as the “exact” or “recommended” priority value relating to any particular vaccine; instead the outputs should be seen only as illustrative examples of how the model and beta software currently operate.
Previous Institute of Medicine reports
Previous Institute of Medicine (IOM) studies from 1985–1986 and 2000 that focused on vaccine prioritization provided specific lists of vaccine ranks. The two-volume IOM study New Vaccine Development, released in 1985–1986, prioritized vaccines both for the United States and from an
Committee on Identifying and Prioritizing New Preventive
Vaccines for Development
Institute of Medicine
Statement of Task
Task 1: Review domestic and global research and development prioritization activities relevant to identifying new preventive vaccine targets.
Task 2: Develop an analytical framework and model for prioritizing vaccines of domestic and global importance. Engage stakeholders to inform the process of the model development and implementation.
Task 3: Test and validate the model using two to three predetermined vaccines, including at least one vaccine candidate of domestic importance and one of global importance.
Task 4: Prepare a report containing the analytical framework and model for evaluating and prioritizing vaccine targets along with recommendations as to how to use the model for reviewing the catalog of preventive vaccines every 2 to 3 years.
international perspective, based on infant mortality equivalents—a proxy measure of health burden.
The 2000 report Vaccines for the 21st Century focused entirely on the U.S. population and, unlike the 1985–1986 report, used an efficiency measure for ranking vaccines: incremental cost per incremental quality-adjusted life years saved ($/QALY), a measure derived from a classic welfare economics model. The cost-effectiveness model of the 2000 report represented important progress toward vaccine prioritization, but it did not provide guidance for answering some challenging questions often encountered in decision making. For instance, the model provided no guidance on how to choose between two diseases with equal QALYs when one was a low-impact disease affecting the majority of the population and the other a disease with few cases but with very high mortality and potential large-scale social disruption.
While both of the earlier reports noted that vaccine prioritization can include aspects of social value beyond net costs (or savings) and health burden reduction, these variables were considered to be beyond the scope of the cost-effectiveness or infant-death-equivalents-prevented framework. SMART Vaccines significantly expands the single criterion framework of the earlier prioritization efforts to include a number of additional criteria that influence decision making in vaccine development.
An overview of SMART Vaccines
The committee’s principal contributions have been broadening the set of criteria for valuing preventive vaccines and demonstrating how the selection of criteria and data can influence the prioritization process. Users are offered a choice of up to 29 attributes drawn from broad categories which include health burden considerations, economic considerations, demographic considerations, public concerns, scientific and business considerations, programmatic considerations, and policy considerations. Table S-1 presents the general list of attributes influencing the rank of vaccine candidates in SMART Vaccines.
Because decision makers may represent different constituencies, their criteria for prioritizing various vaccine candidates are likely to differ as well. Further, each of these selected criteria can be valued and weighed differently in the prioritization process. Thus, not only does SMART Vaccines broaden the scope of the valuation criteria, but it also allows users to select and weigh criteria according to their values or those of the communities they represent.
From the technical standpoint, SMART Vaccines Beta expands the utility function for evaluating vaccines compared to the models published in the earlier reports. But the fact that different users may make different choices when using SMART Vaccines adds further value: It provides a framework to compare, discuss, and perhaps reconcile differing priorities. Thus, rather than pre-specifying which criteria are used and how they should be weighed, the committee has opted to allow the users to select their own.
Model and software development
The modeling strategy of the committee was based on multi-attribute utility theory. The multi-attribute utility approach has a well-grounded theoretical basis, but employing the theory for SMART Vaccines presented various challenges. The report discusses how the committee sought to tackle
Choices of Attributes in SMART Vaccines Beta
• Premature Deaths Averted per Year
• Incident Cases Prevented per Year
• QALYs Gained or DALYs Averted
• One-Time Costs
• Annual Net Direct Costs (Savings) of Vaccine Use
• Annual Net Workforce Productivity Gained
• Benefits Infants and Children
• Benefits Women
• Benefits Socioeconomically Disadvantaged
• Benefits Military Personnel
• Benefits Other Priority Population
• Availability of Alternative Public Health Measures
• Potential Complications Due to Vaccines
• Disease Raises Fear and Stigma in the Public
• Serious Pandemic Potential
Scientific and Business Considerations
• Likelihood of Financial Profitability for the Manufacturer
• Likelihood of Successful Licensure in 10 Years
• Demonstrates New Production Platforms
• Existing or Adaptable Manufacturing Techniques
• Potential Litigation Barriers Beyond Usual
• Interests from NGOs and Philanthropic Organizations
• Potential to Improve Delivery Methods
• Fits into Existing Immunization Schedules
• Reduces Challenges Relating to Cold-Chain Requirements
• Eradication or Elimination of the Disease
• Vaccine Raises Public Health Awareness
• Special Interest for National Security, Preparedness, and Response
• Advances Nation’s Foreign Policy Goals
these challenges throughout the model and software development and evaluation process.
Early prototypes were modeled after the one presented in the 2000 report. The committee then began the development of a user-friendly software interface to enable data input with the aim of incorporating sensitivity testing, advanced dynamic modeling, and improved visualization of results in the future. As mentioned earlier, this software will be available for public use at the end of Phase II. This report provides illustrative
screenshots of SMART Vaccines Beta, which is currently under development. The committee also engaged consultants to serve as concept evaluators to help improve the design and features of SMART Vaccines from the perspective of potential users.
SMART Vaccines uses two submodels—a computational submodel and a value submodel—to combine the levels of various attributes into a single measure of priority “score” for each vaccine under consideration. The weights used for criteria in the model must satisfy a number of conditions in order for the model to work properly. Normally, satisfying these conditions would require users to make many explicit quantitative value comparisons. To minimize these demands on the user in the current version of the model, the committee adopted the rank order centroid method to approximate additive multi-attribute utility weights. The only requirement that this method places on users is that they rank order the importance of attributes selected for their prioritization model. The rank order information is used to derive numerical weights which are then used in a scoring function. This approach is known to produce weights that are robust and predictive of the users’ eventual decisions. SMART Vaccines Beta permits only an ordinal ranking of the vaccine attributes with no tie scores.
The committee selected three diseases for evaluation: influenza, tuberculosis, and group B streptococcus. These diseases were compared between two countries, the United States and South Africa. Representative test results are discussed in this report with the acknowledgement that sensitivity testing and further validation will be required in Phase II of this study.
To demonstrate the extent to which the selection and ranking of attributes affects the priority scores among vaccines generated by the model, the committee conducted a “value experiment” in which committee members and staff selected attributes and provided ranking scores for six hypothetical vaccines: an influenza vaccine with a 1-year immunity; an influenza vaccine with a 5-year immunity; a tuberculosis vaccine with a 3-year immunity; a tuberculosis vaccine with lifetime immunity; an influenza vaccine with a 1-year immunity but with 50 percent increased coverage; and a tuberculosis vaccine with a 3-year immunity but in a setting with a 100-fold increase in disease prevalence. The results of this experiment, as described in this report, show how each user’s selection and weighting of attributes shifted the final rankings among these six hypothetical vaccines. The purpose of this experiment was to emphasize both the importance of the attribute-weighting process in the final rankings and the sensitivity of the ranks to preferences inherent in the decision-making process.
SMART Vaccines Beta requires substantial data inputs from users. In some cases, depending on the country for which the model is employed, the data required to drive the model may be sparse or unavailable. The usefulness of SMART Vaccines will rely upon concerted data collection and future software enhancements.
The model requires refined age- and sex-specific population data; these can generally be imported from the World Health Organization and other existing data sources. SMART Vaccines Beta also requires quantitative inputs concerning age-and sex-specific disease burdens to the population of interest, typical patterns of vaccination and health care use (and their costs) for relevant illnesses with and without the availability of a preventive vaccine, and health complications that might arise from the use of a new vaccine. These data are not widely available at this time and will likely have to be provided at least in part by processes led or guided by expert opinion. “Expert opinion” in this context refers to input from someone who is able to provide knowledgeable, informed estimates about the data needed within the country or region of interest. Economic data are also needed on typical wage rates for workers in each age group in order to compute worker productivity gains achieved by reducing or eliminating disease burden—both in workers directly and, indirectly, in children they may care for—through vaccination.
The model’s computational engine uses all data and other user-supplied entries to calculate a series of attributes, including cost-effectiveness, premature deaths averted, incident cases prevented, annual health care costs saved, and net annual gains in worker productivity. These quantities are computed through detailed modeling of the disease and its prevention through vaccination in the population over time.
SMART Vaccines Beta also allows users to specify qualitative attributes for each potential new vaccine, features that are not captured within the computed attributes, and add additional new attributes per their choice. These include, by general category, attributes focusing on the ability of existing health infrastructures to deliver the new vaccine; whether the vaccine has the capability of disease eradication; whether the vaccine targets major population health risks (such as pandemic diseases or bioterrorism attacks); and the likelihood of successful development, which in turn hinges on the likelihood of scientific progress and regulatory approval. Potential users of SMART Vaccines will have the option to include or not include any of these attributes in generating their final priority ranking; obviously, if an attribute is not used in generating the priority ranking, that obviates the need to provide related data for the candidate vaccines.
Ways to use (and not to use) SMART Vaccines
By design, SMART Vaccines offers users considerable flexibility in specifying attributes and their rank order to determine the final prioritization score. Among other things, this means that SMART Vaccines does not produce one unique list of priorities among vaccine candidates, unlike the techniques in the predecessor IOM reports in 1985–1986 and 2000. The rankings are sensitive to the choice and the order of attributes and to the trade-offs the user is willing to accept in determining priorities.
SMART Vaccines does not “make decisions.” It is intended to be used exclusively as a decision-support tool and only that. The committee expects that a major use of SMART Vaccines will be to facilitate discussions about attributes and values among diverse users, helping them to converge upon mutually beneficial priorities and collaborations.
The committee envisions that various organizations could use SMART Vaccines independently to guide their efforts in vaccine development and implementation. This might begin at the basic science level in organizations conducting and funding research to break through bottlenecks in vaccine development. Other potential users, such as manufacturers, might be involved directly in the development and eventual production of vaccines and thus may wish to emphasize an entirely different set of vaccine attributes (e.g., profitability, development and regulatory risks) compared to a basic research organization. Still some users (or user consortia) might use SMART Vaccines to enhance market stability (say, through pre-purchase agreements) and hence the likelihood of successful vaccine development.
SMART Vaccines can help diverse users understand how and why their rankings differ. Variations in rankings due to differing data inputs can be discussed among users to discover common data sources. When the model produces different results as a consequence of differing values, it can motivate discussions relating to individual or inter-institutional priorities among users. SMART Vaccines may also help inform users of the value of strengthening vaccine delivery methods (e.g., by augmenting the cold-chain capacity) and alternative methods of disease control (e.g., clean water supply, mosquito netting, food safety measures, or health-related education). A further expected benefit of using SMART Vaccines is that it will enable users to identify data needs to ultimately improve their vaccine prioritization process. Future data collection activities, surveillance activities, and resource allocation may be informed and planned by use of SMART Vaccines.
Observations and next steps
This report is intended to introduce potential users to the concept of SMART Vaccines and to encourage stakeholders to inform the development of SMART Vaccines 1.0 in Phase II of this study. The committee will next enhance SMART Vaccines Beta, test its use with three additional vaccine candidates of domestic and global importance, and further improve the user interface as part of the development of SMART Vaccines 1.0.
The value of SMART Vaccines will depend, in part, on data that need to be generated as vaccine candidates evolve and as disease epidemiology becomes better characterized in different parts of the world. In the future—beyond Phase II—an active community of users and an open-source environment would likely lead to future enhancement of the SMART Vaccines’ capabilities. Potential enhancements could include creation and sharing of databases for populations from different countries, the enhancement of validation tools and user interface, and the development of ways to address the risk and uncertainty surrounding the characterization of vaccines that have not yet been developed. This study is the first step in moving toward these goals.