In 1892 Ralph Waldo Emerson famously wrote in The Conduct of Life that “the first wealth is health.” Vaccines not only have profoundly improved health and conferred economic benefits to our societies but also have provided a range of other, broader advantages that are hard to capture. Thus, if one wishes to evaluate and prioritize vaccines, either those vaccines in use or those under development, then it is necessary to use a much broader framework than one based purely on health or cost-effectiveness indicators.
The Foundational Work on SMART Vaccines
At the request of the National Vaccine Program Office of the U.S. Department of Health and Human Services, the Institute of Medicine (IOM) initiated a sequence of projects in early 2011 to help provide guidance in prioritizing new vaccines for development. This effort has proceeded in three phases, each building on the key objective of the U.S. National Vaccine Plan: “Develop a catalogue of priority vaccine targets of domestic and global health importance” (HHS, 2011).
The Phase I report, Ranking Vaccines: A Prioritization Framework (IOM, 2012), introduced an analytical model that employed multi-attribute utility theory, a specific version of the general class of multi-criteria decision-analysis tools. The decision to use multi-attribute utility modeling represents an important change from the previous IOM approaches to prioritizing vaccines for development. A pair of reports from the mid-1980s selected a single attribute for ranking vaccines—infant mortality equivalents, or what would now be considered “life-years saved” (IOM, 1985, 1986). A subsequent report chose an entirely different metric—cost-effectiveness ratio—as the sole criterion for ranking vaccine candidates for development (IOM, 2000). Thus, the 1985–1986 studies used a direct
health benefit measure, and the 2000 study used an efficiency measure to produce lists of rank-ordered priorities.
The Phase I committee’s decision to use a multi-attribute approach was driven in large part by stakeholder feedback indicating that the narrow focus of the earlier studies limited the value of these tools to the many decision makers in the global vaccine community. The Phase I report discussed the testing of the multi-attribute utility model1 using data for hypothetical vaccines to prevent influenza, group B streptococcus, and tuberculosis in the United States and South Africa. The committee also presented the blueprint of a software system called Strategic Multi-Attribute Ranking Tool for Vaccines, or SMART Vaccines Beta. The multi-attribute utility model embedded in SMART Vaccines allows users to specify which attributes are of highest importance to them and also allows users to specify the amount of weight given to each selected attribute. This was a novel approach in an enterprise that had traditionally relied on priority lists.
The Phase II committee enhanced the model and conducted extensive testing using additional data for hypothetical vaccines for the prevention of pneumococcal infection, human papillomavirus, and rotavirus. A broad range of attributes intended to address a variety of stakeholder interests—28 attributes in total, plus 7 user-defined entries (see Table S-1)—were embedded in SMART Vaccines. This software version, which was programmed in a Matlab environment that was operational only with the Windows operating system, was released for public use in fall 2013. The Phase II committee issued specific guiding principles for the future development of SMART Vaccines in its report Ranking Vaccines: A Prioritization Software Tool (IOM, 2013).
Potential Applications of SMART Vaccines
SMART Vaccines is expected to be of use to a variety of decision makers in the public, private, nongovernmental, and other sectors of the vaccine enterprise. Specifically, those users could include ministries and departments of health involved in research, development, delivery, and preparedness efforts relating to new or existing vaccines; industrial manufacturers
1 The multi-attribute utility model embedded in SMART Vaccines consists of a computational submodel and a value submodel. Background information on these submodels, the overall modeling strategy, mathematical functions, and the associated assumptions (e.g., with costs and time horizon) are explained in Ranking Vaccines: A Prioritization Framework (IOM, 2012). Information about the model refinements and testing are included in Ranking Vaccines: A Prioritization Software Tool (IOM, 2013). This information is not repeated in this report.
Choices of Attributes in SMART Vaccines 1.1
|Scientific and Business Considerations||
NOTE: DALYs = disability-adjusted life years; NGOs = nongovernmental organizations; QALYs = quality-adjusted life years.
interested in product profile improvements, among other aspects of vaccines and pharmaceuticals; and donor foundations and global and regional vaccine initiatives involved in or supporting vaccine-implementation programs.
This committee emphasizes a point that the Phase I and Phase II committees have already stressed: SMART Vaccines is only a decision-support system and not intended to be used as a decision maker. The Phase I committee recognized in its report that “a major use of SMART Vaccines
will be to facilitate discussions about attributes and values among diverse users, helping them to converge upon mutually beneficial priorities and collaborations” (IOM, 2012, p. 8).
Furthermore, this committee recapitulates the Phase I committee’s starting vision:
[V]arious organizations could use SMART Vaccines independently to guide their efforts in vaccine development and implementation. This might begin at the basic science level in organizations conducting and funding research to break through bottlenecks in vaccine development. Other potential users, such as manufacturers, might be involved directly in the development and eventual production of vaccines and thus may wish to emphasize an entirely different set of vaccine attributes (e.g., profitability, development and regulatory risks) compared to a basic research organization. Still some users or user consortia might use SMART Vaccines to enhance market stability (say, through pre-purchase agreements) and hence the likelihood of successful vaccine development. (IOM, 2012, p. 8)
Additionally, as the Phase I report noted:
SMART Vaccines can help diverse users understand how and why their rankings differ. Variations in rankings due to differing data inputs can be discussed among users to discover common data sources. When the model produces different results as a consequence of differing values, it can motivate discussions relating to individual or inter-institutional priorities among users. SMART Vaccines may also help inform users of the value of strengthening vaccine delivery methods (e.g., by augmenting the cold-chain capacity) and alternative methods of disease control (e.g., clean water supply, mosquito netting, food safety measures, or health-related education). A further expected benefit of using SMART Vaccines is that it will enable users to identify data needs to ultimately improve their vaccine prioritization process. Future data collection activities, surveillance activities, and resource allocation may be informed and planned by use of SMART Vaccines. (IOM, 2012, p. 8)
Enhancement of SMART Vaccines
This report, Ranking Vaccines: Applications of a Prioritization Software Tool, describes the Phase III work of the committee, which was established by the Institute of Medicine and the National Academy of Engineering to enhance SMART Vaccines 1.0 for prioritizing new preventive vaccines. In
particular, this project, which was commissioned by the U.S. Department of Health and Human Services’ National Vaccine Program Office in collaboration with the National Institutes of Health’s Fogarty International Center, focused on three tasks: (1) the evaluation of the software in four user-based applications, (2) the development of a general data framework for the software, and (3) the definition of next steps that would increase the use and value of SMART Vaccines. The tasks are described more fully in Box S-1. As a deliverable of this work, in addition to this report, the committee has released SMART Vaccines 1.1, software that contains several enhancements informed by use case scenarios and other stakeholder feedback. The same set of attributes used in SMART Vaccines 1.0 has been integrated in SMART Vaccines 1.1 (see Table S-1). The updated software is available for free download at www.nap.edu/smartvaccines.
Committee on Identifying and Prioritizing New Preventive Vaccines for Development, Phase III
Institute of Medicine National Academy of Engineering Statement of Task
Task 1: Evaluate the utility of and support for the vaccine prioritization software in the stakeholder community through four use case scenarios with four potential users of the software. The potential users will be identified in collaboration with the sponsors.
Task 2: Based on the use case scenarios, compile one new dataset for each of the two vaccine candidates to be compared per user. Develop a framework for a data warehouse and data estimation strategy to support the software.
Task 3: Release the datasets and a report containing recommendations for further development, maintenance, and dissemination of the software and the data warehouse.
User Groups and Use Case Scenarios
The committee shortlisted and finalized three user groups to explore three use case scenarios for Phase III. Each user group provided the committee its data for two diseases that it had chosen to evaluate. These datasets compiled by the user groups in conjunction with the committee are available upon request through the Public Access Records Office accessible from the Current Projects System page of the National Academies website.
A software usability expert from Microsoft Corporation conducted the use case studies with the user groups to report on their experiences and feedback for enhancing SMART Vaccines. The user groups were
- The Public Health Agency of Canada, which had a country-level goal of prioritizing new vaccine research and development. It focused its initial efforts in the use of SMART Vaccines on chlamydia and tuberculosis.
- The New York State Department of Public Health, which had a goal not of prioritizing new vaccine development, but of refining the advice it provides to health care providers concerning which of multiple vaccines already available was best suited to use in various populations of New York State. Specifically, the department analyzed two existing vaccines available for vaccinating infants against rotavirus.
- The Serum Institute of India, which had a manufacturing focus on dengue and respiratory syncytial virus vaccines. As a provider of low-cost vaccines for use in many countries besides India, the Serum Institute sought to use the software also to enhance its understanding of potential vaccine markets beyond India.
In addition to these user groups, the committee also worked with two officials from Mexico’s Ministry of Health who served as advisory consultants in exploring the use of an early version of SMART Vaccines 1.1 to compare the value of two existing influenza vaccines from a policy perspective.
For the fourth use case scenario, with the sponsors’ encouragement, the committee tested the use of SMART Vaccines as a mechanism to determine new target product profiles. As a specific case, the committee started with initial SMART Scores and worked backward to understand the impact of different formulations on the desired objective of a hypothetical user. The committee describes this scenario using the case of three pneumococcal vaccine candidates for South Africa. The data for this evaluation were previously synthesized by the Phase II committee members and were available as preloaded information in SMART Vaccines. The use of SMART
Vaccines as a tool to reverse-engineer new vaccine formulations, including, for example, changes in cost, coverage, effectiveness, and the required number of doses is explored in this report.
Lessons from the User Group Studies
The three user groups, and the officials from the Mexican Ministry of Health who served as advisory consultants, fully understood that they were using a preliminary and evolving version of SMART Vaccines and that their feedback was to be applied toward improving the product. As a consequence, none of them attempted to use the software for actual decision making, but rather used the occasion to explore the software, both for their potential future use and to assist in the IOM’s unique product development effort.
In none of the use case scenarios did the users actually develop their official sets of attributes to be used in vaccine evaluation or the formal weights to be attached to those attributes. Moreover, the committee found areas where the data from the user groups were either incomplete or inaccurate, further emphasizing that the results were not real decision support but rather familiarization with the SMART Vaccines tool and its potential uses.
In this report, the committee has summarized key lessons learned beginning with the broadest policy issues and then shifting to more narrow issues in application of SMART Vaccines.2 The committee was highly encouraged by the general positive feedback concerning SMART Vaccines and was especially reassured about the user groups’ imagination to potentially expand the tool for broader use scenarios.
The committee was charged with developing a framework for a data warehouse and for a data estimation strategy to support the software. The committee’s response to this task comes in three parts.
The committee had addressed this charge in several ways. First, the committee has discussed extensively the importance of a permanent home for SMART Vaccines and the roles of an active network of users and developers supporting the software. One of the key roles of such a group would be the development, curation, and improvement of data to use in SMART Vaccines and its future versions.
2 The key insights are summarized in Table 3-1 in Chapter 3 of this report.
Second, the committee has presented three ways in which data could be developed: (1) on a user-by-user basis, (2) with large-scale external funding to develop the data in bulk, and (3) by crowdsourcing the data development, particularly with the intent of engaging students in course work relating to public health strategic planning. Students could both use SMART Vaccines in the course work and develop new data that could be entered into a data warehouse supported by the community of users.
Third, the committee has discussed sources for the most complex data, including discussing the sources used for national and state populations integrated in SMART Vaccines 1.1. The committee has also explored sources for the important category of health outcomes data, identifying and discussing sources from the World Health Organization, the World Bank, and the Institute for Health Metrics and Evaluation. One specific enhancement the committee made was to revise the process by which health care cost data will be entered into SMART Vaccines; the new process provides several different pathways to obtain such data and enter them into the system.
Overall, the committee believes that the community of users will be best served by having the host organization manage a central data warehouse, organized through a data quality control mechanism and a well-supported relational database management system. This system would allow input from users and data creators through a standard spreadsheet format and would also create output reports that feed directly into SMART Vaccines to add new data—demographics, disease burden, treatment cost, and vaccine characteristics.
Guiding Principles for Enhancing the Value of SMART Vaccines
With the interest of guiding further efforts in the enhancement of SMART Vaccines, the committee began with a guiding principle established in the Phase II report: “SMART Vaccines will have the greatest potential and value if it is programmed as a dynamic, continuously evolving software application and made freely available in an open-source environment to all decision makers and developers around the world.”
As Phase III comes to a close, it has become all the more apparent to the committee that a transition strategy to a permanent home for SMART Vaccines is necessary for the ongoing use, enhancement, and even the survival of the software as a tool for strategic planning. Correspondingly, the committee believes that the National Vaccine Program Office and the Fogarty International Center of the National
Institutes of Health will be best served if they promptly create a process to facilitate the transition of SMART Vaccines to a permanent home. They could do this by convening a group of relevant stakeholders in vaccine research, development, deployment, funding, and policy to establish a process for soliciting applications for the permanent host—an individual organization or a consortium—and a method for evaluating and choosing among the candidates.
The Phase II report indicated—and this committee strongly agrees—that the ultimate future applications and benefits of SMART Vaccines depend on the strengths of the organization or consortium that becomes the permanent host. Based on the committee’s analyses of possibilities, it emphasizes the importance of a host organization that is both neutral among many users’ competing viewpoints—and clearly viewed as such—and is well equipped with organizational and technological capabilities. The committee believes that the best hosting organization will not only have a significant international presence and reputation, but also best serve the user community if it is a—or partners with a—research-intensive institution of higher education.
The committee believes that research universities, especially those with a global public health focus, can bring assets to the forefront, including the ready availability of professional expertise pertinent to the endeavor such as the ability to create training modules involving the use of SMART Vaccines at multiple levels and the ability to involve health science and policy students in the production of new or updated datasets for use with SMART Vaccines in a crowdsourcing approach to a broader, globally useful data warehouse development.
In addition to stressing the importance of promptly creating a major outreach and educational effort to expand the awareness and use of SMART Vaccines—hence expanding the user community in parallel—the committee also identifies a set of desirable research activities and a suite of potential extensions of the capabilities of the software into a wider array of decision making.
Furthermore, in terms of data requirements for SMART Vaccines, the committee observes that to carry out any vaccine prioritization task sensibly, decision makers would necessarily have to have these same data in hand, with or without the software tool. Without these basic data, the decisions cannot be made as carefully. The data requirements may seem to loom large in the eyes of potential users, but the software itself does not create the data burden—it merely brings it to the forefront. Once the data are assembled, SMART Vaccines provides a useful tool to enhance decision making that has a significant data-driven basis.
The committee emphasizes the importance of ultimately creating a successor version of SMART Vaccines that is fully Internet-based, rather than relying on the Matlab approach used in its development of the software prototypes. A Web-based program, linked to a centrally maintained data repository, would avoid the challenges associated with the installation of SMART Vaccines in some environments and would make it platform-neutral, rather than limiting its use to those with the Windows operating system, as the current version requires.
A Sense of Focus and Urgency
Over the years, new vaccine development efforts have become extremely expensive compared with earlier vaccine development successes. In times of financial duress, governmental and corporate priorities often shift their approaches toward those that yield greater returns. Because the impacts of vaccines are multi-generational and thus very hard to capture analytically, vaccines are often compared against other lucrative products, for example, in the realm of therapeutics. Unfortunately, standard analytic tools have trouble dealing with some important aspects of vaccines.
Thus, the need has never been greater for systematic evaluations of potential vaccine targets to help guide the discussion among users, purchasers, and developers of vaccines concerning where best to focus new development efforts. Yet the plethora of approaches available to prioritize vaccine development—many of which are quite opaque—creates its own risk: Something new is not always welcome and, even if it is welcome, often not embraced and nurtured. SMART Vaccines offers a first-of-its-kind platform—a decision-support tool and a discussion facilitator that uses a range of attributes that were previously unavailable for analyses in a single tool and for a wide range of decision makers.
The initial user group evaluations and the positive feedback that the software has generated offer great confidence to the committee about the potential applications and extensions of SMART Vaccines for global public health. The committee understands the challenges and opportunities in the pathway of data development for SMART Vaccines—and for vaccine prioritization in general. The promise of SMART Vaccines depends on the commitment of its past and future sponsors, and of a network of users, developers, and advisors. The committee encourages these stakeholders to have a sense of focus and urgency.