4
Observations and Looking Forward
Previous efforts in the realm of prioritizing new vaccines have been limited because of the small number of attributes and value preferences those models were able to accommodate. Moving from the traditional single-attribute modeling framework to a multi-attribute modeling framework involved substantial public input and a wide range of expertise in order to develop the computational model and the user interface that support SMART Vaccines.
As noted in the 2012 Institute of Medicine (IOM) report Ranking Vaccines: A Prioritization Framework (IOM, 2012) and earlier in this edition, the committee’s charge did not include the production of a “list” of vaccines for development. Instead, through this work the committee developed SMART Vaccines as a dynamic, customizable decision-support tool to guide planning for the future development of preventive vaccines. It can also be used to select among existing vaccines for investment in a public health setting or to support other vaccine-related investment and implementation decisions. SMART Vaccines should always be used as a tool to discuss values and preferences across various user groups and not as the ultimate determinant of a decision that needs to be made.
Guiding Principle and Strategy
To inform future efforts in this area, the committee adopted a guiding principle: SMART Vaccines will have the greatest potential and value if it is programmed as a dynamic, continuously evolving software applica-
tion and made freely available in an open-source environment to all decision makers and developers around the world.
As a strategy relating to this principle, the committee believes that it would be most beneficial if the National Vaccine Program Office of the Department of Health and Human Services identifies a host for SMART Vaccines and its future versions. Moreover, any decision-support system such as SMART Vaccines has no intrinsic value without accurate and relevant data. Consequently, the committee places the highest importance on creation of a data architecture and expanding data collection for use in SMART Vaccines.
Consistent with this guiding principle and strategy, the committee believes that SMART Vaccines will achieve optimal value if the following events occur:
1. SMART Vaccines and its future versions are hosted in an open-source setting on a widely trusted website with a distinct identity and appropriately protected from unwarranted modifications or intrusions.
2. The host organization creates, maintains, funds, and facilitates a community of users to curate and manage further development of SMART Vaccines and supporting data.
3. The community of users includes decision makers involved in research, development, regulation, and implementation of new vaccines as well as developers with expertise in such areas as modeling, epidemiology, demography, software engineering, database management, and visual design.
4. The community of users—independently or in collaboration with the host organization—establishes an advisory group to help plan future versions and facilitate the adoption of SMART Vaccines.
5. The community of users, together with the host and sponsors, develops mechanisms to encourage the development and updating of data for populations at regional, national, and sub-national levels; for the disease burdens these populations confront; for the costs of preventing and treating those diseases in each distinct environment; and for the productivity losses associated with these disease burdens. These data are accessible in a standardized format, shared with other users through the common website that
hosts SMART Vaccines, and improved through an editing process agreed upon and overseen by the user community. These processes could ultimately help guide improvements in global communication and coordination of data and initiatives of common interest and shared importance.
6. The community of users studies the application of SMART Vaccines for retrospective analysis, validation, or confirmation of previous decisions relating to new vaccine development. The results would have both an educational and a continuous learning benefit.
No software product remains static—it either evolves or becomes obsolete, as examples from every realm of software development demonstrate. This remains true even when a single software system dominates a market, as in word processing, spreadsheet, graphical presentation, relational database management, or statistical analysis. In light of the continuous cycles of improvement required for SMART Vaccines, the committee developed a set of steps to guide future development, which are listed in priority order here:
Establishing a Data Architecture
If SMART Vaccines is to become a valuable component of decision making in the vaccine enterprise, concerted data collection efforts will need to be initiated and sustained. The most useful first step would be to establish a data warehouse containing the best sources and estimates for various populations. While demographic datasets are widely available and adaptable for SMART Vaccines, data on disease burden, economic factors, and vaccine characteristics are largely unavailable and need to be estimated with expert guidance. Once experts provide such information for settings around the world, then it will be beneficial to develop training and guidance tools regarding common definitions, calculation formulas, estimation principles, and standardized nomenclature because this will enhance comparability among users.
Conducting Usability Studies
The most useful evolution of SMART Vaccines will occur if there is input from early adopters involved in the on-site testing and assessment of Ver-
FIGURE 4-1
A rendering of SMART Vaccines imagined as a potential tablet application. Future versions of SMART Vaccines in an open-source setting have the potential to be instantiated in a range of different formats, including different languages, and using different visual design features.
sion 1.0 as a support tool in actual decision making. Such input will help point to additional modifications that can improve SMART Vaccines and will help inform the requirements for multiple or customizable visual interfaces for users from various sectors and countries. The usability testing may also shed light on the value of developing versions of SMART Vaccines in alternate software and visual design platforms and also in various of the official languages of the United Nations besides English—for example, Arabic, Chinese, French, Russian, or Spanish. An artist’s rendering of SMART Vaccines in the form of a potential tablet application is shown in Figure 4-1.
Developing Standard Profiles to Enhance
Cross-User Comparisons
Multi-attribute value models (including multi-attribute utility theory as used in SMART Vaccines) do not provide a uniform scale of comparison for different users. As has been explained before with the temperature anal-
ogy, each user’s value structure creates its own value metric. While some of the vaccine examples employed in this and in the Phase I report allow such a comparison, they are limited to the United States and South Africa. Expanding the availability of such “comparison cases” to many other settings will improve the ability of users to understand their own and others’ SMART Scores in various settings. Thus, the committee believes that there will be great value in the further development of a series of “standard” vaccine descriptors for which the disease burdens, prevention and treatment costs, mortality and morbidity, productivity losses, and vaccine effectiveness are well understood and there are no misinterpretations of results.
Refining the Software Platform and Model
No model is perfect, and the modeling work the committee has undertaken will require continuous refinements. Thus, future work will have the greatest impact if the model is enhanced and improved using software languages that are widely accessible and platform-neutral. The current model is programmed in MATLAB, which can be compiled for use on standard operating systems from Microsoft, Apple, and Linux.
SMART Vaccines will work best if it incorporates or allows the use of a variety of commonly used data-management platforms, either commercially developed platforms or other open source spreadsheet programs for data entry and export. The current version allows a Microsoft Excel–based spreadsheet structure for data entry and export, an approach widely used with both Microsoft products and the Open Source Initiative.
Users of SMART Vaccines will find numerous existing databases that provide information about populations, their disease burdens, and the desirable attributes of yet-to-exist vaccines which can form the basis for initial uses of SMART Vaccines. Future enhancements can facilitate the easy introduction and use of such extant data sources and accommodate them as they change structure and content.
Different diseases are best modeled with different population dynamics. Future enhancements can enlarge the number of ways in which population growth is treated. The most complex of these could include, for example, a population with evolving death rates and fertility rates and even a fertility rate that changes in response to alterations in the mortality rate caused by various diseases.
As decision sciences and modeling techniques continue to improve, future versions of the software may incorporate improvements in the multi-attribute utility theory modeling or the ranking approach used by SMART Vaccines 1.0, and they could offer options for alternative value modeling, such as mathematical programming or analytic hierarchy process.
Expanding Outreach and Training
The use of SMART Vaccines will increase as more people become aware of its capabilities, which should in turn increase the breadth, depth, and overall competence and expertise within the community of users. Thus, the committee has considered a variety of outreach platforms and tools, and it believes that outreach through presentations, publications, online guides, tutorials, and academic courses will help expand awareness of the software’s capabilities and increase its future potential.
Enhanced Applications of SMART Vaccines
Comparisons of New and Existing Vaccines, Public Health Prevention, and Treatment
This committee was charged with creating a modeling framework that, when fully instantiated, would allow users with different perspectives to create their own rankings of the potential value of new preventive vaccines. While SMART Vaccines was designed to accomplish this specific goal, the committee believes that it can be applied to a broader array of related applications with little or no change in the software structure by assembling the data necessary to describe the particular options being evaluated. Several of these potential applications are described briefly in the following paragraphs.
Choosing Among Existing Vaccines
SMART Vaccines has the potential to help health administrators in various settings compare existing vaccines in order to determine those that best fit their own demographic, economic, and contextual needs. This task is actually easier than ranking new vaccines, because the characteristics of existing vaccines—such as cost, distribution, storage and administration requirements, potential side effects, likely population coverage, and efficacy—are already available and do not have to be estimated.
Alternative Public Health Measures
Although the word “vaccines” appears explicitly in the name “SMART Vaccines,” the software need not be limited to comparing vaccines. Consider, for example, methods of reducing the disease burden of malaria. In principle, one could use SMART Vaccines to evaluate a variety of measures for achieving that goal and could thus compare the performance of vari-
ous vaccines with the medical treatment of malaria and with the prevention of malaria through the use of mosquito netting or mosquito abatement programs. While SMART Vaccines cannot incorporate the exact details of such programs, the committee believes that adept users will be able to guide decision making among these types of choices using the existing capabilities of SMART Vaccines. Thus, “clean water” or “mosquito netting” or “improved sewage disposal” can be entered as “new preventive vaccines” with considerable efficacy and so expand the potential uses of the software application.
Veterinary Vaccines
While this report focuses on human diseases and human vaccine prioritization and development, vaccination is also used to protect hundreds of millions of livestock, poultry, aquatic life, and companion animals worldwide. Interestingly, the second disease completely eradicated worldwide, after smallpox, was rinderpest in 2011. This paramyxovirus does not infect humans, but it was nonetheless responsible for countless human deaths that resulted from the losses of millions of head of livestock, which led to famine and disease.
Currently, veterinary vaccines account for approximately 20 percent of the total vaccine market. SMART Vaccines can be modified for use in the world of veterinary medicine. Intuitively, it seems likely that the greatest interest would be in analyzing the economic attributes of animal vaccines; however, the broad area of animal health would benefit from a tool that guides the making of policy and various other decisions that affect animals. Furthermore, healthier food animals lead to safer food, and a number of animal diseases pose risks of transmission to humans. Thus, an investment in animal vaccines will also have a real benefit to human health, which leverages and adds value to the investment in veterinary vaccines. Except for the new and different data demands, there is no difference between comparing human vaccines and comparing vaccines for domesticated animals. And improving vaccines for food animals has the additional advantage that it could help reduce the use of antibiotics, thereby reducing the risk of developing highly resistant organisms in both animal and human populations.
Disease-Resistant Plants
The hybridization and genetic modification of plants to enhance disease resistance is conceptually similar to vaccinating humans or animals against
disease. The multi-attribute model embedded in SMART Vaccines could be used to compare alternative approaches to creating disease resistance in plant; for example, it could be used to compare the potential risks of creating disease immunity in plants by hybridization versus through gene modification.
Facilitating Discussions Among Various User Groups
From discussions with various user groups throughout Phase I and Phase II, the committee has concluded that interest groups often fail to fully communicate with each other concerning values, choices, and impediments to reaching their goals. The resulting shortcomings in understanding can be seen, for example, in questions that arise during the design phase about the best compromises in vaccine attributes or product profiles. What is more valuable to stakeholders: oral administration or thermal stability of the vaccine? Do vaccines fit within the existing immunization schedules and programs? What social customs might enhance or limit population acceptance of certain vaccines? The answers to these questions will differ widely from setting to setting, but the committee believes that these types of trade-offs have not been fully discussed or understood by various user communities. SMART Vaccines allows the formal consideration of these trade-offs at the local level. Thus, the software can allow users to comprehend and communicate their own preferences more clearly.
Vaccine manufacturers face other constraints and may have different objectives. They seek profitable products, because sales revenues are typically the only source for financing research, development, testing, and production. Intellectual property laws provide temporary market protection for recouping research and development costs, but ultimately the risks surrounding scientific research and eventual commercial products drive many corporate decisions about new product development. Knowing which attributes are more or less valuable to potential user communities could enhance vaccine developers’ decision-making process and ensure that they better understand the values that users place on various potential vaccines as a way of improving their decision making.
In short, it is the committee’s hope that SMART Vaccines will serve as a valuable tool to enhance and clarify discussions among various user communities, which will, over time, lead to improved public health outcomes.
In an immediate follow-up Phase III activity, an Institute of Medicine and National Academy of Engineering committee is expected to evaluate the utility of SMART Vaccines and offer guidance on a data warehouse for the software. Specifically, the committee is expected to produce some use case scenarios in collaboration with actual end users of the software. In doing so, it will also attempt to develop a framework for a data warehouse including estimation strategies to create future datasets for the software.
