Study Designs for the Safety Evaluation of Different Childhood Immunization Schedules
To date, there have been few comparative studies evaluating the safety of different vaccine schedules. A few of the existing studies have shown that there are cases in which the risk of adverse events can depend on the vaccination schedule used. Hence, it is both a feasible and an important area of study. As a relatively new field of investigation, the big question is what types of study designs will be most fruitful for evaluating different childhood vaccine schedules. A number of possible study designs are presented in this review to evaluate different features or components of the vaccine schedule. These include the timing of individual vaccines, the timing between doses of the same vaccine, the interaction effect between vaccines and concurrent health conditions or pharmaceutical medications, the interaction effects of different vaccines given on the same day, the ordering of different vaccines, and the effect of cumulative summary metrics such as the total number of vaccines or the total amount of some vaccine ingredient. Study designs for the comparative evaluation of one or more complete schedules are also considered. Methods are presented both for adverse events with an early onset, which are the easiest to study, and for adverse events with a late onset, including serious chronic conditions. It is concluded that a wide variety of different vaccine schedule components can be studied.
1 Department of Population Medicine, Harvard Medical School and Harvard Pilgrim Health Care Institute.
Before approval by the Food and Drug Administration (FDA), vaccines are evaluated for efficacy and safety using large Phase III randomized controlled trials. For childhood vaccines, the number of children enrolled in these trials is typically in the thousands. That is sufficient to detect common but not rare adverse events. For the latter, there exist several postmarketing vaccine safety surveillance systems using observational data on children who receive the vaccines as part of their general care. In the United States, these include the Vaccine Adverse Event Reporting System (VAERS), the Vaccine Safety Datalink (VSD), and the Clinical Immunization Safety Assessment Network, all sponsored by the Centers for Disease Control and Prevention (CDC), as well as the Post-Licensure Rapid Immunization Safety Monitoring System (PRISM), which is part of the FDA-sponsored Mini-Sentinel Initiative. Internationally, there are other important vaccine safety surveillance systems such as the Epidemiology Vaccine Research Program at the National Institute for Health Data and Disease Control in Denmark; the Vaccine Adverse Event Surveillance and Communication (VAESCO) Network, coordinated by the European Center for Disease Control; the World Health Organization (WHO) Collaborating Centre for International Drug Monitoring; and the Immunization Division at the Communicable Disease Surveillance Centre in England. All these vaccine safety systems have proven to be very useful and important. They have detected unsuspected adverse events leading to revisions in vaccine recommendations and, in other cases, established the safety of vaccines for which important safety concerns existed. Throughout their existence, there has been continuous and rapid development with respect to the types of questions studied and the epidemiological and statistical methods used. For example, for every new childhood vaccine approved by the FDA, VSD now conducts near real-time safety surveillance using weekly data feeds from electronic health records (Lieu et al., 2007; Yih et al., 2011). The credit for these continuously improved vaccine safety surveillance systems goes to the devoted scientists that are building the systems and using them for many important studies, to the government agencies supporting this work, and to the vaccine safety advocacy groups that are the key public voice for improved and expanded vaccine safety surveillance.
Most postmarketing studies evaluate the general question as to whether or not a vaccine causes an adverse event. Very few postmarketing studies have evaluated whether the risk of adverse events depends on the scheduling of the vaccines. For example, few postmarketing studies have evaluated whether the risk of adverse events depends on the age at which a vaccine is given, on the relative timing of two different vaccines, or on a combined cumulative effect generated by the timing of dozens of different vaccines.
These are all different components of the vaccine schedule, and any one of these could potentially be related to the number and severity of adverse events. When evaluating the safety of different vaccine schedules, it is hence important to study the whole range of issues, from the timing of a single vaccine to summary metrics based on the timing of dozens of vaccines.
The paper presented in this appendix was commissioned by the Institute of Medicine Committee on the Assessment of Studies of Health Outcomes Related to the Recommended Childhood Immunization Schedule. The paper considers different types of potential questions and concerns about the safety of vaccine schedules and describes different epidemiological study designs and statistical methods that can be used to answer such questions in a scientifically rigorous manner. The core of this paper is a set of proposals for the type of study designs and methods that would be appropriate for the comparative evaluation of vaccine adverse events under different vaccine schedules, and the paper is written in the context of the many difficulties raised by the speakers at the committee meetings held in February and March 2012. Note, though, that it is not a synthesis, an evaluation, or a review of the many excellent presentations made at those meetings. Instead, it should be viewed as complementary information. Note also that the paper does not say anything about the advantages or disadvantages about specific vaccines or vaccine schedules. Rather, the focus is on potential study designs and methods and their ability, or inability, to answer such questions.
DEFINITIONS OF KEY TERMS
Component of the vaccine schedule: some specific feature of the vaccine schedule, such as the age at which one of the vaccines is given or the total amount of immune-stimulating content received from all vaccines in the schedule. Not to be confused with different components of a single vaccine.
Early onset: an adverse event that manifests itself and can be detected within a few weeks after vaccination.
Late onset: an adverse event that does not manifest itself and/or cannot be detected until a few months or years after vaccination.
Potential adverse event: a health event under evaluation in a vaccine safety study, in order to determine if it is caused by the vaccine(s) or not.
VACCINE SCHEDULES, ADVERSE EVENTS, AND DATA SETS
Vaccine Schedules and Their Components
To study the safety of different childhood vaccine schedules is an important but complex task. With dozens of vaccines, many of which have
multiple doses, there are an almost infinite number of possible vaccine schedules that can be used. To scientifically evaluate the safety of different vaccine schedules, it is necessary to look at specific components of the schedule. Some such components are as follows:
Timing of Specific Vaccines
- The age at which a specific vaccine is given, such as the age at the first dose of the hepatitis B vaccine.
- The relative timing of different doses of the same vaccine, such as the number of months between the first and second doses of the 7-valent pneumococcal conjugate vaccine (PCV7).
- The interaction between the timing of a specific vaccine and time-varying health events or health status, such as a vaccination given to a child taking a temporary or seasonal medication.
Relative Timing of Two or More Different Vaccines
- The interaction among different vaccines given on the same day, such as the effect of giving the measles, mumps, and rubella (MMR) vaccine and varicella vaccines at the same health care visit or different health care visits.
- The order in which different vaccines are given, such as whether measles vaccine is given a few months before or after the diphtheria-tetanus-pertussis (DTP) vaccine.
Summary Metrics of a Vaccine Schedule
- The total number of vaccinations given to the child before a certain age, such as the 6th birthday.
- The average age at which the vaccines were given.
- The cumulative amount of immune-stimulating content present in all vaccines received.
In addition to specific components of the vaccine schedule, one can also try to compare complete vaccine schedules.
Comparison of Complete Vaccine Schedules
- Whether or not the child has approximately followed the CDC-recommended vaccine schedule.
- The comparative safety of a specific alternative vaccine schedule, such as Dr. Bob’s (Sears, 2007), versus the one recommended by CDC.
The study design and statistical methods used depend on which vaccine schedule component is being evaluated. As they are quite different, each component of Components (a) to (e) is dealt with in separate sections of this paper. For cumulative summary metrics, the methods are similar irrespective of what component of the vaccine schedule the metric is designed to measure. Components (f) to (h) are treated together. Methods for comparing different complete vaccine schedules are discussed, and one vaccine schedule with completely unvaccinated children is evaluated. More general methodological issues and financial and ethical considerations are also discussed.
The different types of studies should not be done in isolation from each other. If it is found that one complete vaccine schedule has an excess number of adverse events compared to another, we do not know which component of the schedule caused the difference. Hence, it is not recommended that studies comparing complete schedules be conducted without also evaluating specific components of those schedules. Likewise, when a specific component is studied, results may be confounded by other components of the vaccine schedule. For example, a child receiving vaccine A at an early age may be more likely to also receive vaccine B at an early age, and the timing of vaccine A will then be correlated with the number of adverse events even if it is the timing of vaccine B that is the culprit. It could also be that there are two different vaccine schedule components that cause adverse events but that they cancel each other out when one looks at the difference between two complete schedules, making it impossible to detect the problem if only the complete schedules are studied.
Another reason for studying specific components of the vaccine schedule is that, if a problem is found, we need to know how to revise the schedule in order to reduce the number of adverse events. Just because one complete vaccine schedule is found to cause more adverse events than another, we do not necessarily have to revise all components of that schedule.
Adverse Events with Early Versus Late Onset
In vaccine safety studies, the goal is to evaluate if there is a causal relationship between the vaccine(s) and some health event of interest. The latter is denoted as a potential adverse event, as it may or may not be an actual adverse event caused by the vaccine(s). The type of health event under study determines the appropriate methodological methods for vaccine safety studies. This paper considers two main types. The first type consists of potential adverse events with an early onset that can be detected soon after the onset. The event itself could be either acute and of a passing nature without any permanent damage, such as a febrile seizure, or chronic, lasting many years, such as a stroke. The second type consists of potential adverse events with a late onset several months or years after vaccination
and events with an early or a gradual onset that cannot be detected until long after vaccination. For simplicity, all of these are denoted ”late onset.” These potential adverse events can also be either acute or chronic in nature.
The most suitable study designs and analysis methods are greatly dependent on whether the potential adverse event has an early or late onset, and in the description below, separate methods are proposed for the two outcome types. This is a little bit of a simplification, since there are, of course, also potential adverse events that fall somewhere in between on this spectrum. It should also be pointed out that an early-onset chronic condition can be studied by use of either of the methods described for early or late onset, but the early-onset methods are in most cases preferable.
Another key issue is whether there is a clear time at which the potential adverse event happened, as with, for example, a seizure, or whether the disease evolves more gradually, without a single clearly defined day of onset, as with, for example, narcolepsy or autism. This does not affect the study design as much as the time of onset, but it is an important consideration when defining and collecting the data.
For most potential adverse events, we are interested only in incident diagnoses, that is, the first time that a particular diagnosis has been made. For example, if a child is diagnosed with asthma at age 2 years and then has a follow-up visit for his/her asthma at age 4 years, we do not want to attribute the asthma to a vaccination given at age 3 years. Depending on the potential adverse event under study, one can define an incident diagnosis as a diagnosis that has not occurred during the previous D days. The value of D will depend on the adverse event, but a typical value is about 1 year.
The potential adverse event studied either can be very specific, such as febrile seizures or autism, or can be more general, such as all-cause outpatient physician visits, emergency department visits, or hospitalizations. The latter set of events may seem more desirable, as it includes the combined effect of the vaccine schedule on all important health events, but the opposite is true. Such general definitions are more prone to biases, and they are therefore more difficult to study. This is because people that follow the CDC-recommended vaccine schedule may be different from those that do not, in terms of their health care–seeking behavior. For example, parents that are more prone to take their children to the doctor when the child is sick may also be more prone to take their children to the well care visits during which most vaccines are given.
Data Sets for Postmarketing Vaccine Safety Studies
To facilitate the understanding of the study designs and methods described in subsequent sections, a brief background is first given concerning
some of the data sets that are available and currently used for postmarketing vaccine safety studies.
Premarketing Randomized Trials
Phase III randomized trials are primarily designed to evaluate the efficacy of vaccines. They are also able to find common adverse events, but their sample size is typically not large enough to evaluate rare but serious adverse events. Their primary use for postmarketing vaccine safety surveillance is to generate study hypotheses. For example, a single case of Kawasaki disease in the vaccine arm of a Phase III randomized trial is not evidence that the vaccine causes Kawasaki disease, since it could be pure coincidence, but it may warrant a postmarketing safety evaluation.
Spontaneous Reporting Systems
Most countries in the world have a vaccine safety surveillance system based on spontaneous reports. These are linked together through the WHO Collaborating Centre for International Drug Monitoring in Uppsala, Sweden, so that it is possible to combine data from multiple countries. In the United States, CDC and FDA are joint sponsors of VAERS.
These systems contain spontaneous reports of suspected vaccine adverse events sent in by physicians, nurses, patients, parents, manufacturers, and others. The gender and age of the vaccinated person are some of the variables collected. There is often information about multiple vaccines given on the same day. Analyses are done by the use of proportional reporting ratios (Evans et al., 2001) and similar methods. For example, if 1.5 percent of all vaccine-related adverse event reports are for seizures and there are 1,000 reports for vaccine A, then we would expect 15 seizure reports for vaccine A. If, in reality, there are 45 such reports, the proportional reporting ratio is 3. That is more than what one would expect, and it may indicate that there is an excess risk of seizures after vaccination. Actual analyses are more complex, since it is necessary to adjust for age and other variables. There are also other more sophisticated methods used (Bate et al., 1998; DuMouchel, 1999; Rothman, 2011).
The major advantage of VAERS is that it receives reports from the whole country. The two major disadvantages are that there is underreporting and that there are no reliable denominator data. That is, while we have information about a number of vaccinated children with the potential adverse event of interest, we do not know the total number of children that were vaccinated, how many unvaccinated children had the same type of event, or how many vaccinated children had the event without it being reported.
Some reports to VAERS are studied further in the Clinical Immunization Safety Assessment Network. Among other things, this network aims to “improve the scientific understanding of vaccine safety at the individual patient level” by obtaining and evaluating detailed genetic and other information from each patient (LaRussa et al., 2011).
Electronic Medical Records
For 2011, it is estimated that 57 percent of office-based physicians used electronic medical records (EMRs), up from 24 percent in 2005 (Hsiao et al., 2011). The EMRs most useful for medical research are the ones from large health plans, as they contain medical records for a well-defined member population, including both inpatient and outpatient encounters. The VSD project is the premier EMR-based vaccine safety system in the United States (Baggs et al., 2011; Chen et al., 1997; DeStefano and the Vaccine Safety Datalink Research Group, 2001). Led by CDC, it is a collaboration with 10 health plans: Group Health in the State of Washington; Harvard Pilgrim/Atrius Health in Massachusetts; HealthPartners in Minnesota; Kaiser Permanente in Colorado, Georgia, Hawaii, Northern California, Oregon, and Southern California; and Marshfield Clinic in Wisconsin. Together, these health plans have about 9.5 million members and an annual birth cohort of more than 100,000. The VSD system is used both for retrospective studies and for near-real-time vaccine safety surveillance with weekly analyses of newly approved vaccines. Similar systems exist in a few other countries, including the Epidemiology Vaccine Research Program at the National Institute for Health Data and Disease Control (Statens Serum Institut) in Denmark.
The major advantage with EMR systems is that denominator data are available, as all vaccinated children can be identified. It is then possible to compare the number of adverse events in vaccinated and unvaccinated children or vaccine-exposed and unexposed time periods within the same child. A disadvantage is that the data are registered for purposes other than research, and there is sometimes miscoding of health events. Depending on the health outcome, manual chart review is therefore sometimes warranted.
Health Insurance Claims Data
Health insurance companies have medical information for millions of insured members and their families, which they receive when doctors and hospitals file their financial reimbursement claims. One such system in the United States is the PRISM program, run by FDA as part of its Mini-Sentinel project (Nguyen et al., 2012). Claims data are more limited than EMRs but can be used in much the same way for postmarketing vaccine
safety studies. The major advantage is the large sample size that can be achieved. The major disadvantage is that some health conditions are not captured. Depending on the potential adverse event under study and the confounders that need to be adjusted for, this may or may not be a problem.
Because of their similarities, EMRs and health insurance claims data will be treated as the same type of data in this appendix under the name “health plan data.”
Study-Specific Data Collection
Sometimes, new data are collected specifically for vaccine safety studies, such as a self-controlled case series, a case-control study, a cohort study, or a postmarketing randomized trial. An intermediate option is to obtain some of the data from health plans, disease registries, and/or vaccine registries, while the remaining data are collected from study-specific patient surveys or measurements. The available options are too many to provide a detailed description of each.
TIMING OF SPECIFIC VACCINES
In a randomized childhood vaccine trial, the age at which the vaccine is given is tightly controlled by the study design, to correspond to the future planned vaccine schedule. This is appropriate, but once a vaccine is on the market, it is also given at a wide variety of other ages, for a variety of reasons. There are two scenarios in which it is of great interest to evaluate the risk of a vaccine as a function of the age at which the vaccine was given. (i) If a vaccine safety study has shown that there is a statistically significant excess risk of an adverse event, we want to know if the excess risk varies by the age at which the vaccine was given. (ii) Even if a general safety study covering all age groups has not shown a statistically significant excess risk of the adverse event, there could still be an excess risk if the vaccine is given at certain ages outside the recommended schedule. Such a safety problem could be masked by the noneffect among the most populous age group, and a special study looking at age-specific risks would be warranted.
Known Adverse Events with Early Onset
Some vaccines have been shown to cause an acute adverse event within a few weeks after vaccination. Examples include intussusception 3 to 7 days after vaccination with rotavirus vaccine (RotaShield) (Kramarz et al., 2001; Murphy et al., 2001) and febrile seizure 7 to 10 days after vaccination
with MMR and the measles, mumps, rubella, and varicella (MMRV) vaccine (Klein et al., 2010). There are also several such examples of less severe adverse events like fever and rash. The adverse event may be serious enough to warrant the withdrawal of the vaccine from the market, as with the rotavirus vaccine, or it may be mild enough to keep using the vaccine, as with MMR. A midlevel alternative option is to revise the vaccination schedule to minimize the number of adverse events or to contraindicate the vaccine in a certain age group. Knowledge of the relative and attributable risk of the adverse event as a function of age is one important component when deciding between these options, together with other important factors, such as how the immunogenicity varies by age. This paper discusses only methods for obtaining knowledge about the former and not how to weight different sources of information to arrive at a final decision.
In two different studies, Gargiullo et al. (2006) and Rothman et al. (2006) evaluated the effect of age on the excess risk of intussusceptions after vaccination with the rotavirus vaccine (RotaShield). In a more recent study, Rowhani-Rahbar et al. (2012) evaluated the effect of age on the risk of febrile seizures after vaccinations with MMR and MMRV. All three studies found that the risk of the adverse event varied greatly by age.
EMRs from health plans and health insurance claims from health plans are ideally suited for studying this question. It is also possible to use data from a case-control study. VAERS data cannot easily be used since VAERS does not contain information about the age distribution of vaccinated children. Too few data are available from premarketing randomized because such trials are too small and typically do not include individuals over a wide enough range of ages. In light of existing observational data, specifically designed postmarketing randomized trials could be unethical, depending on the nature of the known adverse event.
The first key step is to determine the time between the vaccination and the adverse event as precisely as possible. Some children will, just by chance, have the adverse event soon after vaccination. To maximize the precision of our age estimates, we want to exclude as many of them as possible, by counting only the adverse events occurring in the true risk window. An efficient way to determine the appropriate risk window is to
use a temporal scan statistic. For a cohort of vaccinees with a subsequent event of interest, record the number of days from vaccination to the event. Ignore events that occur on the same day as the vaccination, as they may have a different background rate, as well as those that occur beyond an upper limit, such as 70 days after vaccination. If there is no relationship between the vaccine and the adverse event, we expect the adverse events to be uniformly distributed during the [1, 70]-day period. The temporal scan statistic scans the time period for any cluster of events, without any assumptions about their location or length. The method determines the statistical significance of such clusters, adjusting for the multiple testing inherent in the hundreds of overlapping time periods evaluated. As an example, temporal scan statistics were used to determine that the excess risk of seizures after vaccination with MMRV is confined to the 7- to 10-day postvaccination period (Klein et al., 2010).
The second step is to evaluate the relationship between age at vaccination and excess risk of the adverse event. The simplest and most common way to do this is to divide age into different groups, such as 6 to 12 months and 12 to 24 months, and compare the risk. It is unrealistic to assume that the risk suddenly jumps at a particular age, and for greater precision, it is better to model risk as a continuous function of age. This can be done by the use of either regression with first-, second-, and higher-degree polynomials or regression splines (Rothman et al., 2006).
In these analyses, it is important to take the underlying natural age-related risk into account. For example, the incidence of intussusceptions is very low immediately after birth, after which it gradually increases until about 5 months of age, after which it gradually decreases (Eng et al., 2012). There are a number of possible ways to adjust for this, depending on the exact study design. In a cohort study of vaccinated individuals, one can use historical data to estimate the age curve, using a polynomial function, and then use that as an offset term in the regression model. An alternative approach is to use both a risk and control interval for each individual, in a self-controlled analysis, evaluating whether the relative risk in these intervals varies by age of vaccination. Note, though, that if the natural incidence rate for the adverse event varies greatly by age in weeks rather than years, it is still necessary to incorporate an offset term based on the natural age curve even when a self-controlled analysis is conducted. In a case-control study, matching by age ensures that the age-based incidence curve is adjusted for.
Vaccine Risk for Specific Age Groups: Early-Onset Adverse Events
Most childhood vaccines are given according to the recommended schedule, but some children may get the vaccine at a much earlier or a much later age. There are many potential reasons for this, including a high risk of exposure due to a current disease outbreak or because family members have the disease, or due to missed well care visits, shortages of the vaccine, parental or physician concerns about the recommended vaccine schedule, misunderstanding of the recommended schedule, or medical errors, etc. As an example, while the first dose of MMR is recommended at age 12 to 15 months, in one health plan, 22 percent of children were recorded to have received it later and 0.7 percent to have received it before their first birthday, with 0.3 percent receiving it before 6 months of age. Nationwide, even half a percent adds up to a fairly large number, and it is important to evaluate the safety of the vaccine for those children, so that a contraindication warning can be issued if there is a major safety problem.
After the 2004 recommendation to give influenza vaccines to 6- to 23-month-old children, Hambidge et al. (2006) used data from VSD to conduct an influenza vaccine safety study specific to this age group, looking at a wide variety of potential adverse events.
Health plan data capture all vaccinations at whatever age they occurred, so such data are useful not only for evaluating the safety of vaccines in special age groups but also for characterizing the real-world age distribution of vaccinated children.
With its national coverage, VAERS data can also be used to monitor vaccine safety in specific age groups. While no denominator data are directly available, a large number of adverse event reports in children outside the recommended age range could be the first indication that an age-specific problem exists.
If a change in the recommended age that a vaccine should be given is anticipated, a randomized trial may be warranted. For that to occur, there needs to be some uncertainty as to whether the currently recommended time is safe and some evidence, based on, for example, observational data, that an alternative age is safer. If the question is simply whether the vaccine should be contraindicated for certain age groups or whether the vaccine
is also safe outside the recommended age of vaccination, without any evidence of harm at the recommended age, then a randomized trial would not be ethical.
For health plan data, there are a few different analysis options. For early-onset events, a self-controlled risk interval design can be used. First, decide on a risk window, such as 1 to 21 days after vaccination, and a control window, such as 22 to 42 days after vaccination. For each vaccinated child in the age group of interest, count how many of them had an adverse event in the risk and control windows, respectively. Suppose that the two windows are of the same length and that there are a total of n adverse events in the two windows combined. The number of adverse events in the risk window then has a binomial distribution with parameters n and p =½.
Since this is a self-controlled analysis, it is only time-varying confounders that may need to be adjusted for, and there is no need to worry about gender, genetics, stable environmental factors, study site, etc. For some adverse events where the incidence rate changes rapidly from one week of age to the next, an age adjustment must be made. If the age distribution of the disease is know, this can easily be done by use of an offset term in a logistic regression model. The same is true if there are strong seasonal trends in the incidence rate. An alternative way to adjust for seasonality is to use a case-centered approach, as proposed by Fireman et al. (2009).
The choice of risk and control windows depends on the vaccine and the adverse event. Sometimes, it is worthwhile to have a washout period between the two windows. To avoid day-of-week effects, the two windows should have the same number of days in any modulus of seven. For example, the risk window may be 1 to 14 days and the control window 22 to 70 days or the risk window may be 1 to 2 days and the control window days 8 to 9 together with days 15 to 16. Theoretically, it is also possible to use a comparison window before vaccination, but that can introduce confounding by indication or contraindication.
In VAERS data, the age of the vaccinated child is one of the variables collected. To evaluate whether a vaccine is safe outside the recommended schedule, it is hence possible to look at specific predefined age groups. This can be done by the same methods that are used for all age groups combined, such as proportional reporting ratios (Evans et al., 2001).
Almost all childhood vaccines are given in multiple doses a few months or years apart. It is conceivable that the length of the time interval between vaccine doses could increase or decrease the risk of adverse events.
Using a randomized trial, Pittman (2002) showed that the risk of adverse events was reduced if the second dose of subcutaneous anthrax vaccine, adsorbed, is given 4 months rather than 2 weeks after the first dose.
Early-Onset Adverse Events
The use of electronic health data is suitable for vaccine doses that are at most a few years apart. If the time between doses is too long, health plan data are less suitable, as only some members will have been enrolled long enough to have information about all the doses of interest.
For simplicity’s sake, first consider the situation where we want to evaluate the length of the time interval between the first two doses of the vaccine with respect to early-onset adverse events after the second dose. First, identify a cohort of children who received the first two doses of the vaccine. Exclude children that do not have a sufficiently long enrollment in the health plan to ensure that these are truly the first two doses. Note the number of days between the doses and whether they had an adverse event during a prespecified risk window after the second dose. For the statistical analysis, use logistic regression. The dependent variable is whether the potential adverse event was present in the risk window or not. The independent variable of interest is the number of days between the two doses. Adjust for gender, age at the second dose, calendar year, seasonality, study site, and any other potential confounders by including these as additional independent variables.
When one is looking at early-onset adverse events after the third dose, the same methods can be used for evaluating the length between the first and third dose or between the second and third dose. A single logistic regression can be used to evaluate both the time from the first to the third
dose and the time from the second to the third dose, by including both of them as two separate independent variables. The same applies for early-onset adverse events after subsequent doses.
If an excess risk is found, it is not clear from this design whether it is an excess risk due to the time length between the vaccinations or if there is an excess risk driven purely by the first dose and where the timing of the adverse event is such that it happens soon after the second dose in children that receive the second dose sooner. By estimating the temporal function of any excess risk due to dose number 1 alone, this can be adjusted for by including it either as an offset term or as an additional variable in the regression model, which then also may include children who never received dose two. An alternative, simpler approach is to limit the study to children where the doses are given at least x days apart, where x is chosen to be large enough that it is unlikely that there is any excess risk beyond that time that is purely due to the first dose.
If there is some evidence from the observational study that there is a differential risk depending on the time between vaccinations but it is not conclusive, then a randomized trial could be conducted. For example, in a study of a rotavirus vaccine, children may be randomized to receive the three doses at age 2, 4, and 6 months of age, according to the CDC-recommended vaccine schedule, plus a placebo dose at age 9 months, versus three doses at age 2, 6, and 9 months, plus a placebo dose at age 4 months. The results from the observational study, with its wide variety of schedules, can be used to inform the definition of the study arms in the randomized trial. Note, though, that if the adverse event is rare, a randomized trial is not a feasible approach, as the required sample size would be very large, and hence, the cost of the trial would be prohibitively expensive.
Late-Onset Adverse Events
The use of electronic health data is suitable for late-onset adverse events that occur within a few years after the last dose and for vaccines for which all doses of interest are given within a year or two. For late-onset events and longer times between doses, health plan data may be less suitable, as only some members will have been enrolled long enough to be informative.
The same methods used for early-onset events can be used for late-onset adverse events, with some modifications. Most importantly, rather than defining adverse events in terms of a predefined risk window after the last
dose, it is more suitable to define them according to a predefined age range. For example, the study may include only children who had all the doses of interest before 18 months of age and would consider only adverse events for which the incident diagnosis occurred between ages 2 and 6 years. That will prevent bias due to age-varying incidence rates.
If suitable health plan data are not available, a case-control approach can be used instead. The first step is then to select children with an incident diagnosis during a predefined age range, together with a set of controls matched by age, gender, calendar month, study site, and other covariates of interest. The next, more challenging step is to obtain the vaccination history of each of the children. This could be done by contacting all the health plans or all the pediatricians that the child has had. The dose interval length is then compared between those children with and without the adverse event.
INTERACTION EFFECTS BETWEEN VACCINES AND HEALTH CONDITIONS
Several vaccines are contraindicated for children with specific health problems. For example, live attenuated influenza vaccine should not be given to 2- to 4-year-old children who have had wheezing during the past 12 months (CDC, 2012). This means that the vaccine schedule may have to be modified for some children on the basis of their personal disease history. To know if and when that is necessary, one needs to study the interaction effects between vaccines and preexisting health conditions.
The study of interaction effects between vaccines and health conditions is especially important when there is a known excess risk of an adverse event. If it is possible to pinpoint that the adverse events are due to an interaction effect, then the number of adverse events can be reduced by contraindicating the vaccine for children with the health condition in question. For example, if the risk of seizures after vaccination with MMR is higher among children with a recent well-defined disease episode, then MMR may potentially be postponed by 3 months for those children.
Early-Onset Adverse Events
Electronic health plan data are suitable for early-onset adverse events.
First consider the scenario in which there is a known increased risk of the adverse event in the population as a whole and we want to know if the excess risk is more severe among children in a specific group. By use of the temporal scan statistic, first determine the true risk window for the adverse event, as described above. Suppose that we have an excess risk in the 7 to 10 days after vaccination. We now define the study population as those who received the vaccine and who had an adverse event in some longer time period, such as 1 to 42 days after vaccination. In a logistic regression model, the dependent variable is whether they had the adverse event inside or outside the true risk window. The independent variables are the various health status variables that we want to examine as potential risk modifiers. Several of these can be included in the same logistic regression, but doing several univariate analyses may be a suitable first step. As long as the baseline risk for the adverse event is fairly constant over the longer time period, it is not necessary to adjust for age. Note that since all subjects had the vaccine and all subjects had the adverse event, there is no actual interaction term in the logistic regression model.
If we do not have a known adverse event but still want to evaluate possible vaccine–health status interaction terms, we can still use the same approach with a reasonable guess of a wider risk interval. For example, the risk interval may be 1 to 42 days, while the comparison interval is 43 to 84 days.
Late-Onset Adverse Events
The approach described above cannot be used to study late-onset adverse events. Instead, we first determine if a vaccinated child had the health condition of interest at the time of vaccination. We then compare the number of late-onset events between the children that did and those that did not. This design is more prone to bias than the self-controlled design described above. One way to partially adjust for this is to include only children that had both the vaccine and the potential adverse event of interest, at any time, and compare the children who had the vaccination at the same time as the health event with those that had them at different times.
VACCINE-VACCINE INTERACTION Background
In the CDC-recommended vaccine schedule, many different vaccines are given on the same day. It is plausible that two vaccines, if given separately
from each other, do not increase the risk of adverse events, but if they are given on the same day, there is a vaccine-vaccine interaction effect, leading to increased risk. It could also be that one or both of the vaccines, when given separately, lead to a modest excess risk of the adverse event but, when given together, lead to a much higher excess risk.
With data from VSD and separate self-controlled risk interval analyses, it was found that there was an increased risk of seizures 0 to 1 days after trivalent inactivated influenza vaccine (TIV) and also that there was an increased risk of seizures 0 to 1 days after the 13-valent pneumococcal conjugate vaccine (PCV13). To tease apart the effects from the two different vaccines and to evaluate the interaction between the two, the author of this paper suggested the approach mentioned below and worked out the formulas for the analysis. It was found that both vaccines had caused an excess risk of seizures 0 to 1 days after vaccination, irrespective of the presence of the other vaccines, and that the effects were independent of each other (Tse et al., 2012). This means that there was a positive additive interaction but no multiplicative interaction. Hence, the estimates obtained indicated that it is safer to give the two vaccines on separate days rather than on the same day.
Early-Onset Adverse Events
Both VAERS and electronic health plan data can be used to evaluate early-onset adverse events due to vaccine-vaccine interaction.
For spontaneous reporting systems, Almenoff et al. (2003) have developed a proportionality-based version of DuMouchel’s (1999) empirical Bayes multi-item gamma Poisson shrinker. Pairs of two drugs are treated as a separate unique drug, different from individual drug users. To signal a possible interaction-induced adverse event, the lower 5th percentile of the empirical base geometric mean estimate must be larger than the upper 95th percentile of the empirical base geometric mean estimate for both of the individual drugs. This approach makes sense in a data mining context, where it is necessary to have some form of formal or informal adjustment for the multiple testing.
Two other methods for evaluating interaction effects in spontaneous reports
have been proposed by Thakrar et al. (2007) and Norén et al. (2008). Both of these, as well as the previously described method by Almenoff et al. (2003), were proposed for drug-drug interactions, but they can also be used for vaccines.
For electronic health plan data, a different methodological approach is needed. With a self-controlled risk interval analysis, it is possible to evaluate the effect of a vaccine on an adverse event by comparing the number of adverse events in a risk interval right after the vaccine is given with the number in a control interval long after vaccination. Now, suppose we have two vaccines, such as PCV13 and TIV, and we want know to if there is an increased risk of seizure 1 to 2 days after vaccination. We can then do a self-controlled risk interval analysis for TIV and another one for PCV13, ignoring any other vaccines given on the same day. Suppose that we see an excess risk in both analyses. Since these particular vaccines are often given on the same day, it then is not clear if
- TIV causes an excess risk and PCV13 is just an innocent bystander with no excess risk.
- PCV13 causes an excess risk and TIV is just an innocent bystander with no excess risk.
- Both vaccines cause an increased risk of seizures independently of each other.
- There is either a positive or a negative interaction effect between the two vaccines.
To better understand the importance of different interaction effects, a few examples are given. Assume that TIV alone causes a twofold excess risk and that PCV7 alone also causes a twofold excess risk:
- If, when taken together, they also cause a twofold excess risk, then there is a negative interaction and it is safer to take the two vaccines on the same day.
- If, when taken together, there is a threefold excess risk, then there is a negative multiplicative interaction, while there is no additive interaction. In this scenario, it is equally safe to take the two vaccines on the same day or on separate days. This is because a threefold excess risk has twice as many excess cases than a twofold excess risk but there is only one rather than two times of exposure, which evens out the excess risk.
- If, when taken together, there is a fourfold excess risk, then the two vaccines act independently on a multiplicative scale (i.e., TIV doubles the risk and then PCV7 doubles the risk on top of that), while there is a positive interaction on an additive scale. In this scenario, it is safer to give the two vaccines on separate days, since one time period with a fourfold excess risk is worse than two time periods with twofold excess risk.
- If, when taken together, there is a fivefold excess risk, then there is a positive interaction on both the multiplicative and the additive scales, and again, it is safer to take the vaccines on separate days.
While it is possible to do three separate self-controlled risk interval analyses for the vaccines when given alone and when given together, the best approach is to combine all the information into one logistic regression model that includes both the main effects and the interaction terms. This can be done, and it is then possible to formally test for a multiplicative interaction effect.
While not a common concern, it has occasionally been suggested that the order in which vaccines are given may influence the risk of adverse events. Here, we are not thinking of vaccines given a couple of minutes apart at the same health care visit but of vaccines given a few days, weeks, or months apart. For example, in a study of DTP and the measles vaccine in a low-income African country, Aaby et al. (2004) hypothesized that “DTP as the last vaccine received may be associated with slightly increased mortality.” Veirum et al. (2005) suggested that “it might be examined whether provision of BCG [bacille Calmette-Guérin] or measles vaccine shortly after the last dose of DTP could secure specific protection and prevent the negative immune stimulation associated with having received DTP,” and that “different sequences of vaccinations” might have to be considered.
In a three-arm randomized vaccine trial with a total of 1,027 children, Leonardi et al. (2011) compared both immunogenicity and safety for different orders of vaccination with MMRV and PCV7. In the study, MMRV was given 6 weeks prior to, on the same day, or 6 weeks after the fourth dose of PCV7. The incidence of local and systemic adverse events was comparable among the groups, while no serious adverse events were reported
Electronic health plan data are ideally suited to study this question. Alternatively, children with the outcome of interest could be identified by the use of, for example, hospital data or data from a disease registry, followed by a vaccination history survey to their parents.
Late-Onset Adverse Events
The following is a study design for comparing the order of vaccines A and B with health plan data. For simplicity, this description assumes a single dose of each vaccine, but it can be generalized to multiple doses. First, identify children with the purported adverse event outcome of interest. Include only those children that had both vaccines A and B at least x days prior to the onset of the disease, except those that had both vaccines on the same day. With health plan data, the comparison group will be all other children. For each child with the purported adverse event, (i) note the exact age at disease onset, (ii) calculate how many of the comparison children of the same gender that also had both vaccines A and B at least x days before that age but not on the same day, and (iii) note the proportion of those that had vaccine A before vaccine B. Under the null hypothesis of no effect of vaccine order, this proportion is the estimated probability that the study child had vaccine A before vaccine B. The analysis is adjusted for age and gender, and other covariates can be adjusted for in the same way as gender.
The reason for excluding children that had one of the vaccines less than x days before the adverse event is to remove the effect of vaccine-specific early-onset adverse events that is caused by one of the vaccines independently of the presence of the other. The value of x will depend on the vaccines and adverse event studied. To avoid bias, it should be large enough so that any adverse events caused by one vaccine independently of the other do not vary in time on the basis of the number of days after vaccination.
An alternative is to use a case-control design. First, the children with the adverse event are selected as before. Second, identify a comparison group of children who did not have the adverse event outcome, matched by age, gender, and any other variables, and with the same inclusion criteria with respect to vaccination history. Then compare how many of the cases and how many of the controls had vaccine A before vaccine B, and vice versa.
The above-described design cannot be used for acute adverse events because of the bias mentioned above. Instead, the following design can be used. By the use of health plan data, select children that had vaccine A at any time. Separate them by whether they have had vaccine B prior to vaccine A or not. Then compare these two groups in term of how many of them had the adverse event of interest on the 1 to D days after they received vaccine A. The value of D defines the risk window and will depend on the vaccines and adverse event studied. The analysis can be performed using unconditional logistic regression, adjusting for covariates such as age at vaccination, gender, calendar years, and study site.
This study design cannot in itself distinguish between an effect due to the order of the vaccines and an interaction effect, where the risk increases with the same amount after the second vaccine irrespective of their order. By collection of the above-described data both for vaccine A and then, in the corresponding manner, for vaccine B, it is possible to compare the two risk estimates. If the increased risk is due to vaccine-vaccine interaction but not the order of the vaccination, these estimates should be the same.
CUMULATIVE SUMMARY METRICS OF THE VACCINE SCHEDULES
It is conceivable that it is neither the timing of individual vaccines nor the interaction between vaccines that is responsible for adverse events but, rather, some more general component of the vaccine schedule, such as the total number of vaccines given or the cumulative amount of immune-stimulating content, immunogenic adjuvants, or preservatives in all vaccines received. Similar study designs and statistical methods can be used for most of these types of summary measures or metrics of the vaccine schedule, so they are considered together.
The total amount of immunogens that a child has been exposed to is being used in studies of autism (DeStefano et al., 2012) and neuropsychological outcomes (Iqbal and DeStefano, 2012).
Cumulative Summary Metrics for General Features of Vaccine Schedules
The first and most critical step is to define one or more suitable metrics reflecting the general feature of the vaccine schedule that should be evaluated. The number of options is large. Here are some examples:
- The total number of vaccines received before the child’s 6th birthday.
- The total number of health care visits before the child’s 6th birthday on which the child received at least one vaccine.
- The total amount of immunogens (antibody-stimulating proteins and polysaccharides) that a child was exposed to from all vaccines combined (DeStefano et al., 2012).
- The total amount of immunogenic adjuvants that a child was exposed to from all vaccines combined.
- The total amount of the thimerosal preservative that a child was exposed to from all vaccines combined (Price et al., 2010).
For all the metrics listed above, completely unvaccinated children will have a value of zero. These children are at one end of the exposure spectrum, and together with the fully vaccinated children at the other end, they provide the most informative data points for statistical analyses. Hence, they should be included in these types of studies in order to ensure the highest possible statistical power.
All of the above metrics are continuous or ordinal in nature. Each one could be dichotomized into a 0/1 variable. For example, in the first example mentioned above, the children could be split into those receiving at least 10 vaccines and those receiving less than 10 vaccines. Such dichotomization is not recommended. If there is a difference in risks between receiving 9 and 10 vaccines, there is also probably a difference between 5 and 9 vaccines and between 10 and 15 vaccines. Such information is thrown away when the data are dichotomized, and hence, statistical power is lost. It may be tempting to compare only fully vaccinated and completely unvaccinated children, but that is not recommended either. By excluding the children in between, statistical power is lost, as they provide valuable information about the intermediate group. It also makes it impossible to look for a dose-response relationship, as described below.
Electronic health plan data provide one of the best opportunities to study the safety of vaccine schedules with respect to cumulative summary metrics. As the complete vaccination history is needed to calculate the
metric of interest, population-based studies will be limited to children with a sufficiently long enrollment in the same health plan. In some European countries with a national health care system, such as Denmark, these studies are easier to conduct, as only a small percentage of children immigrate to or emigrate from the country. From the U.S. perspective, a drawback of doing these types of studies in foreign countries is, of course, that their recommended vaccine schedule is different from ours.
Once the outcome definition has been decided, the methods that one can use for these more general vaccine schedule components are very similar to those described above concerning the time between vaccine doses. With health plan data, first classify each child according to one or more of the metrics indicated above. Include only children with a sufficiently long enrollment period. As the next step, determine if they have the adverse event of interest during some predefined age period. For the statistical analysis, use logistic regression with the potential adverse event as the dependent variable and the vaccine schedule metric as the independent variable. More than one metric for the vaccine schedule can be included in the same regression model, by which it is possible to try to tease apart the relative influence of each one on the adverse event. Gender, calendar year, study site, and other covariates can be adjusted for by also including them as independent variables in the logistic regression.
To include as many children as possible in the study, irrespective of their length of enrollment, a survival analysis model could be used instead of logistic regression. Children leaving the health plan are then censored at the time of departure. Note, though, that the enrollment period must still be long enough to determine their vaccine schedule in sufficient detail to calculate the metric of interest. It is only the follow-up period that can be sensored.
The key strength of health plan data is the availability of detailed longitudinal vaccination and disease histories for millions of children. They do not contain all potential information of interest, though. If some of the exposure history is unavailable, such as the particular brand of a vaccine, one can instead conduct a case-control study. If the potential adverse event is rare, cases can be identified through the health plan data, together with a set of matched controls. Chart review can then be conducted on this limited population to obtain more detailed information about each of the vaccines given, about the exact nature of the potential adverse event, or about various potential confounders.
There is not necessarily a linear dose-response relationship between the metric of choice and the risk of a potential adverse event, but a linear
function can be used in the regression model as a test for trend. Quadratic and other nonlinear functions can then be explored and formally tested for statistical significance in order to get a better understanding about the dose-response curve.
When doing these types of studies, it is important to look for a dose-response relationship in which an excess risk observed is a monotonically increasing or decreasing function of the summary metric being evaluated. If it instead is, for example, a U-shaped function with a high risk among the least and most vaccinated and a low risk among the middle group, it is likely to be something else responsible for the differences, rather than the cumulative metric under study.
VACCINE SCHEDULE SUMMARY METRICS, INDEPENDENT OF VACCINES RECEIVED
A fundamentally different set of metrics compares children who receive the same set of vaccines, but through different schedules. The question is then exclusively focused on how the vaccinations are scheduled, and we want to adjust away any differences due to the different sets of vaccines received.
The number of potential vaccine schedule summary metrics is large. Here are some examples:
- The maximum number of vaccines received on a single day.
- The maximum amount of immunogens received on a single day (DeStefano et al., 2012).
- The maximum amount of adjuvants received on a single day.
- The average number of vaccines given at each visit. For example, if one child had 15 vaccines spread out over five visits, the average is 3. If another child had only 3 vaccines in total, all given at the same visit, the average is also 3.
- The number of days undervaccinated (Glanz, 2012). For each vaccine, calculate the number of days between the recommended age and the actual age of vaccination and then sum over all vaccines. For a perfectly compliant child, the value is zero.
- The age at which a certain portion of the recommended vaccine schedule was completed. For example, one could take the set of vaccines that CDC recommends for the first 18 months and determine the age at which all of them have been given.
- Average age at which vaccines were given.
None of these definitions is the “right” one, but they serve as examples of what could be used rather than recommendations of what should be
used. The choice will and must depend on the scientific hypothesis that the scientist is evaluating.
Electronic health plan data are the most suitable for studying vaccine schedule metrics. Since the complete vaccination history is needed to calculate the metric of interest (such as average age at vaccinations), a sufficiently long enrollment in the same health plan is needed.
The methods described in the previous section can be used for these types of studies as well, with one critical modification. To truly evaluate the metric of the vaccine schedule and not the collection of vaccines received, the latter must be adjusted for. This can be done by classifying the children by the set of vaccines received and then conditioning the analysis on these sets. In this way, children are compared only with other children that received the same set of vaccines. All children who received at least one vaccine can be included in the same study, maximizing the statistical power. Children who did not receive any vaccines are not informative in this type of study and must be excluded.
COMPARISON OF COMPLETE VACCINE SCHEDULES
Some parents have consciously decided to follow a specific alternative vaccine schedule other than the one recommended by the CDC. Hence, there is a natural interest in comparing the safety of these complete schedules as discrete entities rather than through the various components of the schedule, as discussed in previous sections. For example, one may compare the number of potential adverse events in (i) children that have followed the CDC-recommended vaccine schedule, (ii) children that have followed one of Dr. Bob’s recommended schedules (Sears, 2007), and (iii) children that have not received any vaccines at all. The statistical methods are the same irrespective of which vaccination schedules are compared or whether vaccinated children are compared to completely unvaccinated children. Hence, we treat them together in this section.
In a pioneering study on vaccine schedules, Glanz (2012) used a matched cohort design where children on the CDC-recommended vaccine schedule were matched with children not on that schedule. It was then evaluated whether there were any differences in a few different outcomes: pertussis, upper respiratory infections, fever, sinusitis, outpatient physician visits, and hospitalizations. The data used were from VSD. In a companion study, Hambidge (2012) used the same data to look at febrile seizures.
From a purely scientific perspective, the best design would be a double-blind placebo-controlled randomized trial with an intention-to-treat analysis where children are randomized to one of several vaccination schedules and/or no vaccinations at all. There are major financial, logistical, and ethical issues with conducting such trials, as discussed below.
To avoid ethical issues, observational health plan data can be used as an alternative to randomized trials. As in the previous section, complete vaccination histories are required to classify children into alternative vaccine schedules, so the length of enrollment must be long enough for a sufficiently large number of children.
Very few children are 100 percent compliant with any one particular schedule. Hence, the first challenge with these studies is to define some criteria of how divergent they can be from the schedule and still be considered compliant. For example, one could require that all the recommended vaccines have been received and that the average temporal divergence from the schedule is at most 1 month, taken over all the vaccines. Alternatively, one could require that the sum of the temporal divergences, taken over all vaccines, is at most, say, 1 year. If the criteria are too strict, the sample size will be too low and the statistical power will suffer. If the criteria are too wide, many of the children will not be appropriate representatives of the schedule that they are set to represent. In that case, the study is not actually evaluating the vaccines schedules that it is meant to evaluate.
Children with an incomplete vaccination history must be excluded from the study, together with children whose vaccine schedule is too divergent from either of the schedules being studied.
Once children in the health plan have been classified by the vaccine schedule, a variety of potential adverse events can be studied. A key difficulty here is to define the time period during which the events will be
counted. The cleanest option is to make the vaccine schedule classification based on data up to a certain age and consider only potential adverse events that occur after this age. This ensures that there is no bias if the adverse event studied causes subsequent changes in the vaccination schedule. It is not an ideal solution, though, since we must ignore either the adverse events occurring early during the vaccination schedule or the possible effect of later parts of the vaccination schedule. If the potential adverse event under study is such that there is little risk that its presence will change any aspect of the vaccination schedule, then one could include adverse events that occur before the end of the vaccine schedule considered, but such an approach is risky.
Since different children will have different lengths of follow-up, time-to-event data will best be analyzed by survival analysis methods, adjusting for possible confounders. When two alternative schedules are compared, an alternative way to adjust for covariates is to use a matched cohort design (Glanz, 2012), where each child with the recommended schedule is matched with a child with the alternative schedule having the same age, gender, study site, and calendar year of immunization. This is especially useful if additional health data that are not available in the health plan data sets are to be gathered from the children, since it is typically infeasible to collect such data for all children in a health plan.
To use observational health plan data to compare complete vaccination schedules, there must be enough children in the health plan that follow each schedule under study. That may be a problem if one of the comparison groups consists of followers of a very specific alternative schedule or of completely unvaccinated children. It may then be necessary to include a larger number of health plans or health plans with a larger number of members.
ADDITIONAL METHODOLOGICAL ISSUES
Bias and Confounding
The observational study designs described above are, like all observational studies, prone to various sources of bias. The type of bias is often different for different designs, and it is hence often wise to use multiple study designs for the same question. Of special concern in these types of studies is confounding due to the innocent bystander effect. This is when an adverse event that is seemingly due to one component of the vaccine schedule under study is actually due to another component that is correlated with the first one.
As a first step, it is natural to do a study considering a single component of the vaccine schedule, adjusting only for demographic variables. If
a significant relationship is observed, though, it is sometimes important to consider other aspects of the schedule as possible confounders. This is especially true for late-onset events. This can be tackled in one of several ways. As a second step, additional studies evaluating other components of the vaccine schedule can be conducted. One way to do this is to incorporate multiple components in the same regression model. For example, one regression model may include variables representing the age at which each vaccine was given, the total number of vaccines given, the total exposure to immune-stimulating content, and the total exposure to adjuvants. Such a design will provide information as to whether the component that is suspected of being the culprit still has a statistically significant relationship with the outcome after adjustment for other components of the schedule. It is important to note, though, that with many different correlated components in the same schedule, none may be statistically significant after adjustment for all the others. This does not mean that the risk of the adverse events does not depend on the vaccine schedule. It just means that it is not possible to determine which component is responsible, and that is important information.
Combining Health Plan Data with Study-Specific Data Collection
In the data sections presented above, health plans are often recommended as the best source of data to use. There are some potential adverse events that are not fully captured in the electronic health plan data, though, such as neuropsychological performance or immune function. Such outcomes must be measured specifically for a research study, but that can obviously not be done for all members of a health plan. Depending on the outcome, there are at least three different ways to go about doing this:
- Select a random sample of children from the health plan, measure the outcome of interest for each one, and evaluate the relationship between the relevant component of the vaccine schedule and the outcome measurement values. This is the simplest approach.
- Select a nonrandom sample of children from the health plan, oversampling children on both end of the metric used in the study. For example, if the variable of interest is the timing of the first dose of the hepatitis B vaccine, children who received it long after birth would be oversampled. After that, proceed as described above. This design will in many cases increase the statistical power. It is important that the probability of selection be unrelated to other health events.
- Select a nonrandom sample of children from the health plan, on the basis of a health outcome that is present in the health plan data and that is correlated with the outcome of interest. The goal here is to get a larger variance in the health outcome being measured, thereby increasing the statistical power. It is important that the probability of selection be unrelated to any aspect of the vaccine schedule. After the study population has been selected, proceed as described for the first scenario.
Vaccine-Specific Versus General Components of Vaccine Schedules
The more general components of the vaccine schedule described above, as well as the comparison between complete schedules discussed earlier, are considerably more difficult to study than the more vaccine-specific components described above. There are several reasons for this.
First and foremost, there are many alternative vaccine schedules, and slightly different schedules have to be lumped together in the same comparison group. For the cumulative summary metrics, many different vaccination schedules will have the same value, for example, the average age at vaccination. If one vaccine schedule is safer than an alternative vaccine schedule in terms of a specific outcome but they both have the same average age at vaccination, then the effect size will be attenuated and go undetected.
If a statistically significant excess number of adverse events is found, a second problem with these designs is that it can be hard to know which aspect of the schedule caused the excess or reduced risk. Is it the timing of one specific vaccine, is it an interaction between two or more vaccines, or is it something else? Hence, any statistically significant findings will have to be followed up with studies concerning more specific vaccine schedule components.
A third issue is confounding. While confounding is present in all observational studies, it is likely to be a greater problem when complete vaccine schedules are studied. For example, children for whom most of the vaccines are delayed from the recommended schedule may be different in terms of both health care utilization and socioeconomic factors. This may bias the results, and the bias may exist whether the delayers are deliberately following an alternative schedule or not, and the bias may go in different directions for these two groups. The same type of confounding can be present when one is looking at more specific components of the vaccine schedule, but it is likely to be less strong, as such individual components are likely to have more random and less systematic variability than a complete schedule. A way to intuitively see this is to note that whatever it is that causes a general parental tendency to delay vaccinations, that will likely be more correlated with the average timing of all vaccinations than with the timing of a single vaccination.
To date, there have been few comparative studies evaluating the safety
of different vaccine schedules. For the above-mentioned reasons, it is suggested that, initially, the majority of such studies focus on the most vaccine-specific components of the vaccine schedule described earlier, as well as the content-defined components mentioned above. Information from such studies will greatly facilitate the design and understanding of subsequent studies evaluating the more general components discussed earlier as well as the comparison of complete vaccines schedules described above.
Different countries have different recommended vaccine schedules, so it may seem natural to do cross-national studies to compare the safety of the schedules in an ecological study design. Unfortunately, this is very difficult to do well and generally not recommended. The problem is that the incidence of most diseases varies by geographical region for reasons other than the vaccine schedule, such as genetics, diet, physical exercise, or other environmental factors. Any such cross-national study may hence be heavily biased. This does not mean that one cannot do studies that include data from multiple countries or regions, as long as each one has a range of different exposures in each place. In such studies, the geographical region can easily be adjusted for in the analysis, in order to take the differential disease incidences into account.
Time Trend Evaluations
Another ecological study design is to take a particular country or region and compare time trends in disease incidence with temporal changes in the vaccination rate or vaccination schedule. This is also not recommended. In addition to vaccinations, there are many other reasons why the reported disease incidence may increase or decrease over time, including changes in environmental risk factors and changes in health care practice and diagnosis. Hence, an apparent temporal correlation between increasing disease incidence and increasing vaccine usage could be completely spurious. It should be pointed out that the bias can also go in the other direction. Even if there is no temporal correlation between disease incidence and vaccinations, a true relationship can be hidden by a compensatory effect from an unknown confounder.
Near-Real-Time Safety Surveillance
In what is called “rapid-cycle analysis,” the VSD project has pioneered near-real-time vaccine safety surveillance (Lieu et al., 2007; Yih et al., 2011). For newly approved vaccines and selected adverse events, weekly
data feeds are received from the health plans and the data are analyzed by continuous sequential statistical methods (Kulldorff et al., 2011). If there are specific concerns regarding the newly revised vaccine schedule, such rapid-cycle analysis can also be implemented for many of the study designs described above.
Most vaccine safety studies evaluate a specific vaccine-event pair. For VAERS data, data mining methods are also used, where thousands of potential vaccine and adverse event pairs are evaluated simultaneously, without there being any prior hypothesis about their being an excess risk of the event. This is done to cast the net as wide as possible. Recently, data mining methods are also started to be used for health plan data. As vaccine safety data mining develops further, it may also be used to study questions regarding the vaccine schedule.
Disease-Causing Complications Versus Adverse Events
It should be noted that in these types of studies, it is not always clear what is an adverse event and what is not. For example, a child may have a febrile seizure that was caused by one or more of the vaccines or a febrile seizure caused by a disease, where the child got the disease because he or she was not immunized against it. Hence, an excess risk of seizures due to a particular vaccine schedule could be due either to vaccines given at a certain time when the child is more sensitive to adverse events or to vaccines not given at a certain time when the child needed the vaccine protection.
If the same type of health event is caused by the vaccine among one group of children as an adverse event and by the disease among the same or another group of children as a complication, then the vaccine may be found to cause an excess number of the adverse events in a vaccinated population, since the nonvaccinated children benefit from herd immunity. Hence, findings about the risk to individuals in a mostly vaccinated population cannot necessarily be generalized to the population level.
Vaccine Efficacy and Effectiveness
This paper covers only the study of potential adverse events after vaccination. If a study does not find an excess risk, all is fine and there is no need to worry about vaccine efficacy. On the other hand, if a true differential risk of adverse events is found with respect to some component of the vaccine schedule, vaccine efficacy and effectiveness must also be considered when contemplating a revised vaccine schedule. Some vaccines, such as MMR,
have a different immune response depending on the age of the child, and vaccine efficacy therefore depends on the vaccine schedule. The timing of a vaccination also influences the time period during which the child is protected from the disease and the herd immunity of the population at large. Herd immunity can also be affected if a parent refuses future vaccinations after his or her child has had an adverse event vaccine that could have been avoided with a different schedule. While such an analysis is outside the scope of this paper, all these factors must be considered in a joint cost-benefit analysis before the recommended vaccine schedule is revised, if and when there is a finding of a vaccine schedule-dependent adverse event.
When deciding what to study and what study design to use, cost is an important consideration. The study designs mentioned in this paper range from very cheap to very expensive. For some designs, the cost depends on how common the potential adverse event is. While we cannot do any precise sample size calculations, we will for illustrative purposes consider three classes of health outcomes: common, moderately rare, and very rare. Common events are those that affect more than 1 out of every 100 children, such as allergies and some learning disorders. Moderately rare outcomes are those that affect more than 1 out of 100 but less than 1 in 10,000, such as intussusception. Very rare outcomes are those affecting less than 1 in 10,000, such as Guillain-Barré syndrome,
The least expensive studies are those using VAERS data. Since those data are already collected, only the investigator’s time needs to be covered. Unfortunately, VAERS data are of limited use when one is studying vaccine schedules. The cost is independent of the adverse event.
The second least expensive study designs are the ones based on fully automated health plan data. While they involve no new data collection, the extraction of data from large administrative databases is a complex activity involving detailed knowledge of the database structure and content, sophisticated computer programming, and thorough data quality control. To set up a new system from scratch is very costly, but the marginal cost of additional studies in existing systems is not. In most cases, the cost is independent of the potential adverse event under study. For common and medium-rare outcomes, a VSD-size study population of about 100,000 annual births should be enough for most study designs. For very rare outcomes, data from more and larger health plans may be needed in order to achieve sufficient power, resulting in additional expenses. Bigger datasets may also be needed for common events and moderately rare adverse events when complete vaccination schedules are evaluated, if only a small proportion of health plan members follow the particular schedule of interest. In
summary, the cost of these types of studies is similar to the cost of current vaccine safety studies conducted in VSD, PRISM, and similar systems.
With most health plan data, vaccine information has a high positive predictive value, but that is usually not the case for disease outcomes. For such adverse events, it is often necessary to conduct chart reviews to confirm whether or not a patient actually had the health event of interest, and that will increase the cost. For very rare adverse events, it is not a major additional cost, but for medium-rare and common adverse events, it can be. One way to reduce this cost is to first do a study on fully automated data and do chart review only when that study shows an excess risk of adverse events, to confirm or dismiss that finding.
The next level of cost is incurred by study designs that combine health plan data with specially collected outcome data that are not available as part of the EMRs. The cost will depend on the type of data collected but will in most situations be very high. For medium-rare and very rare outcomes, a very large number of health plan enrollees will have to be enrolled, potentially making such studies prohibitively expensive.
Randomized trials are the most expensive study design. For medium-rare and very rare adverse events, the study needs to have a very large sample size to detect a potential problem. For example, if a vaccine causes a specific adverse event in 1 of every 1,000 children, that is not something that can be detected in a randomized trial with 4,000 children in each arm, for a total of 8,000, even if the baseline rate of the event is very rare. To see this, suppose that there are four adverse events in the vaccinated arm and none in the control arm receiving the placebo. Under the null hypothesis, the probability of all four being in the vaccinated arm is (1/2)4 = 0.0625, which is not statistically significant, and hence, we cannot conclude that it was the vaccine that caused the adverse events. So, for medium-rare and very rare adverse events, we need data with tens or hundreds of thousands of vaccinated children, and for such sample sizes, randomized trials are prohibitively expensive. A cost advantage of randomized trials over case-control studies is that multiple potential adverse events can be evaluated within the same study.
Irrespective of the design, studies evaluating late-onset events are more expensive than those evaluating early-onset events, since the individuals must be followed for a much longer time. With health plan data, this requires larger population sizes since many children will be lost to follow-up. When health plan data are augmented with specially collected data on health outcomes, children must be tracked and monitored for a longer time, which is costly. The same is true for randomized trials.
As with all medical research, ethical considerations are very important when one is designing vaccine safety studies. With observational studies with health plan data, the key ethical issue is patient confidentiality, which can be ensured through existing research practices.
For randomized trials, ethical considerations play a much more important role. Depending on the vaccine component of interest, a randomized trial can sometimes be conducted in a way so that both arms fall within the recommended vaccination schedule, in which case there are no ethical concerns. An example of such a trial would be whether to give children two vaccines on the same day or a week apart. At the other extreme, it would be unethical to do a randomized trial where children in one arm are completely unvaccinated, since the scientist will then knowingly put some of the children at increased risk for vaccine-preventable diseases, some of which may result in death. Somewhere in between these two extremes there is a gray zone where randomized trials may or may not be ethical, depending on the vaccine schedules being compared and on the available strength of the evidence regarding efficacy and potential adverse events. Experts on medical ethics should then be consulted.
For more common adverse events, randomized trials have a potential role to play in postmarketing vaccine safety studies. There is little reason to use them to evaluate the general safety of a particular vaccine, since that evaluation is already covered by the Phase III trials. Questions for which randomized trials may be used include the order in which different vaccines are given, the exact timing between doses of the same vaccine, and whether two different vaccines are given on the same day or a week apart.
If a randomized trial is conducted, it is important to consider the effect on herd immunity. If the two arms differ by delaying one or more vaccines by at most a few weeks, it is not a major issue. If vaccination in one arm is delayed for a much longer time period or not given at all, it may reduce herd immunity. This may put children that are not participating in the study at increased risk for the disease, and this can be especially serious for immune-compromised children for which a vaccination is contraindicated. To minimize the negative effect on herd immunity, such randomized trials should be spread out geographically, so that there are at most a few additional unvaccinated children in any given location. In that way, nonparticipating children will not be at an increased risk of the disease, and equally important, those children randomized to the delayed vaccination will still have some protection against the disease from herd immunity.
Randomized trials are the “gold standard” for scientific studies, and premarketing Phase III randomized trials play an important role in the evaluation of vaccine-related adverse events. Because of their limited sample size, rare adverse events may not be detected, though. For financial and ethical reasons, the utility of randomized trials is more limited for postmarketing vaccine safety studies.
On the basis of utility and cost, health plan–based study designs are the most promising for the safety evaluation of different vaccine schedules. This is definitely true for medium-rare and very rare adverse events that cannot be detected in Phase III randomized trials, but such data can also be used to study common adverse events. The key is to always consider potential problems with confounding, and it is often a good idea to use different study designs with different potential biases for the same research question.
Hypotheses about potential adverse events may come from Phase III trials or from observational postmarketing studies with data from health plans or spontaneous reporting systems.
The comparative safety evaluation of different vaccine schedules is a complex and multifaceted task, and all aspects of the vaccine schedule are currently understudied with regards to potential adverse events. A number of different study designs and methods can be used to evaluate different components of the schedule. For all known and most potential adverse events, it is recommended that a wide variety of vaccine schedule components be evaluated. Direct evaluation of complete vaccine schedules is more difficult and probably less fruitful, but it is not impossible. Such studies are most useful when conducted in parallel with studies of specific components of the schedule. This is especially important when there is a significant adverse event finding, since it is otherwise impossible to know which of the many features of the complete schedule are actually causing the adverse events.
This paper should not be utilized as a cookbook where definite study designs and methods are obtained and used for different classes of problems in a black box–type approach. Each study is unique, depending on the vaccine(s) under study, the potential adverse event(s) of interest, the data used, and the scientific research question. All those aspects need to guide the methodology. The goal of this paper is simply to show that a wide variety of study designs and methods are available to study the comparative safety of different vaccine schedules, and the hope is that some of the proposed methods can serve as a starting point when thinking about the most suitable designs and statistical methods to use for different studies.
This paper does not present an exhaustive list of study designs and methods that can be used for the comparative evaluation of potential
adverse events due to different childhood vaccination schedules. As more such studies are performed, additional designs and methods will surely be developed and used. The paper should not be interpreted as a recommendation to use all of the study designs and statistical methods mentioned. The scientific question should drive which designs and methods are used, and while some of them may become widely used, others may not be used at all.
What the paper attempts to show is that the comparative safety evaluation of vaccine schedules is complex and multifaceted and that a wide variety of study designs and statistical methods are available to a scientist who wishes to conduct such studies.
Aaby, P., H. Jensen, J. Gomes, M. Fernandes, and I. M. Lisse. 2004. The introduction of diphtheria-tetanus-pertussis vaccine and child mortality in rural guinea-bissau: An observational study. International Journal of Epidemiology 33(2):374-380.
Almenoff, J. S., W. DuMouchel, L. A. Kindman, X. Yang, and D. Fram. 2003. Disproportionality analysis using empirical Bayes data mining: A tool for the evaluation of drug interactions in the post-marketing setting. Pharmacoepidemiol Drug Safety 12(6):517-521.
Baggs, J., J. Gee, E. Lewis, G. Fowler, P. Benson, T. Lieu, A. Naleway, N. P. Klein, R. Baxter, E. Belongia, J. Glanz, S. J. Hambidge, S. J. Jacobsen, L. Jackson, J. Nordin, and E. Weintraub. 2011. The Vaccine Safety Datalink: A model for monitoring immunization safety. Pediatrics 127(Suppl 1):S45-S53.
Bate, A., M. Lindquist, I. R. Edwards, S. Olsson, R. Orre, A. Lansner, and R. M. De Freitas. 1998. A Bayesian neural network method for adverse drug reaction signal generation. The European Journal of Clinical Pharmacology 54(4):315-321.
CDC (Centers for Disease Control and Prevention). 2012. Recommended immunization schedule for persons aged 0 through 6 years—United States, 2012. Atlanta, GA: Centers for Disease Control and Prevention.
Chen, R. T., J. W. Glasser, P. H. Rhodes, R. L. Davis, W. E. Barlow, R. S. Thompson, J. P. Mullooly, S. B. Black, H. R. Shinefield, C. M. Vadheim, S. M. Marcy, J. I. Ward, R. P. Wise, S. G. Wassilak, and S. C. Hadler. 1997. Vaccine Safety Datalink project: A new tool for improving vaccine safety monitoring in the United States. The Vaccine Safety Datalink team. Pediatrics 99(6):765-773.
DeStefano, F., and the Vaccine Safety Datalink Research Group. 2001. The Vaccine Safety Datalink project. Pharmacoepidemiol Drug Safety 10(5):403-406.
DeStefano, F., E. Weintraub, and C. Price. 2012. Immunogenic stimulation from vaccines and risk of autism. Paper presented at Vaccine Safety Datalink Annual Meeting, Denver, CO.
DuMouchel, W. 1999. Bayesian data mining in large frequency tables, with an application to the FDA spontaneous reporting system. Journal of the American Statistical Association 53(3):177-190.
Eng, P. M., T. C. Mast, J. Loughlin, C. R. Clifford, J. Wong, and J. D. Seeger. 2012. Incidence of intussusception among infants in a large commercially insured population in the United States. Pediatric Infectious Disease Journal 31(3):287-291.
Evans, S. J., P. C. Waller, and S. Davis. 2001. Use of proportional reporting ratios (PRRs) for signal generation from spontaneous adverse drug reaction reports. Pharmacoepidemiol Drug Safety 10(6):483-486.
Fireman, B., J. Lee, N. Lewis, O. Bembom, M. van der Laan, and R. Baxter. 2009. Influenza vaccination and mortality: Differentiating vaccine effects from bias. American Journal of Epidemiology 170(5):650-656.
Gargiullo, P. M., T. V. Murphy, and R. L. Davis. 2006. Is there a safe age for vaccinating infants with tetravalent rhesus-human reassortant rotavirus vaccine? The Journal of Infectious Diseases 194(12):1793-1794.
Glanz, J. 2012. Alternative immunization schedules and the risk of pertussis. Paper presented at Vaccine Safety Datalink Annual Meeting, Denver, CO.
Hambidge, S. J. 2012. Seizures in children with recommended versus delayed infant immunization schedules. Paper presented at Vaccine Safety Datalink Annual Meeting, Denver, CO.
Hambidge, S. J., J. M. Glanz, E. K. France, D. McClure, S. Xu, K. Yamasaki, L. Jackson, J. P. Mullooly, K. M. Zangwill, S. M. Marcy, S. B. Black, E. M. Lewis, H. R. Shinefield, E. Belongia, J. Nordin, R. T. Chen, D. K. Shay, R. L. Davis, F. DeStefano, and the Vaccine Safety Datalink Team. 2006. Safety of trivalent inactivated influenza vaccine in children 6 to 23 months old. Journal of the American Medical Association 296(16):1990-1997.
Hsiao, C. J., E. Hing, T. C. Socey, and B. Cai. 2011. Electronic health record systems and intent to apply for meaningful use incentives among office-based physician practices: United States, 2001-2011. NCHS Data Brief (79):1-8.
Iqbal, S., and F. DeStefano. 2012. Immunological-stimulating content of vaccines and neurophysiological outcomes in children. Paper presented at Vaccine Safety Datalink Annual Meeting, Denver, CO.
Klein, N. P., B. Fireman, W. K. Yih, E. Lewis, M. Kulldorff, P. Ray, R. Baxter, S. Hambidge, J. Nordin, A. Naleway, E. A. Belongia, T. Lieu, J. Baggs, E. Weintraub, and Vaccine Safety Datalink. 2010. Measles-mumps-rubella-varicella combination vaccine and the risk of febrile seizures. Pediatrics 126(1):e1-e8.
Kramarz, P., E. K. France, F. DeStefano, S. B. Black, H. Shinefield, J. I. Ward, E. J. Chang, R. T. Chen, D. Shatin, J. Hill, T. Lieu, and J. M. Ogren. 2001. Population-based study of rotavirus vaccination and intussusception. Pediatric Infectious Disease Journal 20(4):410-416.
Kulldorff, M., R. L. Davis, M. Kolczak, E. Lewis, T. Lieu, and R. Platt. 2011. A maximized sequential probability ratio test for drug and vaccine safety surveillance. Sequential Analysis 30(1):58-78.
LaRussa, P. S., K. M. Edwards, C. L. Dekker, N. P. Klein, N. A. Halsey, C. Marchant, R. Baxter, R. J. Engler, J. Kissner, and B. A. Slade. 2011. Understanding the role of human variation in vaccine adverse events: The Clinical Immunization Safety Assessment Network. Pediatrics 127(Suppl 1):S65-S73.
Leonardi, M., K. Bromberg, R. Baxter, J. L. Gardner, S. Klopfer, O. Nicholson, M. Brockley, J. Trammel, V. Leamy, W. Williams, B. Kuter, and F. Schodel. 2011. Immunogenicity and safety of MMRV and PCV-7 administered concomitantly in healthy children. Pediatrics 128(6):e1387-e1394.
Lieu, T. A., M. Kulldorff, R. L. Davis, E. M. Lewis, E. Weintraub, K. Yih, R. Yin, J. S. Brown, R. Platt, and for the Vaccine Safety Datalink Rapid Cycle Analysis Team. 2007. Real-time vaccine safety surveillance for the early detection of adverse events. Medical Care 45(10 Suppl 2):S89-S95.
Murphy, T. V., P. M. Gargiullo, M. S. Massoudi, D. B. Nelson, A. O. Jumaan, C. A. Okoro, L. R. Zanardi, S. Setia, E. Fair, C. W. LeBaron, M. Wharton, J. R. Livengood, and the Rotavirus Intussusception Investigation Team. 2001. Intussusception among infants given an oral rotavirus vaccine. New England Journal of Medicine 344(8):564-572.
Nguyen, M., R. Ball, K. Midthun, and T. A. Lieu. 2012. The Food and Drug Administration’s post-licensure rapid immunization safety monitoring program: Strengthening the federal vaccine safety enterprise. Pharmacoepidemiol Drug Safety 21(Suppl 1):S291-S297.
Norén, G. N., R. Sundberg, A. Bate, and I. R. Edwards. 2008. A statistical methodology for drug-drug interaction surveillance. Statistics in Medicine 27(16):3057-3070.
Pittman, P. R. 2002. Aluminum-containing vaccine associated adverse events: Role of route of administration and gender. Vaccine 20(Suppl 3):S48-S50.
Price, C. S., W. W. Thompson, B. Goodson, E. S. Weintraub, L. A. Croen, V. L. Hinrichsen, M. Marcy, A. Robertson, E. Eriksen, E. Lewis, P. Bernal, D. Shay, R. L. Davis, and F. DeStefano. 2010. Prenatal and infant exposure to thimerosal from vaccines and immunoglobulins and risk of autism. Pediatrics 126(4):656-664.
Rothman, A. L. 2011. Immunity to dengue virus: A tale of original antigenic sin and tropical cytokine storms. Nature Reviews Immunology 11(8):532-543.
Rothman, K. J., Y. Young-Xu, and F. Arellano. 2006. Age dependence of the relation between reassortant rotavirus vaccine (RotaShield) and intussusception. The Journal of Infectious Diseases 193(6):898.
Rowhani-Rahbar, A., N. Lewis, B. Fireman, R. Baxter, N. Klein, and other VSD colleagues. 2012. Effect of age on the risk of seizure following immunization with measles-containing vaccines: VSD study 233—preliminary results. Paper presented at Vaccine Safety Datalink Annual Meeting, Denver, CO.
Sears, R. 2007. The vaccine book: Making the right decision for your child, 1st ed. New York: Little, Brown.
Thakrar, B. T., S. B. Grundschober, and L. Doessegger. 2007. Detecting signals of drug-drug interactions in a spontaneous reports database. The British Journal of Clinical Pharmacology 64(4):489-495.
Tse, A., H. F. Tseng, S. K. Greene, C. Vellozzi, G. M. Lee, and the VSD Rapid Cycle Analysis Influenza Working Group. 2012. Signal identification and evaluation for risk of febrile seizures in children following trivalent inactivated influenza vaccine in the Vaccine Safety Datalink project, 2010-2011. Vaccine 30(11):2024-2031.
Veirum, J. E., M. Sodemann, S. Biai, M. Jakobsen, M. L. Garly, K. Hedegaard, H. Jensen, and P. Aaby. 2005. Routine vaccinations associated with divergent effects on female and male mortality at the paediatric ward in Bissau, Guinea-Bissau. Vaccine 23(9):1197-1204.
Yih, W. K., M. Kulldorff, B. H. Fireman, I. M. Shui, E. M. Lewis, N. P. Klein, J. Baggs, E. S. Weintraub, E. A. Belongia, A. Naleway, J. Gee, R. Platt, and T. A. Lieu. 2011. Active surveillance for adverse events: The experience of the Vaccine Safety Datalink project. Pediatrics 127(Suppl 1):S54-S64.