Evidence of Bias in Studies of Influenza Vaccine Effectiveness in Elderly Patients

Influenza is associated with significant morbidity [1, 2]. Vaccines have been available for many years, and for over 4 decades, they have been recommended in the United States for individuals who are elderly or have other medical illnesses [3]. These recommendations were based on extrapolations of data on younger and healthier persons [4]. Only 1 randomized controlled trial among the elderly population was completed, which showed decreasing vaccine efficacy against influenza with increasing age [5]. Because of the availability of effective influenza vaccines, randomized placebo-controlled trials would be unethical, but observational studies of influenza vaccine efficacy, which have predominated since the 1970s, have generally shown excellent vaccine effectiveness in preventing death from all causes [6, 7], with reductions in mortality during influenza season of 30%–50%.

Recently, this benefit has been called into question. Simonsen et al [8] found that total excess mortality from influenza was in the range 5%–10%, making claims of 30%–50% reduction in mortality from the vaccine unrealistic. Jackson et al [9] reported that vaccination appeared to be even better at preventing death before the influenza season than during the time when the virus was circulating. This revealed a flaw in observational studies, and what had previously been taken for vaccine effectiveness was then thought to actually be selection bias.

If bias is the reason that the influenza vaccine seems to work so well, what are the sources of this bias, and is it possible to adjust for them to determine the true value of the vaccine? We recently described a new method for bypassing bias to determine the true effectiveness of the seasonal influenza vaccine [10]. In the present study, we investigate the relationship between vaccination and markers of death, to reveal potential sources of bias in the observational studies.

Methods. The study population consisted of elderly patients who were members of the Kaiser Permanente of Northern California (KPNC) health insurance system. KPNC provides care to >3.1 million members, who receive essentially all care within the system, including outpatient care, hospitalization, medications, and vaccinations. Throughout the study period, KPNC provided outreach programs each autumn to inform health plan members ⩾65 years old that they could be vaccinated conveniently and at no cost; vaccination campaigns began after the second week of October. Patients in skilled nursing facilities and nursing homes generally received their influenza vaccinations via the KPNC pharmacies, and immunization records returned from these facilities were recorded in the KPNC databases, which include all information on members' medical encounters, including diagnoses, laboratory tests, vaccinations, and medications. With approval from the KPNC institutional review board, we accessed the records, from October 2001 through September 2005, of all KPNC members ⩾65 years old. The age, sex, and membership status of the study population were ascertained from KPNC administrative databases. The KPNC immunization tracking system was the source of information on receipt of influenza vaccinations. Information on inpatient and outpatient diagnoses was obtained from KPNC clinical databases, as were claims for services delivered outside of but covered by KPNC. Specified health conditions were identified through the KPNC disease registries. Data on mortality (including date, site, and cause of death) were obtained from California death certificate files.

During the study period, specimens were sent to a regional laboratory in Berkeley, California, to be tested for the influenza virus. Specimens were tested for influenza virus types A or B with direct immunofluorescence and culture. Each year of the study period, the beginning and end of the influenza season was identified on the basis of the number and percentage of specimens that tested positive for influenza virus.

In addition to maintaining clinical databases, KPNC assigns each member a risk score that is calculated using DxCG software (Verisk Health). The risk score predicts future cost on the basis of underlying health conditions, determined from previous diagnoses and procedures [11].

We plotted vaccination coverage in relation to age, sex, and diagnostic risk score. We calculated a predicted probability of death from a logistic regression of death on age, diagnoses from the disease registries, and risk score and then plotted vaccination coverage in relation to the predicted probability of death.

For members ⩾75 years old, we stratified the population by the number of influenza vaccinations that had been received in the previous 5 years. Within each stratum, we examined mortality during the months of January through April in relation to vaccinations during the previous years, after adjusting for age, sex, 4 chronic diseases (heart failure, chronic obstructive pulmonary disease, diabetes mellitus, and coronary artery disease), and whether a pneumonia vaccination had been received. Weeks when influenza virus was circulating were excluded from this analysis to illustrate selection bias. This study was approved by and followed all guidelines from our institution's institutional review board.

Results. During the 4-year study period, the total study population of KPNC members ⩾65 years old increased from ∼350,000 to ∼400,000, of which 1,153,939 (77.3%) were <80 years old. The study included 1,492,351 person-years of followup. Of the total study population, 28,673 members died during the study period. The underlying health of 68.2% of the population was ranked as better than average, according to the risk score; 5.4% of the population had a risk score of ⩾2 (twice the average predicted cost).

The percentage of the population that was vaccinated varied with age. After age 65, influenza vaccination increased until age 78 in women and age 81 in men, then decreased with increasing age. Vaccination coverage also varied in a curvilinear fashion with risk score, increasing with risk score to a risk score percentile of ∼80%, then decreasing (figure 1). In addition, as the predicted probability of death increased, vaccination coverage increased. Vaccination coverage was highest among members with a probability of death of 3%–7.5%. Those with a predicted probability of death in the coming year of >17.5% had a decreasing likelihood of influenza vaccination (figure 1).

View larger version:

• In this window
• In a new window

• Download as PowerPoint Slide

Figure 1.

Influenza vaccination coverage as a function of risk score percentile and as a function of probability of death in the upcoming year. Probability of death in the next 12 months was calculated from a logistic regression of age, diagnoses, and health status.

A change in the pattern of vaccination had a striking effect on mortality. For members ⩾75 years old who had been receiving influenza vaccinations in previous years, not receiving a seasonal influenza vaccination was strongly associated with mortality in the months ahead (table 1). A person who had received an influenza vaccination every year in the previous 5 years had a more than double probability of death outside the influenza season if he or she missed a vaccination in the current year, compared with a person who was vaccinated as usual (odds ratio, 2.17; P<.001). On the other hand, if a person did not receive any seasonal influenza vaccination in the previous 5 years, then receipt of a vaccination in the current year was associated with a greater probability of death. This is shown in table 1 by the odds ratio of 0.67 for members with no previous vaccinations, which can also be interpreted as an odds ratio of 1/0.67=1.49 of death for members who did receive the vaccine. As a reminder, because we examined mortality only during weeks when influenza virus was not circulating, these associations between vaccination and mortality are due to bias, not the effect of the vaccine.

View larger version:

• In this window
• In a new window

• Download as PowerPoint Slide

Table 1.

Association between Mortality and Vaccination in People with Routine Influenza Vaccination and in People with No Previous Influenza Vaccination

Discussion. The discrepancy in deaths between vaccinated and unvaccinated individuals has previously been taken as a marker of effectiveness of the vaccine [7, 12]. Our study confirms that selection bias greatly confounds the analysis of observational studies of influenza vaccine effectiveness in elderly patients [8, 9]. To clarify the source of this bias, we looked at the relationship between vaccination and age, sex, and underlying illness. We showed that, despite strong efforts to increase vaccination among the elderly population, vaccination is relatively low in the oldest and sickest portions of the population. Persons >65 years old with a >7.5% chance of death in the upcoming year are less likely to receive the influenza vaccine. Because persons who are most likely to die are less likely to receive the vaccine, vaccination appears to be associated with a much lower chance of dying; thus, the “effectiveness” of the vaccine is in great part due to the selection of healthier individuals for vaccination, rather than due to true effectiveness of the vaccine. Previous studies have argued that worsening health is associated with increasing vaccination. We found this to be a curvilinear relationship, in which increasing illness means increasing vaccination, up to a point, and then, as people come closer to the end of life, there is a decrease in vaccination coverage. A disproportionate number of deaths occur among people at the extremes of age and among those with underlying illness, and people closer to the end of life appear to forego influenza vaccination.

Why is missing a vaccination an especially strong predictor of death in elderly patients who have been receiving the vaccine regularly for the past 5 years? Perhaps people who have been receiving regular vaccinations but then forego them do so because of frailty, decreased mobility, or other factors associated with the end of life. Because we studied the association between vaccination and mortality only outside the influenza season, the effect is a measure of bias, rather than of the vaccine itself. For those who never receive vaccination, we might expect that a healthier subgroup is selected, one whose doctors promote vaccination less aggressively, because the patients are doing well. This is borne out by the “protective” effect of not receiving the vaccine after years of the same. If, in one year, a person starts receiving vaccination, after years of not doing so, then perhaps it is due to the onset of a new chronic disease, which prompts more aggressive promotion of the vaccine. In both cases, the patient has moved along the curve of increasing morbidity, but receipt of the vaccine appears to have a very different effect, depending on past vaccination.

Why are patients near death not getting vaccinated, despite outreach programs? Decreased mobility may lead to an inability to come in for the vaccine. On the other hand, patients may decide to forgo further intervention when approaching death. Although reported functional status may improve adjustment for confounders, bringing the preseason benefit closer to zero [13], this additional marker of frailty is difficult to derive from administrative data. We suspect it may not be possible to adjust for the curvilinear relationship between underlying morbidity and vaccination, because the severity of the underlying morbidity is difficult to measure and changes over time.

Elsewhere we have shown that it is possible to obtain a better estimate of the effectiveness of the influenza vaccine by using a difference-in-differences approach, whereby the apparent effect of the vaccine outside the season could be subtracted from the benefit while the virus is circulating [10], in a sense bypassing the underlying bias. But from our results it appears that, regardless of the method used to determine effectiveness, the vaccine will prove to be less effective at preventing death than was previously thought.We hope this knowledge will stimulate research into better vaccines for elderly patients (perhaps by use of higher doses or adjuvants) and will lend more weight to the importance of vaccinating schoolchildren to prevent disease in the rest of the population [14].

• Received June 23, 2009.
• Accepted August 31, 2009.

• © 2010 by the Infectious Diseases Society of America