Bias alone could account for benefit attributed to flu vaccine, study finds

BMJ 2008; 337 doi: (Published 03 September 2008) Cite this as: BMJ 2008;337:a1550
  1. Jeanne Lenzer
  1. 1New York

    The reduction in mortality seen among elderly people who have been given the flu vaccine could have more to do with a “healthy user” effect than any protective value of the vaccine itself, a study has found.

    The researchers, from the University of Alberta in Edmonton, Canada, prospectively obtained clinical, laboratory, and functional status data and identified 1813 adults with community acquired pneumonia who were admitted to six hospitals in Alberta (American Journal of Respiratory and Critical Care Medicine 2008;178:527-33, doi:10.1164/rccm.200802-282OC). Of these adults, 354 vaccine recipients met the inclusion criteria and were matched with 354 controls.

    The outcome of in-hospital mortality was examined during the months before and after the flu season, when no benefit of the flu jab would be expected.

    The researchers replicated a 51% mortality difference between the groups found in previous observational studies (JAMA 1994;272:1661-5): 53 (15%) of the vaccinated control patients and 28 (8%) vaccine recipients died in hospital (unadjusted odds ratio 0.49 (95% confidence interval 0.3 to 0.79)).

    The dramatic benefit shown of vaccination before and after the flu season could have been due only to a baseline imbalance or bias, say the researchers.

    The study’s principal investigator, Sumit R Majumdar, associate professor in the department of medicine at the University of Alberta, said, “There is no possible mechanism for reducing deaths from influenza when there’s no influenza circulating.”

    He added, “What we did was exactly the opposite of what researchers have been doing for the past 20 years. They throw away all results during the summer.”

    By examining results from outside the flu season, he and his team were able to show that the difference in mortality most likely represents a “healthy user effect” in which healthier people who tend to watch what they eat, exercise regularly, and don’t smoke are also the people who are most likely to comply with drug and vaccine regimens.

    Their findings support earlier work that found similar “healthy vaccinee” or healthy user effects (International Journal of Epidemiology 2006;35:345-52, doi:10.1093/ije/dyi275). Although some researchers tried to control for this effect by using administrative data on comorbidity and drug use, the current study extended the findings of another research group in Seattle, Washington, which found that such administrative data, generally derived from diagnostic or ICD-9 (international classification of diseases, 9th revision) codes used for billing, did not accurately reflect patients’ actual clinical or functional status.

    When the Alberta researchers used prospectively acquired data to control for socioeconomic status on the basis of neighbourhood census data and functional status (walking independently, consistently needing a walking aid or wheelchair, or bedridden) the adjusted mortality difference was not significant (odds ratio 0.81 (0.35 to 1.85)).

    These findings are consistent with a small benefit or no benefit, said the researchers, who concluded that calls for a policy of more immunisation of elderly people were premature.

    They concluded, “We hope our findings might tilt the balance towards clinical equipoise and permit much needed and adequately powered randomized trials of influenza vaccine in the elderly to take place.”

    However, the study was criticised by Gary Greenberg, a specialist in preventive medicine at the School of Public Health at the University of North Carolina. He said that the study could not properly determine the protective effects of vaccination, as the case-control design excluded people who were protected by the vaccine and never developed pneumonia that required hospitalisation, “leaving them entirely out of the evaluated cohort.”

    Although accepting some criticisms of his study, Dr Majumdar argued that an earlier study (Lancet 2008;372:398-405, doi:10.1016/S0140-6736(08)61160-5) showed that vaccination doesn’t prevent pneumonia in the very population that Dr Greenberg is worried about and that population based studies similarly fail to show reductions in the incidence of pneumonia or in mortality despite dramatic increases in flu vaccination among elderly people.

    Dr Majumdar concluded that policies calling for more vaccination of elderly people are based on the “flimsiest of evidence” and reiterated his call for controlled studies.


    Cite this as: BMJ 2008;337:a1550

    View Abstract

    Sign in

    Log in through your institution

    Free trial

    Register for a free trial to to receive unlimited access to all content on for 14 days.
    Sign up for a free trial