BMJ  2006;333:912-915 (28 October), doi:10.1136/bmj.38995.531701.80

Analysis and comment

Public health

Influenza vaccination: policy versus evidence

¹ Cochrane Vaccines Field, Anguillara Sabazia, Roma 00061, Italy jefferson.tom{at}gmail.com

Each year enormous effort goes into producing influenza vaccines for that specific year and delivering them to appropriate sections of the population. Is this effort justified?

Viral infections of the respiratory tract impose a high burden on society. In the last half of the 20th century, efforts to prevent or minimise their impact centred on the use of influenza vaccines. Each year enormous effort goes into producing that year's vaccine and delivering it to appropriate sections of the population. Here, I will discuss policies on the use of inactivated vaccines for seasonal influenza; the evidence for their efficacy, effectiveness, and safety ("effects"); and possible reasons for the gap between policy and evidence.

Policies

Every vaccination campaign has stated aims against which its effects must be measured. The US Advisory Committee on Immunisation Practices produces a regularly updated rationale for vaccination against influenza.¹ The current version identifies 11 categories of patients at high risk of complications from influenza (box).

The rationale rests on the heavy burden that influenza imposes on the population and the benefits of vaccination. For example, reductions in cases, admissions to hospital, mortality of elderly people in families with children, contacts with healthcare professionals, antibiotic prescriptions, and absenteeism for children and household contacts are the main arguments for extending vaccination to healthy children aged 6-23 months in the United States.² Canada introduced a similar policy in 2004.³ Less comprehensive policies recommending vaccination for all people aged 60 or 65 and over are in place in 40 of 51 developed or rapidly developing countries.⁴ On the basis of single studies, the World Health Organization estimates that "vaccination of the elderly reduces the risk of serious complications or of death by 70-85%."⁵ Given the global nature of these recommendations, what type of evidence should we expect to support them and what does available evidence tell us?⁴

Which evidence?

When considering the best evidence for vaccination we must take into account the unique epidemiological features of influenza viruses and the rationale for immunisation. The incidence and circulation of seasonal influenza and other respiratory viruses vary greatly each year, each season, and even in each setting. A systematic review of the incidence of influenza in people up to 19 years' old reported a seasonal variability of 0-46%; during a five year period the average incidence was 4.6% in this age group. During a period of 25 years the incidence was 9.5% in children under 5.⁶ Because of this variability and lack of carryover protection from one year's vaccine to the next,⁷ especially if the virus changes its antigenic configuration, single studies reporting data from one or two seasons are difficult to interpret. Single studies are also not reliable sources for generalising and forecasting the effects of vaccines, especially when numbers are small. They introduce further instability into already problematic forecasting. Additional limitations to our forecasting ability are imposed by our use (and misuse) of studies assessing the effects of influenza vaccines. Although the effect assessed depends on the aims of the particular campaign, most concentrate on serious effects (such as pneumonia or death) and person to person transmission (table 1). Field efficacy studies are only relevant when viral circulation is high, but no one can forecast with precision the impact on next year's influenza.

View this table: [in this window] [in a new window]   Table 1 Effects of inactivated influenza vaccines and preferred designs of primary studies to assess them

 

Flu vaccination: is the effort justified? Credit: SATURN STILLS/SPL

 

Studies of the effects on influenza-like illness and its complications most closely replicate real life conditions because no one knows what agent (if any) causes this disease. Influenza-like illness is an acute respiratory disease caused by many different viruses (including influenza A and B), which presents with symptoms and signs that cannot be distinguished from those of influenza. Influenza-like illness does not have documented laboratory isolation of the causative agent and is the syndrome that most commonly presents to doctors ("the flu").

In general the most powerful and reliable studies are those that "average" out several years and perform subanalyses by setting, population, viral circulation, and viral-vaccine antigenic match—variables that affect interpretation of the effects of a vaccine. Systematic reviews are the best way to perform such analyses, and provide powerful evidence weighted by the methodological quality of the studies involved. Large datasets containing several decades of observations help us to assess the performance of vaccines more accurately.

People for whom vaccination is recommended in the United States1 People aged 65 or more Patients in institutions who have chronic medical conditions Adults and children with chronic disorders of the cardiovascular and respiratory systems (including asthma but excluding hypertension) Adults and children who have been treated in hospital in the preceding 12 months for a range of conditions (for example, diabetes or haemoglobinopathy) Adults and children with conditions that compromise respiratory function or handling of infected secretions Children aged 6 months to 18 years being treated with aspirin Women who are pregnant during the influenza "season" Children aged 6-59 months Adults aged 50-64 years Carers and household contacts (including children) of those in the above risk categories and of children aged 0-59 months Healthcare workers

The evidence

I searched for relevant systematic reviews when updating and expanding the Clinical Evidence chapter on influenza (see bmj.com)—evidence was plentiful. The examples in table 2 show the strength of the evidence and the contradictions in relation to the stated aims of the vaccination campaign. Whenever possible, I chose evidence gathered in the optimal circumstances (for inactivated vaccines)—high viral circulation and a good match between the viral antigen and the vaccine.

View this table: [in this window] [in a new window]   Table 2 Examples of evidence from systematic reviews comparing inactivated influenza vaccines with placebo or no intervention

 

Three problems are immediately apparent. The first is heavy reliance on non-randomised studies (chiefly cohort studies), especially in the elderly. This makes assessment of methodological quality an important part of data interpretation. For example, of 40 datasets assessing the effects of influenza vaccines in elderly people in institutions, only 26 reported data on viral types in circulation and only 21 gave information on vaccine content. Insufficient data were available in 11 of 17 retrospective studies of elderly people in institutions to allow reviewers to assess the authors' claim of "high" or "epidemic" viral circulation.^{11 14} A metaanalysis of inactivated vaccines in elderly people showed a gradient from no effect against influenza or influenza-like illness to a large effect (up to 60%) in preventing all-cause mortality. These findings are both counterintuitive and implausible, as other causes of death are far more prevalent in elderly people even in the winter months.^{15 16} It is impossible for a vaccine that does not prevent influenza to prevent its complications, including admission to hospital.

A more likely explanation for such a finding is selection bias, where one half of the study population (hemi-cohort) systematically differs from the other in one or more key characteristics.^14-16 In this case, the vaccinated hemi-cohort may have been more mobile, healthy, and wealthy than the control hemi-cohort, thus explaining the differences in all-cause mortality.^{11 14} The same effect is seen in stronger study designs (such as cluster randomised trials) that are badly executed, which introduces bias.¹⁰ Its presence seems to be a marker of confounders that persist even after adjusting for known ones, and it makes accurate interpretation of the data difficult. Caution in interpretation should thus be the rule, not the exception. This problem (in the opposite direction—with frailer people more likely to be vaccinated) has been identified before but not heeded.¹⁷ The only way that all known and unknown confounders can be adequately controlled for is by randomisation.

The influence of poor study quality is also seen in the outcome of a review of evidence supporting the vaccination of all children to minimise transmission to family contacts.¹⁸ Five randomised studies and five non-randomised studies were reviewed, but although data were suggestive of protection, its extent was impossible to measure because of the weak methods used in the primary studies.¹⁸

The second problem is either the absence of evidence or the absence of convincing evidence on most of the effects at the centre of campaign objectives (table 2). In children under 2 years inactivated vaccines had the same field efficacy as placebo,⁸ and in healthy people under 65 vaccination did not affect hospital stay, time off work, or death from influenza and its complications.⁹ Reviews found no evidence of an effect in patients with asthma or cystic fibrosis, but inactivated vaccines reduced the incidence of exacerbations after three to four weeks by 39% in those with chronic obstructive pulmonary disease.^{12 13 19} All reviewers reported small data sets (such as 180 people with chronic obstructive pulmonary disease¹³), which may explain the lack of demonstrable effect.

The third problem is the small and heterogeneous dataset on the safety of inactivated vaccines, which is surprising given their longstanding and widespread use. A Cochrane Database Systematic Review found only one old trial with data from 35 participants aged 12-28 months.⁸ In the general population of elderly people, despite a dataset of several million observations, safety was only reported in five randomised controlled trials (2963 observations in total) on local and systemic adverse events seen within a week of giving parenteral inactivated vaccine.¹¹ Although there appears to be no evidence that annual revaccination is harmful, such a lack of knowledge is surprising.

Gap between policy and evidence

The large gap between policy and what the data tell us (when rigorously assembled and evaluated) is surprising. The reasons for this situation are not clear and may be complex. The starting point is the potential confusion between influenza and influenza-like illness, when any case of illness resembling influenza is seen as real influenza, especially during peak periods of activity. Some surveillance systems report cases of influenza-like illness as influenza without further explanation. This confusion leads to a gross overestimation of the impact of influenza, unrealistic expectations of the performance of vaccines, and spurious certainty of our ability to predict viral circulation and impact. The consequences are seen in the impractical advice given by public bodies on thresholds of the incidence of influenza-like illness at which influenza specific interventions (antivirals) should be used.²⁰

The confusion between influenza and influenza-like illness is compounded by the lack of accurate and fast surveillance systems that can tell what viruses are circulating in a setting or community within a short time frame, and after the "season" is finished give an accurate picture of what went on to enable better forecasting of future trends.²¹ Accurate surveillance must be based on a properly worked out sampling system for cases of influenza-like illness that meet set criteria, with accurate and quick feedback of a presumptive microbiological diagnosis. Without this, we cannot generalise from random sampling.

Another reason may be "availability creep." In their efforts to deal with, or be seen to deal with, policy makers favour intervention with what is available—registered influenza vaccines. A similar philosophy is the "we have to make decisions and cannot wait to have perfect data" approach. This attitude may have an altruistic basis but has two important consequences. Firstly, it uses up resources that could be invested in a proper evaluation of influenza vaccines or on other health interventions of proven effectiveness. Secondly, the inception of a vaccination campaign seems to preclude the assessment of a vaccine through placebo controlled randomised trials on ethical grounds. Far from being unethical, however, such trials are desperately needed and we should invest in them without delay. A further consequence is reliance on non-randomised studies once the campaign is under way. It is debatable whether these can contribute to our understanding of the effectiveness of vaccines. Ultimately non-randomised designs cannot answer questions on the effects of influenza vaccines.

Summary points Public policy worldwide recommends the use of inactivated influenza vaccines to prevent seasonal outbreaks Because viral circulation and antigenic match vary each year and non-randomised studies predominate, systematic reviews of large datasets from several decades provide the best information on vaccine performance Evidence from systematic reviews shows that inactivated vaccines have little or no effect on the effects measured Most studies are of poor methodological quality and the impact of confounders is high Little comparative evidence exists on the safety of these vaccines Reasons for the current gap between policy and evidence are unclear, but given the huge resources involved, a re-evaluation should be urgently undertaken

The optimistic and confident tone of some predictions of viral circulation and of the impact of inactivated vaccines, which are at odds with the evidence, is striking. The reasons are probably complex and may involve "a messy blend of truth conflicts and conflicts of interest making it difficult to separate factual disputes from value disputes"²² or a manifestation of optimism bias (an unwarranted belief in the efficacy of interventions).²³

Whatever the reasons, it is a sobering thought that Archie Cochrane's 1972 statement that we should use what has been tested and found to reach its objectives is as revolutionary now as it was then.

---

Details of the search strategy are on bmj.com

Contributor: TJ designed and wrote the paper and is the sole contributor and guarantor.

Competing interests: TJ owned shares in Glaxo SmithKline and received consultancy fees from Sanofi-Synthelabo (2002) and Roche (1997-9).

References

1. Centers for Disease Control and Prevention. Prevention and control of influenza: recommendations of the Advisory Committee on Immunization Practices (ACIP). Morbid Mortal Wkly Rep 2006;55: 1-41.
2. American Academy of Pediatrics Committee on Infectious Diseases. Recommendations for influenza immunization of children. Pediatrics 2004;113: 1441-7.[Abstract/Free Full Text]
3. Orr P. An advisory committee statement (ACS). National Advisory Committee on Immunization (NACI). Statement on influenza vaccination for the 2004-2005 season. Can Commun Dis Rep 2004;30: 1-32.[Medline]
4. Van Essen GA, Palache AM, Forleo E, Fedson DS. Influenza vaccination in 2000: recommendations and vaccine use in 50 developed and rapidly developing countries. Vaccine 2003;21: 1780-5.[CrossRef][Web of Science][Medline]
5. World Health Organization. Influenza vaccines. WHO position paper. Wkly Epidemiol Rec 2002;77: 230-40.[Medline]
6. Bueving HJ, van der Wouden JC, Berger MY, Thomas S. Incidence of influenza and associated illness in children aged 0-19 years: a systematic review. Rev Med Virol 2005;15: 383-91.[CrossRef][Web of Science][Medline]
7. Beyer WE, de Bruijn IA, Palache AM, Westendorp RG, Osterhaus AD. Protection against influenza after annually repeated vaccination: a metaanalysis of serologic and field studies. Arch Intern Med 1999;159: 182-8.[Abstract/Free Full Text]
8. Smith S, Demicheli V, Di Pietrantonj C, Harnden AR, Jefferson T, Matheson NJ, et al. Vaccines for preventing influenza in healthy children. Cochrane Database Syst Rev 2006;(1):CD004879.
9. Demicheli V, Rivetti D, Deeks JJ, Jefferson TO. Vaccines for preventing influenza in healthy adults. Cochrane Database Syst Rev 2004;(3):CD001269.
10. Thomas RE, Jefferson T, Demicheli V, Rivetti D. Influenza vaccination for healthcare workers who work with the elderly. Cochrane Database Syst Rev 2006;(3):CD005187.
11. Rivetti D, Demicheli V, Di Pietrantonj C, Jefferson TO, Thomas R. Vaccines for preventing influenza in the elderly. Cochrane Database Syst Rev 2006;(3):CD004876.
12. Cates CJ, Jefferson TO, Bara AL, Rowe BH. Vaccines for preventing influenza in people with asthma. Cochrane Database Syst Rev 2003;(4):CD000364.
13. Poole PJ, Chacko E, Wood-Baker RWB, Cates CJ. Influenza vaccine for patients with chronic obstructive pulmonary disease. Cochrane Database Syst Rev 2006;(1):CD002733.
14. Jefferson T, Rivetti D, Rivetti A, Rudin M, Di Pietrantonj C, Demicheli V. Efficacy and effectiveness of influenza vaccines in elderly people: a systematic review. Lancet 2005;366: 1165-74.[CrossRef][Web of Science][Medline]
15. Simonsen L, Reichert TA, Viboud C, Blackwelder WC, Taylor RJ, Miller MA. Impact of influenza vaccination on seasonal mortality in the US elderly population. Arch Intern Med 2005;165: 265-72.[Abstract/Free Full Text]
16. Simonsen L, Viboud C, Taylor R. Influenza vaccination in elderly people. Lancet 2005;366: 2086.[Web of Science][Medline]
17. Hak E, Verheij TJ, Grobbee DE, Nichol KL, Hoes AW. Confounding by indication in non-experimental evaluation of vaccine effectiveness: the example of prevention of influenza complications. J Epidemiol Community Health 2002;56: 951-5.[Abstract/Free Full Text]
18. Jordan R, Connock M, Albon E, Fry-Smith A, Olowokure B, Hawker J, et al. Universal vaccination of children against influenza: are there indirect benefits to the community? A systematic review of the evidence. Vaccine 2006;24: 1047-62.[CrossRef][Web of Science][Medline]
19. Bhalla P, Tan A, Smyth R. Vaccines for preventing influenza in people with cystic fibrosis. Cochrane Database Syst Rev 2000;(1):CD001753.
20. Harling R, Hayward A, Watson JM. Implications of the incidence of influenza-like illness in nursing homes for influenza chemoprophylaxis: descriptive study [see comments]. BMJ 2004;329: 663-4.[Free Full Text]
21. Carman WF, Wallace LA, Walker J. Rapid virological surveillance of community influenza infection in general practice. BMJ 2000;321: 736-7.[Free Full Text]
22. MacCoun RJ. Biases in the interpretation and use of research results. Annu Rev Psychol 1998;49: 259-87.[CrossRef][Web of Science][Medline]
23. Chalmers I, Matthews R. What are the implications of optimism bias in clinical research? Lancet 2006;367: 449-50.[CrossRef][Web of Science][Medline]

(Accepted 21 September 2006)

CiteULike    Complore    Connotea    Del.icio.us    Digg    Facebook    Reddit    StumbleUpon    Technorati    Twitter    What's this?

Relevant Articles

Relation of study quality, concordance, take home message, funding, and impact in studies of influenza vaccines: systematic review

T Jefferson, C Di Pietrantonj, M G Debalini, A Rivetti, and V Demicheli
BMJ 2009 338: b354. [Abstract] [Full Text] [PDF]

Influenza in elderly people in care homes

Rachel E Jordan and Jeremy I Hawker
BMJ 2006 333: 1229-1230. [Extract] [Full Text] [PDF]

Influenza vaccination: policy versus evidence: No gap between policy and evidence

David S Fedson and Kristin L Nichol
BMJ 2006 333: 1020. [Extract] [Full Text] [PDF]

Influenza vaccination: policy versus evidence: Protection from disease versus disease severity

Christian W Mandl
BMJ 2006 333: 1020. [Extract] [Full Text] [PDF]

Influenza vaccination: policy versus evidence: Policy is in the lead

Peter Doshi
BMJ 2006 333: 1020-1021. [Extract] [Full Text] [PDF]

Influenza vaccination: policy versus evidence: Vested interests will always trump evidence

G H Hall
BMJ 2006 333: 1021. [Extract] [Full Text] [PDF]

In brief
BMJ 2006 333: 934. [Extract] [Full Text] [PDF]

Of measles and flu

Fiona Godlee
BMJ 2006 333: 0. [Extract] [Full Text] [PDF]

Implications of the incidence of influenza-like illness in nursing homes for influenza chemoprophylaxis: descriptive study

Richard Harling, Andrew Hayward, and John M Watson
BMJ 2004 329: 663-664. [Extract] [Full Text] [PDF]

Rapid virological surveillance of community influenza infection in general practice

William F Carman, Lesley A Wallace, Jacqueline Walker, Sheena McIntyre, Ahilya Noone, Peter Christie, James Millar, and James D Douglas
BMJ 2000 321: 736-737. [Full Text] [PDF]

This article has been cited by other articles:

• Winchester, C. C., Macfarlane, T. V., Thomas, M., Price, D. (2009). Antibiotic Prescribing and Outcomes of Lower Respiratory Tract Infection in UK Primary Care. Chest 135: 1163-1172 [Abstract] [Full text]  
• Jefferson, T, Di Pietrantonj, C, Debalini, M G, Rivetti, A, Demicheli, V (2009). Relation of study quality, concordance, take home message, funding, and impact in studies of influenza vaccines: systematic review. BMJ 338: b354-b354 [Abstract] [Full text]  
• Nichol, K. L., D'Heilly, S., Ehlinger, E. P. (2008). Influenza Vaccination Among College and University Students: Impact on Influenzalike Illness, Health Care Use, and Impaired School Performance. Arch Pediatr Adolesc Med 162: 1113-1118 [Abstract] [Full text]  
• Doshi, P. (2008). Trends in Recorded Influenza Mortality: United States, 1900-2004. AJPH 98: 939-945 [Abstract] [Full text]  
• Nichol, K. L., Nordin, J. D., Nelson, D. B., Mullooly, J. P., Hak, E. (2007). Effectiveness of Influenza Vaccine in the Community-Dwelling Elderly. NEJM 357: 1373-1381 [Abstract] [Full text]  
• Borrill, Z., Woodhead, M. (2007). Universal influenza vaccination in the elderly: another piece in the jigsaw. Eur Respir J 30: 407-408 [Full text]  
• Jordan, R. E, Hawker, J. I (2006). Influenza in elderly people in care homes. BMJ 333: 1229-1230 [Full text]  
• Fukuda, K., Kieny, M. P. (2006). Different Approaches to Influenza Vaccination. NEJM 355: 2586-2587 [Full text]  
• Fedson, D. S, Nichol, K. L (2006). Influenza vaccination: policy versus evidence: No gap between policy and evidence. BMJ 333: 1020-1020 [Full text]  
• Hall, G H (2006). Influenza vaccination: policy versus evidence: Vested interests will always trump evidence. BMJ 333: 1021-1021 [Full text]  
• Mandl, C. W (2006). Influenza vaccination: policy versus evidence: Protection from disease versus disease severity. BMJ 333: 1020-1020 [Full text]  
• Doshi, P. (2006). Influenza vaccination: policy versus evidence: Policy is in the lead. BMJ 333: 1020-1021 [Full text]  
• (2006). In brief. BMJ 333: 934-934 [Full text]  

Rapid Responses:

Read all Rapid Responses

Emperor's clothes exposed

Nick Hardwick
bmj.com, 28 Oct 2006 [Full text]

Vested interests will always trump evidence

GH Hall
bmj.com, 28 Oct 2006 [Full text]

Policy versus evidence: policy in the lead

Peter Doshi
bmj.com, 29 Oct 2006 [Full text]

Influenza vaccination: what evidence can we rely on?

Stuart J cornell
bmj.com, 29 Oct 2006 [Full text]

Protection from disease versus disease severity

Christian W Mandl
bmj.com, 30 Oct 2006 [Full text]

No gap between policy and evidence for influenza vaccination

David S. Fedson, et al.
bmj.com, 31 Oct 2006 [Full text]

Some published UK data

John Stone
bmj.com, 2 Nov 2006 [Full text]

Flu vaccination: handle with care

Richard M Dawood
bmj.com, 3 Nov 2006 [Full text]

influenza and fatal heart disease

Paul D Griffiths
bmj.com, 4 Nov 2006 [Full text]

Is the death rate from influenza declining in the elderly?

John J Cannell
bmj.com, 6 Nov 2006 [Full text]

Can influenza vaccine overcome vitamin D deficiency?

John J Cannell
bmj.com, 8 Nov 2006 [Full text]

Influenza vaccination: the case for a gap in the evidence is flawed

Punam Mangtani, et al.
bmj.com, 7 Nov 2006 [Full text]

Re: Influenza vaccination: the case for a gap in the evidence is (NOT)flawed

GH Hall
bmj.com, 8 Nov 2006 [Full text]

Flu vaccine production capacity

Dr Charlie Easmon
bmj.com, 12 Nov 2006 [Full text]

Re: Flu vaccine production capacity

John Heptonstall
bmj.com, 15 Nov 2006 [Full text]

Evidence hunters versus public health workers

Angus Nicoll, et al.
bmj.com, 13 Nov 2006 [Full text]

Side effects of mercury containing vaccines like influenza

Joachim Mutter
bmj.com, 22 Nov 2006 [Full text]

Author’s response

Tom Jefferson
bmj.com, 20 Nov 2006 [Full text]

Policy vs Evidence: A Case Study

Alan R Haycox, et al.
bmj.com, 21 Nov 2006 [Full text]

Re: Side effects of mercury containing vaccines like influenza

John Stone
bmj.com, 23 Nov 2006 [Full text]

Call to order: assessing the evidence

Jochen Mau
bmj.com, 23 Nov 2006 [Full text]

Influenza vaccine uptake among British Muslims attending Hajj-2005 and 2006.

Shuja Shafi, et al.
bmj.com, 29 Nov 2006 [Full text]

A provocation that makes reflection

Luisella Grandori
bmj.com, 6 Dec 2006 [Full text]

Influenza seasonal vaccination: is it enough?

Sergio Mariotti
bmj.com, 13 Dec 2006 [Full text]

Blogs

Podcasts

Online poll

Find out more