Analysis And Comment Evidence based medicine

The case of the misleading funnel plot

BMJ 2006; 333 doi: http://dx.doi.org/10.1136/bmj.333.7568.597 (Published 14 September 2006) Cite this as: BMJ 2006;333:597
  1. Joseph Lau (jlau1@tufts-nemc.org), professor1,
  2. John P A Ioannidis, professor2,
  3. Norma Terrin, associate professor1,
  4. Christopher H Schmid, professor of medicine1,
  5. Ingram Olkin, professor3
  1. 1 Institute for Clinical Research and Health Policy Studies, Tufts-New England Medical Center, Boston, MA 02111, USA,
  2. 2Department of Hygiene and Epidemiology, University of Ioannina School of Medicine, Ioannina, Greece,
  3. 3Statistics Department, Stanford University, Palo Alto, USA
  1. Correspondence to: J Lau
  • Accepted 16 June 2006

Evidence based medicine insists on rigorous standards to appraise clinical interventions. Failure to apply the same rules to its own tools could be equally damaging

The advent of evidence based medicine has generated considerable interest in developing and applying methods that can improve the appraisal and synthesis of data from diverse studies. Some methods have become an integral part of systematic reviews and meta-analyses, with reviewers, editors, instructional handbooks, and guidelines encouraging their routine inclusion. However, the evidence for using these methods is sometimes lacking, as the reliance on funnel plots shows.

What is a funnel plot?

The funnel plot is a scatter plot of the component studies in a meta-analysis, with the treatment effect on the horizontal axis and some measure of weight, such as the inverse variance, the standard error, or the sample size, on the vertical axis. Light and Pillemer proposed in 1984: “If all studies come from a single underlying population, this graph should look like a funnel, with the effect sizes homing in on the true underlying value as n increases. [If there is publication bias] there should be a bite out of the funnel.”1 Many meta-analyses show funnel plots or perform various tests that examine whether there is asymmetry in the funnel plot and directly interpret the results as showing evidence for or against the presence of publication bias.

The plot's wide popularity followed an article published in the BMJ in 1997.2 That pivotal article has already received over 800 citations (as of December 2005) in the Web of Science. With two exceptions, this is more citations than for any other paper published by the BMJ in the past decade. The authors were careful to state many reasons why funnel plot asymmetry may not necessarily reflect publication bias. However, apparently many readers did not go beyond the …

Sign in

Subscribe