Analysis

Uncertainty in heterogeneity estimates in meta-analyses

BMJ 2007; 335 doi: 10.1136/bmj.39343.408449.80 (Published 1 November 2007)
Cite this as: BMJ 2007;335:914

Access to the full text of this article requires a subscription or payment. Please log in or subscribe below.

  1. John P A Ioannidis, professor,
  2. Nikolaos A Patsopoulos, research associate,
  3. Evangelos Evangelou, research associate
  1. Clinical Trials and Evidence-Based Medicine Unit, Department of Hygiene and Epidemiology, University of Ioannina School of Medicine, Ioannina 45110, Greece
  1. Correspondence to: J P A Ioannidis jioannid{at}cc.uoi.gr
  • Accepted 29 August 2007

John Ioannidis, Nikolaos Patsopoulos, and Evangelos Evangelou argue that, although meta-analyses often measure heterogeneity between studies, these estimates can have large uncertainty, which must be taken into account when interpreting evidence

Summary points

  • The extent of between study heterogeneity should be measured when interpreting results of meta-analyses

  • Meta-analyses rarely document uncertainty in estimates of heterogeneity

  • Our evaluation of a large number of meta-analyses shows a wide range of uncertainty about the extent of heterogeneity in most

  • Confidence intervals of I2 should be calculated and considered when interpreting meta-analyses

An important aim of systematic reviews and meta-analyses is to assess the extent to which different studies give similar or dissimilar results.1 Clinical, methodological, and biological heterogeneity are often topic specific, but statistical heterogeneity can be examined with the same methods in all meta-analyses. Therefore, the perception of statistical heterogeneity or homogeneity often influences meta-analysts and clinicians in important decisions. These decisions include whether the data are similar enough to combine different studies; whether a treatment is applicable to all or should be “individualised” because of variable benefits or harms in different types of patients; and whether a risk factor affects all people exposed or only select populations. How uncertain is the extent of statistical heterogeneity in meta-analyses? Moreover, is this uncertainty properly factored in when interpreting the results?

Evaluating heterogeneity between studies

Many statistical tests are available for evaluating heterogeneity between studies.2 3 Until recently, the most popular was Cochran's Q, a statistic based on the χ2 test.4 Cochran's Q usually has only low power to detect heterogeneity, however. It also depends on the number of studies and cannot be compared across different meta-analyses.2 3 Higgins and colleagues, in two highly cited papers,5 6 proposed the routine use of the I2 statistic. I2 is calculated as …

Access to the full text of this article requires a subscription or payment

Subscribe

Article access

Article access for 1 day

Purchase this article for £20 $30 €32*

The PDF version can be downloaded as your personal record

* Prices do not include VAT

Rapid responses

THIS WEEK'S POLL

BMJ most popular