Intended for healthcare professionals


Methodological problems in the use of indirect comparisons for evaluating healthcare interventions: survey of published systematic reviews

BMJ 2009; 338 doi: (Published 03 April 2009) Cite this as: BMJ 2009;338:b1147
  1. Fujian Song, reader in research synthesis1,
  2. Yoon K Loke, senior lecturer in clinical pharmacology1,
  3. Tanya Walsh, lecturer in dental statistics2,
  4. Anne-Marie Glenny, lecturer in evidence based oral care2,
  5. Alison J Eastwood, senior research fellow3,
  6. Douglas G Altman, professor and director4
  1. 1Faculty of Health, University of East Anglia, Norwich NR4 7TJ
  2. 2School of Dentistry, University of Manchester, Manchester
  3. 3Centre for Reviews and Dissemination, University of York, York
  4. 4Centre for Statistics in Medicine, Oxford
  1. Correspondence to: F Song{at}
  • Accepted 10 November 2008


Objective To investigate basic assumptions and other methodological problems in the application of indirect comparison in systematic reviews of competing healthcare interventions.

Design Survey of published systematic reviews.

Inclusion criteria Systematic reviews published between 2000 and 2007 in which an indirect approach had been explicitly used.

Data extraction Identified reviews were assessed for comprehensiveness of the literature search, method for indirect comparison, and whether assumptions about similarity and consistency were explicitly mentioned.

Results The survey included 88 review reports. In 13 reviews, indirect comparison was informal. Results from different trials were naively compared without using a common control in six reviews. Adjusted indirect comparison was usually done using classic frequentist methods (n=49) or more complex methods (n=18). The key assumption of trial similarity was explicitly mentioned in only 40 of the 88 reviews. The consistency assumption was not explicit in most cases where direct and indirect evidence were compared or combined (18/30). Evidence from head to head comparison trials was not systematically searched for or not included in nine cases.

Conclusions Identified methodological problems were an unclear understanding of underlying assumptions, inappropriate search and selection of relevant trials, use of inappropriate or flawed methods, lack of objective and validated methods to assess or improve trial similarity, and inadequate comparison or inappropriate combination of direct and indirect evidence. Adequate understanding of basic assumptions underlying indirect and mixed treatment comparison is crucial to resolve these methodological problems.

Appendix 1 PubMed search strategy

Appendix 2 Characteristics of identified reports

Appendix 3 Identified studies

References of included studies


  • Contributors: FS conceived the idea for the study and is guarantor. FS and YKL carried out the literature search and identified relevant systematic reviews. FS extracted data from included reviews, and YKL, TW, and A-MG checked data extracted. FS analysed data and prepared a first draft, which was substantially revised according to comments and suggestions from AJE, DGA, TW, A-MG, and YKL.

  • Funding: No specific funding was received for this study.

  • Competing interests: None declared.

  • Ethical approval: Not required.

This is an open-access article distributed under the terms of the Creative Commons Attribution Non-commercial License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

View Full Text