Intended for healthcare professionals

  1. Research
  2. Inconsistency between...
  3. Inconsistency between direct and indirect comparisons of competing interventions: meta-epidemiological study
CCBYNC Open access
Research

Inconsistency between direct and indirect comparisons of competing interventions: meta-epidemiological study

BMJ 2011; 343 doi: https://doi.org/10.1136/bmj.d4909 (Published 16 August 2011) Cite this as: BMJ 2011;343:d4909
  1. Fujian Song, reader in research synthesis,
  2. Tengbin Xiong, research associate12,
  3. Sheetal Parekh-Bhurke, research fellow13,
  4. Yoon K Loke, senior lecturer in clinical pharmacology1,
  5. Alex J Sutton, professor of medical statistics4,
  6. Alison J Eastwood, senior research fellow5,
  7. Richard Holland, professor of public health medicine1,
  8. Yen-Fu Chen, research fellow6,
  9. Anne-Marie Glenny, senior lecturer in evidence based oral care7,
  10. Jonathan J Deeks, professor of biostatistics6,
  11. Doug G Altman, professor and director8
  1. 1Norwich Medical School, Faculty of Medicine and Health Science, University of East Anglia, Norwich NR4 7TJ, UK
  2. 2Department of Oncology, University of Cambridge, Cambridge, UK
  3. 3NIHR Trials and Studies Coordinating Centre, University of Southampton, Southampton, UK
  4. 4Department of Health Science, University of Leicester, Leicester, UK
  5. 5Centre for Reviews and Dissemination, University of York, York, UK
  6. 6Public Health, Epidemiology and Biostatistics, University of Birmingham, Birmingham, UK
  7. 7School of Dentistry, University of Manchester, Manchester, UK
  8. 8Centre for Statistics in Medicine, University of Oxford, Oxford, UK
  1. Correspondence to: F Song fujian.song{at}uea.ac.uk
  • Accepted 1 June 2011

Abstract

Objective To investigate the agreement between direct and indirect comparisons of competing healthcare interventions.

Design Meta-epidemiological study based on sample of meta-analyses of randomised controlled trials.

Data sources Cochrane Database of Systematic Reviews and PubMed.

Inclusion criteria Systematic reviews that provided sufficient data for both direct comparison and independent indirect comparisons of two interventions on the basis of a common comparator and in which the odds ratio could be used as the outcome statistic.

Main outcome measure Inconsistency measured by the difference in the log odds ratio between the direct and indirect methods.

Results The study included 112 independent trial networks (including 1552 trials with 478 775 patients in total) that allowed both direct and indirect comparison of two interventions. Indirect comparison had already been explicitly done in only 13 of the 85 Cochrane reviews included. The inconsistency between the direct and indirect comparison was statistically significant in 16 cases (14%, 95% confidence interval 9% to 22%). The statistically significant inconsistency was associated with fewer trials, subjectively assessed outcomes, and statistically significant effects of treatment in either direct or indirect comparisons. Owing to considerable inconsistency, many (14/39) of the statistically significant effects by direct comparison became non-significant when the direct and indirect estimates were combined.

Conclusions Significant inconsistency between direct and indirect comparisons may be more prevalent than previously observed. Direct and indirect estimates should be combined in mixed treatment comparisons only after adequate assessment of the consistency of the evidence.

Footnotes

  • Contributors: FS had the idea for the study and is the guarantor. FS, YKL, AJS, RH, AJE, A-MG, JJD, and DGA developed the research protocol. SP-B, YKL, and FS searched for and identified relevant studies. SP-B, TX, YKL, AMG, and FS extracted and checked data from the included cases. TX and FS analysed data. AJS, AJE, RH, Y-FC, A-MG, JJD, and DGA provided methodological support and helped to interpret findings. FS drafted the manuscript, and all authors critically commented on it.

  • Funding: The study was funded by the UK Medical Research Council (G0701607). The study design, data collection, and interpretation have not been influenced by the funder.

  • Competing interests: All authors have completed the Unified Competing Interest form at www.icmje.org/coi_disclosure.pdf (available on request from the corresponding author) and declare: all authors had support from the UK Medical Research Council for the submitted work; no financial relationships with any organisations that might have an interest in the submitted work in the previous three years; no other relationships or activities that could appear to have influenced the submitted work.

  • Ethical approval: Not required.

  • Data sharing: No additional data available.

This is an open-access article distributed under the terms of the Creative Commons Attribution Non-commercial License, which permits use, distribution, and reproduction in any medium, provided the original work is properly cited, the use is non commercial and is otherwise in compliance with the license. See: http://creativecommons.org/licenses/by-nc/2.0/ and http://creativecommons.org/licenses/by-nc/2.0/legalcode.

View Full Text
Back to top