Inconsistency between direct and indirect comparisons of competing interventions: meta-epidemiological study

doi:10.1136/bmj.d4909

From trainee to consultant, BMJ Group offers doctors around the world tailored information, special events, learning resources and recruitment services at every step along their career path.

... by doctors, for doctors, for patients
We are open for entries!

Online learning

BMJ Learning

High-quality CME / CPD for doctors and other healthcare professionals. BMJ Learning features hundreds of accredited, peer-reviewed learning modules in text, video, and audio formats. Find out more

Courses and Qualifications

BMJ Masterclasses

BMJ Masterclasses, led by experts, help clinicians to use the latest evidence and recent guidelines in practice and meet their CPD/CME requirements. Find out more

Exam Preparation

The leading provider of online exam preparation, helping over 167,000 healthcare professionals to pass their exams. Find out more

onExamination

Search all BMJ research articles:

From

Limit by

Our online table of contents is updated at least twice each day. Read all articles published in the last 7 days.
You can use bmj.com to help you with your continuing medical education. Find out about CME/CPD credits for BMJ articles
Keep up to date with cardiology: Access the latest cardiovascular medicine resources from across BMJ Group.
OPEN ACCESS: All research articles are freely available online, with no word limit. Find out more about the BMJ's open access policy. Submit your paper.
Find out how study types differ in our How to read a paper section.
Countdown to London 2012: BMJ Group's Olympics portal highlights latest Olympics and sports medicine-themed research, comment and learning

Fujian Song, reader in research synthesis,
Tengbin Xiong, research associate1 2,
Sheetal Parekh-Bhurke, research fellow1 3,
Yoon K Loke, senior lecturer in clinical pharmacology1,
Alex J Sutton, professor of medical statistics4,
Alison J Eastwood, senior research fellow5,
Richard Holland, professor of public health medicine1,
Yen-Fu Chen, research fellow6,
Anne-Marie Glenny, senior lecturer in evidence based oral care7,
Jonathan J Deeks, professor of biostatistics6,
Doug G Altman, professor and director8

¹Norwich Medical School, Faculty of Medicine and Health Science, University of East Anglia, Norwich NR4 7TJ, UK
²Department of Oncology, University of Cambridge, Cambridge, UK
³NIHR Trials and Studies Coordinating Centre, University of Southampton, Southampton, UK
⁴Department of Health Science, University of Leicester, Leicester, UK
⁵Centre for Reviews and Dissemination, University of York, York, UK
⁶Public Health, Epidemiology and Biostatistics, University of Birmingham, Birmingham, UK
⁷School of Dentistry, University of Manchester, Manchester, UK
⁸Centre for Statistics in Medicine, University of Oxford, Oxford, UK

Correspondence to: F Song fujian.song{at}uea.ac.uk

Accepted 1 June 2011

Abstract

Objective To investigate the agreement between direct and indirect comparisons of competing healthcare interventions.

Design Meta-epidemiological study based on sample of meta-analyses of randomised controlled trials.

Data sources Cochrane Database of Systematic Reviews and PubMed.

Inclusion criteria Systematic reviews that provided sufficient data for both direct comparison and independent indirect comparisons of two interventions on the basis of a common comparator and in which the odds ratio could be used as the outcome statistic.

Main outcome measure Inconsistency measured by the difference in the log odds ratio between the direct and indirect methods.

Results The study included 112 independent trial networks (including 1552 trials with 478 775 patients in total) that allowed both direct and indirect comparison of two interventions. Indirect comparison had already been explicitly done in only 13 of the 85 Cochrane reviews included. The inconsistency between the direct and indirect comparison was statistically significant in 16 cases (14%, 95% confidence interval 9% to 22%). The statistically significant inconsistency was associated with fewer trials, subjectively assessed outcomes, and statistically significant effects of treatment in either direct or indirect comparisons. Owing to considerable inconsistency, many (14/39) of the statistically significant effects by direct comparison became non-significant when the direct and indirect estimates were combined.

Conclusions Significant inconsistency between direct and indirect comparisons may be more prevalent than previously observed. Direct and indirect estimates should be combined in mixed treatment comparisons only after adequate assessment of the consistency of the evidence.

Footnotes

Contributors: FS had the idea for the study and is the guarantor. FS, YKL, AJS, RH, AJE, A-MG, JJD, and DGA developed the research protocol. SP-B, YKL, and FS searched for and identified relevant studies. SP-B, TX, YKL, AMG, and FS extracted and checked data from the included cases. TX and FS analysed data. AJS, AJE, RH, Y-FC, A-MG, JJD, and DGA provided methodological support and helped to interpret findings. FS drafted the manuscript, and all authors critically commented on it.
Funding: The study was funded by the UK Medical Research Council (G0701607). The study design, data collection, and interpretation have not been influenced by the funder.
Competing interests: All authors have completed the Unified Competing Interest form at www.icmje.org/coi_disclosure.pdf (available on request from the corresponding author) and declare: all authors had support from the UK Medical Research Council for the submitted work; no financial relationships with any organisations that might have an interest in the submitted work in the previous three years; no other relationships or activities that could appear to have influenced the submitted work.
Ethical approval: Not required.
Data sharing: No additional data available.

This is an open-access article distributed under the terms of the Creative Commons Attribution Non-commercial License, which permits use, distribution, and reproduction in any medium, provided the original work is properly cited, the use is non commercial and is otherwise in compliance with the license. See: http://creativecommons.org/licenses/by-nc/2.0/ and http://creativecommons.org/licenses/by-nc/2.0/legalcode.