Empirical evidence of bias in treatment effect estimates in controlled trials with different interventions and outcomes: meta-epidemiological study
BMJ 2008; 336 doi: https://doi.org/10.1136/bmj.39465.451748.AD (Published 13 March 2008) Cite this as: BMJ 2008;336:601
- Lesley Wood, research student1,
- Matthias Egger, head of department and professor of epidemiology and public health2,
- Lise Lotte Gluud, senior registrar3,
- Kenneth F Schulz, vice president of quantitative sciences and clinical professor4,
- Peter Jüni, head of division and reader in clinical epidemiology2,
- Douglas G Altman, director and professor of statistics in medicine5,
- Christian Gluud, head of department3,
- Richard M Martin, reader in clinical epidemiology1,
- Anthony J G Wood, research assistant1,
- Jonathan A C Sterne, professor of medical statistics and epidemiology1
- 1Department of Social Medicine, University of Bristol, Bristol BS8 2PR
- 2Institute of Social and Preventive Medicine (ISPM), University of Bern, Switzerland
- 3Copenhagen Trial Unit, Centre for Clinical Intervention Research, Rigshospitalet, Copenhagen University Hospital, Copenhagen, Denmark
- 4Family Health International and Department of Obstetrics and Gynaecology, School of Medicine, University of North Carolina at Chapel Hill, NC, USA
- 5Centre for Statistics in Medicine, Oxford, UK
- Correspondence to: J A C Sterne jonathan.sterne{at}bristol.ac.uk
- Accepted 19 December 2007
Abstract
Objective To examine whether the association of inadequate or unclear allocation concealment and lack of blinding with biased estimates of intervention effects varies with the nature of the intervention or outcome.
Design Combined analysis of data from three meta-epidemiological studies based on collections of meta-analyses.
Data sources 146 meta-analyses including 1346 trials examining a wide range of interventions and outcomes.
Main outcome measures Ratios of odds ratios quantifying the degree of bias associated with inadequate or unclear allocation concealment, and lack of blinding, for trials with different types of intervention and outcome. A ratio of odds ratios <1 implies that inadequately concealed or non-blinded trials exaggerate intervention effect estimates.
Results In trials with subjective outcomes effect estimates were exaggerated when there was inadequate or unclear allocation concealment (ratio of odds ratios 0.69 (95% CI 0.59 to 0.82)) or lack of blinding (0.75 (0.61 to 0.93)). In contrast, there was little evidence of bias in trials with objective outcomes: ratios of odds ratios 0.91 (0.80 to 1.03) for inadequate or unclear allocation concealment and 1.01 (0.92 to 1.10) for lack of blinding. There was little evidence for a difference between trials of drug and non-drug interventions. Except for trials with all cause mortality as the outcome, the magnitude of bias varied between meta-analyses.
Conclusions The average bias associated with defects in the conduct of randomised trials varies with the type of outcome. Systematic reviewers should routinely assess the risk of bias in the results of trials, and should report meta-analyses restricted to trials at low risk of bias either as the primary analysis or in conjunction with less restrictive analyses.
Footnotes
We thank all authors and funders of the meta-epidemiological studies that contributed data to the present study.
Contributors: LW performed data extraction, data checking, statistical analyses, and writing the first draft of the paper and contributed to the final draft of the paper. ME was responsible for study conception, contributions to study design, supervision of research, drafting the paper, and contributions to the final draft and was author of a contributing study. LLG, KFS, PJ, DGA, and CG were authors of contributing studies, contributed to the study design, performed data checking, and contributed to the final draft of the paper. RMM supervised the research and contributed to the final draft of the paper. AJGW performed data extraction and data checking and contributed to the final draft of the paper. JACS: Supervision of research, contributions to study design, author of contributing study, drafting the paper, contributions to the final draft of the paper. LW and JACS had full access to all of the study data and are guarantors for the study.
Funding: LW was funded by a PhD studentship from the UK Medical Research Council Health Services Research Collaboration. The MRC had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; or preparation, review, or approval of the manuscript.
Competing interests: None declared.