BMJ, doi: 10.1136/bmj.39465.451748.AD, (Published 3 March 2008)

Research

Empirical evidence of bias in treatment effect estimates in controlled trials with different interventions and outcomes: meta-epidemiological study

¹ Department of Social Medicine, University of Bristol, Bristol BS8 2PR, ² Institute of Social and Preventive Medicine (ISPM), University of Bern, Switzerland, ³ Copenhagen Trial Unit, Centre for Clinical Intervention Research, Rigshospitalet, Copenhagen University Hospital, Copenhagen, Denmark, ⁴ Family Health International and Department of Obstetrics and Gynaecology, School of Medicine, University of North Carolina at Chapel Hill, NC, USA, ⁵ Centre for Statistics in Medicine, Oxford, UK

Objective To examine whether the association of inadequate or unclear allocation concealment and lack of blinding with biased estimates of intervention effects varies with the nature of the intervention or outcome.

Design Combined analysis of data from three meta-epidemiological studies based on collections of meta-analyses.

Data sources 146 meta-analyses including 1346 trials examining a wide range of interventions and outcomes.

Main outcome measures Ratios of odds ratios quantifying the degree of bias associated with inadequate or unclear allocation concealment, and lack of blinding, for trials with different types of intervention and outcome. A ratio of odds ratios <1 implies that inadequately concealed or non-blinded trials exaggerate intervention effect estimates.

Results In trials with subjective outcomes effect estimates were exaggerated when there was inadequate or unclear allocation concealment (ratio of odds ratios 0.69 (95% CI 0.59 to 0.82)) or lack of blinding (0.75 (0.61 to 0.93)). In contrast, there was little evidence of bias in trials with objective outcomes: ratios of odds ratios 0.91 (0.80 to 1.03) for inadequate or unclear allocation concealment and 1.01 (0.92 to 1.10) for lack of blinding. There was little evidence for a difference between trials of drug and non-drug interventions. Except for trials with all cause mortality as the outcome, the magnitude of bias varied between meta-analyses.

Conclusions The average bias associated with defects in the conduct of randomised trials varies with the type of outcome. Systematic reviewers should routinely assess the risk of bias in the results of trials, and should report meta-analyses restricted to trials at low risk of bias either as the primary analysis or in conjunction with less restrictive analyses.

CiteULike    Complore    Connotea    Del.icio.us    Digg    Reddit    StumbleUpon    Technorati    What's this?

Relevant Articles

Risk of bias versus quality assessment of randomised controlled trials: cross sectional study

Lisa Hartling, Maria Ospina, Yuanyuan Liang, Donna M Dryden, Nicola Hooton, Jennifer Krebs Seida, and Terry P Klassen
BMJ 2009 339: b4012. [Abstract] [Full Text] [PDF]

The effects of excluding patients from the analysis in randomised controlled trials: meta-epidemiological study

Eveline Nüesch, Sven Trelle, Stephan Reichenbach, Anne W S Rutjes, Elizabeth Bürgi, Martin Scherer, Douglas G Altman, and Peter Jüni
BMJ 2009 339: b3244. [Abstract] [Full Text] [PDF]

Novel methods to deal with publication biases: secondary analysis of antidepressant trials in the FDA trial registry database and related journal publications

Santiago G Moreno, Alex J Sutton, Erick H Turner, Keith R Abrams, Nicola J Cooper, Tom M Palmer, and A E Ades
BMJ 2009 339: b2981. [Abstract] [Full Text] [PDF]

Comparison of descriptions of allocation concealment in trial protocols and the published reports: cohort study

Julie Pildal, An-Wen Chan, Asbjørn Hróbjartsson, Elisabeth Forfang, Douglas G Altman, and Peter C Gøtzsche
BMJ 2005 330: 1049. [Abstract] [Full Text] [PDF]

Assessment of methodological quality of primary studies by systematic reviews: results of the metaquality cross sectional study

Lorenzo P Moja, Elena Telaro, Roberto D'Amico, Ivan Moschetti, Laura Coe, Alessandro Liberati on behalf of the Metaquality Study Group
BMJ 2005 330: 1053. [Abstract] [Full Text] [PDF]

Bad reporting does not mean bad methods for randomised trials: observational study of randomised controlled trials performed by the Radiation Therapy Oncology Group

Heloisa P Soares, Stephanie Daniels, Ambuj Kumar, Mike Clarke, Charles Scott, Suzanne Swann, and Benjamin Djulbegovic
BMJ 2004 328: 22-24. [Abstract] [Full Text] [PDF]

Meta-analysis and the meta-epidemiology of clinical research

C David Naylor
BMJ 1997 315: 617-619. [Extract] [Full Text]

This article has been cited by other articles:

• O'Connell, N. E., Wand, B. M., Goldacre, B. (2009). Interpretive Bias in Acupuncture Research?: A Case Study. Eval Health Prof 32: 393-409 [Abstract]  
• Thomson, H., Thomas, S., Sellstrom, E., Petticrew, M. (2009). The Health Impacts of Housing Improvement: A Systematic Review of Intervention Studies From 1887 to 2007. AJPH 99: S681-S692 [Abstract] [Full text]  
• Bausch, B., Spaar, A., Kleijnen, J., Puhan, M. A. (2009). Quality of randomised trials in COPD. Eur Respir J 34: 1060-1065 [Abstract] [Full text]  
• McCauley, R. J., Strand, E., Lof, G. L., Schooling, T., Frymark, T. (2009). Evidence-Based Systematic Review: Effects of Nonspeech Oral Motor Exercises on Speech. AJSLP 18: 343-360 [Abstract] [Full text]  
• Wilner, A. N., French FAAN, J., Gronseth FAAN, G. S. (2009). Lost in a jungle of evidence: We need a compass; Practice parameters and technology assessments: What they are, what they are not, and why you should care. Neurology 73: 1337-1338 [Full text]  
• Hartling, L., Ospina, M., Liang, Y., Dryden, D. M, Hooton, N., Krebs Seida, J., Klassen, T. P (2009). Risk of bias versus quality assessment of randomised controlled trials: cross sectional study. BMJ 339: b4012-b4012 [Abstract] [Full text]  
• Ioannidis, J. P.A. (2009). Integration of evidence from multiple meta-analyses: a primer on umbrella reviews, treatment networks and multiple treatments meta-analyses. CMAJ 181: 488-493 [Full text]  
• Peters, K. L., Rosychuk, R. J., Hendson, L., Cote, J. J., McPherson, C., Tyebkhan, J. M. (2009). Improvement of Short- and Long-Term Outcomes for Very Low Birth Weight Infants: Edmonton NIDCAP Trial. Pediatrics 124: 1009-1020 [Abstract] [Full text]  
• Nuesch, E., Trelle, S., Reichenbach, S., Rutjes, A. W S, Burgi, E., Scherer, M., Altman, D. G, Juni, P. (2009). The effects of excluding patients from the analysis in randomised controlled trials: meta-epidemiological study. BMJ 339: b3244-b3244 [Abstract] [Full text]  
• Moreno, S. G, Sutton, A. J, Turner, E. H, Abrams, K. R, Cooper, N. J, Palmer, T. M, Ades, A E (2009). Novel methods to deal with publication biases: secondary analysis of antidepressant trials in the FDA trial registry database and related journal publications. BMJ 339: b2981-b2981 [Abstract] [Full text]  
• Jespersen, C. M., Kolmos, H. J., Frydendall, N., Hilden, J., Gluud, C., Hansen, J. F., on behalf of the CLARICOR Trial Group, (2009). Compliance with and short-term adverse events from clarithromycin versus placebo in patients with stable coronary heart disease: the CLARICOR trial. J Antimicrob Chemother 64: 411-415 [Abstract] [Full text]  
• Monami, M., Marchionni, N., Mannucci, E. (2009). Glucagon-like peptide-1 receptor agonists in type 2 diabetes: a meta-analysis of randomized clinical trials. Eur J Endocrinol 160: 909-917 [Abstract] [Full text]  
• Howick, J., Glasziou, P., Aronson, J. K (2009). The evolution of evidence hierarchies: what can Bradford Hill's 'guidelines for causation' contribute?. JRSM 102: 186-194 [Full text]  
• Assiri, A. M., Alasmari, F. A., Zimmerman, V. A., Baddour, L. M., Erwin, P. J., Tleyjeh, I. M. (2009). Corticosteroid Administration and Outcome of Adolescents and Adults With Acute Bacterial Meningitis: A Meta-analysis. Mayo Clin Proc. 84: 403-409 [Abstract] [Full text]  
• Machado, L. A. C., Kamper, S. J., Herbert, R. D., Maher, C. G., McAuley, J. H. (2009). Analgesic effects of treatments for non-specific low back pain: a meta-analysis of placebo-controlled randomized trials. Rheumatology (Oxford) 48: 520-527 [Abstract] [Full text]  
• Leff, H. S., Chow, C. M., Pepin, R., Conley, J., Allen, I. E., Seaman, C. A. (2009). Does One Size Fit All? What We Can and Can't Learn From a Meta-analysis of Housing Models for Persons With Mental Illness. Psychiatr. Serv. 60: 473-482 [Abstract] [Full text]  
• Diaz-Insua, M. (2009). Review: Methodological quality in medical evidence, quo vadis?. Therapeutic Advances in Urology 1: 51-59 [Abstract]  
• Brok, J., Thorlund, K., Wetterslev, J., Gluud, C. (2009). Apparently conclusive meta-analyses may be inconclusive--Trial sequential analysis adjustment of random error risk due to repetitive testing of accumulating data in apparently conclusive neonatal meta-analyses. Int J Epidemiol 38: 287-298 [Abstract] [Full text]  
• Nuesch, E., Juni, P. (2009). Commentary: Which meta-analyses are conclusive?. Int J Epidemiol 38: 298-303 [Full text]  
• Chartapisak, W, Opastiraku, S L, Willis, N S, Craig, J C, Hodson, E M (2009). Prevention and treatment of renal disease in Henoch-Schonlein purpura: a systematic review. Arch. Dis. Child. 94: 132-137 [Abstract] [Full text]  
• Huss, A. PhD, Scott, P. MSc, Stuck, A. E. MD, Trotter, C. PhD, Egger, M. MD MSc (2009). Efficacy of pneumococcal vaccination in adults: a meta-analysis. CMAJ 180: 48-58 [Abstract] [Full text]  
• Needleman, I., Moher, D., Altman, D.G., Schulz, K.F., Moles, D.R., Worthington, H. (2008). Improving the Clarity and Transparency of Reporting Health Research: a Shared Obligation and Responsibility. JDR 87: 894-895 [Full text]  
• Speedie, S. M., Taweel, A., Sim, I., Arvanitis, T. N., Delaney, B., Peterson, K. A. (2008). The Primary Care Research Object Model (PCROM): A Computable Information Model for Practice-based Primary Care Research. J. Am. Med. Inform. Assoc. 15: 661-670 [Abstract] [Full text]  
• Gluud, L. L., Thorlund, K., Gluud, C., Woods, L., Harris, R., Sterne, J. A.C. (2008). Correction: Reported Methodologic Quality and Discrepancies between Large and Small Randomized Trials in Meta-Analyses. ANN INTERN MED 149: 219-219 [Full text]  

Rapid Responses:

Read all Rapid Responses

Better evidence is needed for the validity and the reliability of tools assessing the risk of bias of individual trials

Jose M Valderas, et al.
bmj.com, 21 Mar 2008 [Full text]

Online poll

Find out more>>