The unpredictability paradox: review of empirical comparisons of randomised and non-randomised clinical trials

BMJ 1998;317:1185-1190 ( 31 October )

Papers

The unpredictability paradox: review of empirical comparisons of randomised and non-randomised clinical trials

Regina Kunz, registrar, ^a Andrew D Oxman, director. ^b

^a Department of Nephrology, Charité, Berlin, Germany, ^b Health Services Research Unit, National Institute of Public Health, Oslo, Norway

Correspondence to: Dr Oxman andrew.oxman{at}labmed.uio.no

Objective To summarise comparisons of randomised clinical trials and non-randomised clinical trials, trials with adequatelyconcealed random allocation versus inadequately concealed randomallocation, and high quality trials versus low quality trialswhere the effect of randomisation could not be separated fromthe effects of other methodologicalmanoeuvres.
Design Systematicreview.
Selection criteria Cohorts or meta-analyses of clinical trials that included an empirical assessment of the relationbetween randomisation and estimates ofeffect.
Data sources Cochrane Review Methodology Database, Medline, SciSearch, bibliographies, hand searching of journals,personal communication with methodologists, and the referencelists of relevantarticles.
Main outcome measures Relation between randomisation and estimates ofeffect.
Results Eleven studies that compared randomised controlled trials with non-randomised controlled trials (eight forevaluations of the same intervention and three across differentinterventions), two studies that compared trials with adequatelyconcealed random allocation and inadequately concealed randomallocation, and five studies that assessed the relation betweenquality scores and estimates of treatment effects, were identified.Failure to use random allocation and concealment of allocationwere associated with relative increases in estimates of effectsof 150% or more, relative decreases of up to 90%, inversion ofthe estimated effect and, in some cases, no difference. On average,failure to use randomisation or adequate concealment of allocationresulted in larger estimates of effect due to a poorer prognosisin non-randomly selected control groups compared with randomlyselected controlgroups.
Conclusions Failure to use adequately concealed random allocation can distort the apparent effects of care in eitherdirection, causing the effects to seem either larger or smallerthan they really are. The size of these distortions can be aslarge as or larger than the size of the effects that are to bedetected.

Key messages

Empirical studies support using random allocation in clinical trials and ensuring that the allocation process is concealed $---$ that is, that assignment is impervious to any influence by the people making the allocation
The effect of not using concealed random allocation can be as large or larger than the effects of worthwhile interventions
On average, failure to use concealed random allocation results in overestimates of effect due to a poorer prognosis in non-randomly selected control groups compared with randomly selected control groups, but it can result in underestimates of effect, reverse the direction of effect, mask an effect, or give similar estimates of effect
The adequacy of allocation concealment may be a more sensitive measure of bias in clinical trials than scales used to assess the quality of clinical trials
It is a paradox that the unpredictability of randomisation is the best protection against the unpredictability of the extent and direction of bias in clinical trials that are not properly randomised

© BMJ 1998

CiteULike

Complore

Connotea

Del.icio.us Add to Digg

Digg

StumbleUpon Add to Technorati

Technorati What's this?

Relevant Articles

The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate healthcare interventions: explanation and elaboration: Alessandro Liberati, Douglas G Altman, Jennifer Tetzlaff, Cynthia Mulrow, Peter C Gøtzsche, John P A Ioannidis, Mike Clarke, P J Devereaux, Jos Kleijnen, and David Moher
BMJ 2009 339: b2700. [Abstract] [Full Text]

Use of primary care electronic medical record database in drug efficacy research on cardiovascular outcomes: comparison of database and randomised controlled trial findings: Richard L Tannen, Mark G Weiner, and Dawei Xie
BMJ 2009 338: b81. [Abstract] [Full Text] [PDF]

Perinatal mortality and other severe adverse pregnancy outcomes associated with treatment of cervical intraepithelial neoplasia: meta-analysis: M Arbyn, M Kyrgiou, C Simoens, A O Raifu, G Koliopoulos, P Martin-Hirsch, W Prendiville, and E Paraskevaidis
BMJ 2008 337: a1284. [Abstract] [Full Text] [PDF]

Oral misoprostol for induction of labour at term: randomised controlled trial: Jodie M Dodd, Caroline A Crowther, and Jeffrey S Robinson
BMJ 2006 332: 509-513. [Abstract] [Full Text] [PDF]

Fifty years of randomised controlled trials
BMJ 1998 317: 0. [Full Text] [PDF]

Concealed random allocation protects against bias in clinical trials
BMJ 1998 317: 0. [Full Text]

This article has been cited by other articles:

Hanson, R. K., Bourgon, G., Helmus, L., Hodgson, S. (2009). The Principles of Effective Correctional Treatment Also Apply To Sexual Offenders: A Meta-Analysis. Criminal Justice and Behavior 36: 865-891 [Abstract]
Liberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gotzsche, P. C., Ioannidis, J. P.A., Clarke, M., Devereaux, P. J., Kleijnen, J., Moher, D. (2009). The PRISMA Statement for Reporting Systematic Reviews and Meta-Analyses of Studies That Evaluate Health Care Interventions: Explanation and Elaboration. ANN INTERN MED 151: W-65-W-94 [Abstract] [Full text]
Liberati, A., Altman, D. G, Tetzlaff, J., Mulrow, C., Gotzsche, P. C, Ioannidis, J. P A, Clarke, M., Devereaux, P J, Kleijnen, J., Moher, D. (2009). The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate healthcare interventions: explanation and elaboration. BMJ 339: b2700-b2700 [Abstract] [Full text]
Ene-Iordache, B., Carminati, S., Antiga, L., Rubis, N., Ruggenenti, P., Remuzzi, G., Remuzzi, A. (2009). Developing Regulatory-compliant Electronic Case Report Forms for Clinical Trials: Experience with The Demand Trial. J. Am. Med. Inform. Assoc. 16: 404-408 [Abstract] [Full text]
Bhandari, M., Morshed, S., Tornetta, P. III, Schemitsch, E. H. (2009). Design, Conduct, and Interpretation of Nonrandomized Orthopaedic Studies: A Practical Approach. (All) Evidence Matters. JBJS 91: 1-1 [Full text]
Hoppe, D. J., Schemitsch, E. H., Morshed, S., Tornetta, P. III, Bhandari, M. (2009). Hierarchy of Evidence: Where Observational Studies Fit in and Why We Need Them. JBJS 91: 2-9 [Abstract] [Full text]
Simunovic, N., Sprague, S., Bhandari, M. (2009). Methodological Issues in Systematic Reviews and Meta-Analyses of Observational Studies in Orthopaedic Research. JBJS 91: 87-94 [Abstract] [Full text]
Riley, M., Bearn, D. R. (2009). A systematic review of clinical trials of aligning archwires. J. Orthod. 36: 42-51 [Abstract] [Full text]
Sjauw, K. D., Engstrom, A. E., Vis, M. M., van der Schaaf, R. J., Baan, J. Jr, Koch, K. T., de Winter, R. J., Piek, J. J., Tijssen, J. G.P., Henriques, J. P.S. (2009). A systematic review and meta-analysis of intra-aortic balloon pump therapy in ST-elevation myocardial infarction: should we change the guidelines?. Eur Heart J 30: 459-468 [Abstract] [Full text]
Tannen, R. L, Weiner, M. G, Xie, D. (2009). Use of primary care electronic medical record database in drug efficacy research on cardiovascular outcomes: comparison of database and randomised controlled trial findings. BMJ 338: b81-b81 [Abstract] [Full text]
French, J., Gronseth, G. (2008). Invited Article: Lost in a jungle of evidence: We need a compass. Neurology 71: 1634-1638 [Full text]
Gronseth, G., French, J. (2008). Invited Article: Practice parameters and technology assessments: What they are, what they are not, and why you should care. Neurology 71: 1639-1643 [Full text]
Onslow, M., Jones, M., O'Brian, S., Menzies, R., Packman, A. (2008). Defining, Identifying, and Evaluating Clinical Trials of Stuttering Treatments: A Tutorial for Clinicians. AJSLP 17: 401-415 [Abstract] [Full text]
Martens, E. P, Pestman, W. R, de Boer, A., Belitser, S. V, Klungel, O. H (2008). Systematic differences in treatment effect estimates between propensity score methods and logistic regression. Int J Epidemiol 37: 1142-1147 [Abstract] [Full text]
Arbyn, M, Kyrgiou, M, Simoens, C, Raifu, A O, Koliopoulos, G, Martin-Hirsch, P, Prendiville, W, Paraskevaidis, E (2008). Perinatal mortality and other severe adverse pregnancy outcomes associated with treatment of cervical intraepithelial neoplasia: meta-analysis. BMJ 337: a1284-a1284 [Abstract] [Full text]
Thomson, A. (2008). The Enteral vs Parenteral Nutrition Debate Revisited. JPEN J Parenter Enteral Nutr 32: 474-481 [Abstract] [Full text]
Karanicolas, P. J., Kunz, R., Guyatt, G. H. (2008). Point: Evidence-Based Medicine Has a Sound Scientific Base. Chest 133: 1067-1071 [Full text]
Pannu, N., Klarenbach, S., Wiebe, N., Manns, B., Tonelli, M., for the Alberta Kidney Disease Network, (2008). Renal Replacement Therapy in Patients With Acute Renal Failure: A Systematic Review. JAMA 299: 793-805 [Abstract] [Full text]
Morshed, S., Bhandari, M. (2008). Clinical Trial Design in Fracture-Healing Research: Meeting the Challenge. JBJS 90: 55-61 [Abstract] [Full text]
Yang, K. (2007). A Review of Yoga Programs for Four Leading Risk Factors of Chronic Diseases. Evid Based Complement Alternat Med 4: 487-491 [Abstract] [Full text]
Shrier, I., Boivin, J.-F., Steele, R. J., Platt, R. W., Furlan, A., Kakuma, R., Brophy, J., Rossignol, M. (2007). Should Meta-Analyses of Interventions Include Observational Studies in Addition to Randomized Controlled Trials? A Critical Examination of Underlying Principles. Am J Epidemiol 166: 1203-1209 [Abstract] [Full text]
Nicholl, J. (2007). Case-mix adjustment in non-randomised observational evaluations: the constant risk fallacy. J. Epidemiol. Community Health 61: 1010-1013 [Abstract] [Full text]
Adams, S. G., Smith, P. K., Allan, P. F., Anzueto, A., Pugh, J. A., Cornell, J. E. (2007). Systematic Review of the Chronic Care Model in Chronic Obstructive Pulmonary Disease Prevention and Management. Arch Intern Med 167: 551-561 [Abstract] [Full text]
Bonovas, S., Filioussi, K., Tsavaris, N., Sitaras, N. M. (2006). Statins and Cancer Risk: A Literature-Based Meta-Analysis and Meta-Regression Analysis of 35 Randomized Controlled Trials. JCO 24: 4808-4817 [Abstract] [Full text]
Foley, N. C, Bhogal, S. K, Teasell, R. W, Bureau, Y., Speechley, M. R (2006). Estimates of Quality and Reliability With the Physiotherapy Evidence-Based Database Scale to Assess the Methodology of Randomized Controlled Trials of Pharmacological and Nonpharmacological Interventions. ptjournal 86: 817-824 [Abstract] [Full text]
Gartlehner, G., Hansen, R. A., Carson, S. S., Lohr, K. N. (2006). Efficacy and Safety of Inhaled Corticosteroids in Patients With COPD: A Systematic Review and Meta-Analysis of Health Outcomes. Ann Fam Med 4: 253-262 [Abstract] [Full text]
Dodd, J. M, Crowther, C. A, Robinson, J. S (2006). Oral misoprostol for induction of labour at term: randomised controlled trial. BMJ 332: 509-513 [Abstract] [Full text]
Bonovas, S., Filioussi, K., Tsavaris, N., Sitaras, N. M. (2005). Use of Statins and Breast Cancer: A Meta-Analysis of Seven Randomized Clinical Trials and Nine Observational Studies. JCO 23: 8606-8612 [Abstract] [Full text]
Whelan, T., Levine, M. (2005). In Reply:. JCO 23: 6267-6267 [Full text]
Stengel, D., Bauwens, K., Rademacher, G., Mutze, S., Ekkernkamp, A. (2005). Association between Compliance with Methodological Standards of Diagnostic Research and Reported Test Accuracy: Meta-Analysis of Focused Assessment of US for Trauma. Radiology 236: 102-111 [Abstract] [Full text]
Hartling, L., McAlister, F. A., Rowe, B. H., Ezekowitz, J., Friesen, C., Klassen, T. P. (2005). Challenges in Systematic Reviews of Therapeutic Devices and Procedures. ANN INTERN MED 142: 1100-1111 [Abstract] [Full text]
van Exel, N.J.A., Koopmanschap, M.A., Scholte op Reimer, W., Niessen, L.W., Huijsman, R. (2005). Cost-effectiveness of integrated stroke services. QJM 98: 415-425 [Abstract] [Full text]
Lee, A., Cheng, F. F. K., St Leger, L. (2005). Evaluating health-promoting schools in Hong Kong: development of a framework. HEALTH PROMOT INT 20: 177-186 [Abstract] [Full text]
Hargrove, P., Lund, B., Griffer, M. (2005). A Guideline for Applying Systematic Reviews to Child Language Intervention. Communication Disorders Quarterly 26: 226-235 [Abstract]
Weisburd, D., Eck, J. E. (2004). What Can Police Do to Reduce Crime, Disorder, and Fear?. The ANNALS of the American Academy of Political and Social Science 593: 42-65 [Abstract]
Pitkin, R. M., Scott, J. R. (2004). Evaluating Gynecological Surgical Procedures: Trials and Tribulations. JAMA 291: 1503-1504 [Full text]
Lin, S Y-C, Davey, R C, Cochrane, T (2004). Community rehabilitation for older adults with osteoarthritis of the lower limb: a controlled clinical trial. Clin Rehabil 18: 92-101 [Abstract]
Marks, H. M (2003). Rigorous uncertainty: why RA Fisher is important. Int J Epidemiol 32: 932-937 [Full text]
Eddleston, M., Wilks, M.F., Buckley, N.A. (2003). Prospects for treatment of paraquat-induced lung fibrosis with immunosuppressive drugs and the need for better prediction of outcome: a systematic review. QJM 96: 809-824 [Abstract] [Full text]
Chalmers, I. (2003). Trying to do more Good than Harm in Policy and Practice: The Role of Rigorous, Transparent, Up-to-Date Evaluations. The ANNALS of the American Academy of Political and Social Science 589: 22-40 [Abstract]
Turlik, M. A., Kushner, D., Stock, D. (2003). Assessing the Validity of Published Randomized Controlled Trials in Podiatric Medical Journals. J. Am. Podiatr. Med. Assoc. 93: 392-398 [Abstract] [Full text]
Maher, C. G, Sherrington, C., Herbert, R. D, Moseley, A. M, Elkins, M. (2003). Reliability of the PEDro Scale for Rating Quality of Randomized Controlled Trials. ptjournal 83: 713-721 [Abstract] [Full text]
Sudlow, C. L M, Counsell, C. E (2003). Problems with UK government's risk sharing scheme for assessing drugs for multiple sclerosis. BMJ 326: 388-392 [Full text]
Jubb, A M (2002). Palliative care research: trading ethics for an evidence base. J. Med. Ethics 28: 342-346 [Abstract] [Full text]
van der Schaaf, I. C., Ruigrok, Y. M., Rinkel, G. J.E., Algra, A., van Gijn, J. (2002). Study Design and Outcome Measures in Studies on Aneurysmal Subarachnoid Hemorrhage. Stroke 33: 2043-2046 [Abstract] [Full text]
ZIGURAS, S.J., STUART, G.W., JACKSON, A.C. (2002). Assessing the evidence on case management. Br. J. Psychiatry 181: 17-21 [Abstract] [Full text]
Ciliska, D., Cullum, N., Marks, S. (2001). Evaluation of systematic reviews of treatment or prevention interventions. Evid. Based Nurs. 4: 100-104 [Full text]
Chalmers, I. (2001). Comparing like with like: some historical milestones in the evolution of methods to create unbiased comparison groups in therapeutic experiments. Int J Epidemiol 30: 1156-1164 [Abstract] [Full text]
McMahon, A. D. (2001). Observation and Experiment with the Efficacy of Drugs: A Warning Example from a Cohort of Nonsteroidal Anti-inflammatory and Ulcer-healing Drug Users. Am J Epidemiol 154: 557-562 [Abstract] [Full text]
Ioannidis, J. P. A., Haidich, A.-B., Pappa, M., Pantazis, N., Kokori, S. I., Tektonidou, M. G., Contopoulos-Ioannidis, D. G., Lau, J. (2001). Comparison of Evidence of Treatment Effects in Randomized and Nonrandomized Studies. JAMA 286: 821-830 [Abstract] [Full text]
Wiebe, S., Blume, W. T., Girvin, J. P., Eliasziw, M., the Effectiveness and Efficiency of Surgery for Te, (2001). A Randomized, Controlled Trial of Surgery for Temporal-Lobe Epilepsy. NEJM 345: 311-318 [Abstract] [Full text]
Renders, C. M., Valk, G. D., Franse, L. V., Schellevis, F. G., van Eijk, J. Th.M., van der Wal, G. (2001). Long-Term Effectiveness of a Quality Improvement Program for Patients With Type 2 Diabetes in General Practice. Diabetes Care 24: 1365-1370 [Abstract] [Full text]
Evans, A., Kalra, L. (2001). Are the Results of Randomized Controlled Trials on Anticoagulation in Patients With Atrial Fibrillation Generalizable to Clinical Practice?. Arch Intern Med 161: 1443-1447 [Abstract] [Full text]
Bhandari, M., Guyatt, G. H., Swiontkowski, M. F. (2001). User's Guide to the Orthopaedic Literature: How to Use an Article About a Surgical Therapy. JBJS 83: 916-926 [Abstract] [Full text]
Devereaux, P.J., Manns, B. J., Ghali, W. A., Quan, H., Guyatt, G. H. (2001). Reviewing the reviewers: the quality of reporting in three secondary journals. CMAJ 164: 1573-1576 [Abstract] [Full text]
Altman, D. G., Schulz, K. F., Moher, D., Egger, M., Davidoff, F., Elbourne, D., Gotzsche, P. C., Lang, T., for the CONSORT Group, (2001). The Revised CONSORT Statement for Reporting Randomized Trials: Explanation and Elaboration. ANN INTERN MED 134: 663-694 [Abstract] [Full text]
Donnelly, C A, Cox, D R (2001). Mathematical biology and medical statistics: contributions to the understanding of AIDS epidemiology. Stat Methods Med Res 10: 141-154 [Abstract]
Bigby, M. (2001). Challenges to the Hierarchy of Evidence: Does the Emperor Have No Clothes?. Arch Dermatol 137: 345-346 [Full text]
Weisburd, D., Lum, C. M., Petrosino, A. (2001). Does Research Design Affect Study Outcomes in Criminal Justice?. The ANNALS of the American Academy of Political and Social Science 578: 50-70 [Abstract]
Kunz, R., Khan, K. S., Neumayer, H.-H., Sacks, H. S., Liu, P.-Y., Anderson, G., Crowley, J. J., Friedman, H. S., Smith, R. P., Meier, P., Benson, K., Hartz, A. J., Concato, J., Shah, N., Horwitz, R. I. (2000). Observational Studies and Randomized Trials. NEJM 343: 1194-1197 [Full text]
Guyatt, G. H., Haynes, R. B., Jaeschke, R. Z., Cook, D. J., Green, L., Naylor, C. D., Wilson, M. C., Richardson, W. S., for the Evidence-Based Medicine Working Group, (2000). Users' Guides to the Medical Literature: XXV. Evidence-Based Medicine: Principles for Applying the Users' Guides to Patient Care. JAMA 284: 1290-1296 [Abstract] [Full text]
Barton, S. (2000). Which clinical studies provide the best evidence?. BMJ 321: 255-256 [Full text]
Benson, K., Hartz, A. J. (2000). A Comparison of Observational Studies and Randomized, Controlled Trials. NEJM 342: 1878-1886 [Abstract] [Full text]
Concato, J., Shah, N., Horwitz, R. I. (2000). Randomized, Controlled Trials, Observational Studies, and the Hierarchy of Research Designs. NEJM 342: 1887-1892 [Abstract] [Full text]
Dickinson, K., Bunn, F., Wentz, R., Edwards, P., Roberts, I. (2000). Size and quality of randomised controlled trials in head injury: review of published studies. BMJ 320: 1308-1311 [Abstract] [Full text]
Adetugbo, K., Williams, H. (2000). How Well Are Randomized Controlled Trials Reported in the Dermatology Literature?. Arch Dermatol 136: 381-385 [Abstract] [Full text]
Mariani, L., Marubini, E. (2000). Content and Quality of Currently Published Phase II Cancer Trials. JCO 18: 429-429 [Abstract] [Full text]
Goodacre, S. W (2000). Should we establish chest pain observation units in the UK? A systematic review and critical appraisal of the literature. Emerg. Med. J. 17: 1-6 [Abstract] [Full text]
Roberts, I., Diguiseppi, C. (1999). Smoke alarms, fire deaths, and randomised controlled trials. Inj. Prev. 5: 244-245 [Full text]
Juni, P., Witschi, A., Bloch, R., Egger, M. (1999). The Hazards of Scoring the Quality of Clinical Trials for Meta-analysis. JAMA 282: 1054-1060 [Abstract] [Full text]
McKee, M., Britton, A., Black, N., McPherson, K., Sanderson, C., Bain, C. (1999). Methods in health services research: Interpreting the evidence: choosing between randomised and non-randomised studies. BMJ 319: 312-315 [Full text]
Chalmers, I. (1998). Unbiased, relevant, and reliable assessments in health care. BMJ 317: 1167-1168 [Full text]