Intended for healthcare professionals

News

Randomised controlled trials may have many unrecognised potential biases

BMJ 2018; 361 doi: https://doi.org/10.1136/bmj.k1561 (Published 05 April 2018) Cite this as: BMJ 2018;361:k1561

Re: Randomised controlled trials may have many unrecognised potential biases

This news item draws attention to a provocatively titled article claiming that all randomised trials are biased. This isn’t the place for a detailed critique of the paper by Krauss [1], but we alert BMJ readers not to accept at face value the claim that randomised controlled trials “inevitably produce bias”. Krauss’s paper is confused and confusing.

We do agree with Krauss that randomisation isn’t by itself adequate for a scientifically rigorous clinical trial and that there are, indeed, several ways in which bias may affect trials. However, Krauss confounds issues of internal and external validity, confuses bias and imprecision, and mixes methodology and reporting issues. He doesn’t really understand randomisation, overstating the importance of lack of balance of numbers of participants and baseline covariates between treatment arms, and he incorrectly believes that all participants can be randomised at the same time. Among other issues, he mistakenly believes that the participants in a trial should be fully representative of the wider population of such patients; he incorrectly labels small trials as biased; and he makes the remarkable observation that “trials generally report baseline data about participants, which assumes, erroneously, that they do not change during the course of a trial.”

Krauss’s sample was 10 highly cited trials published between 1993 and 2004 and presumably designed 5-10 years earlier than that. Having criticised these elderly trials for overgeneralising their findings he does just the same, extrapolating the problems identified to all modern trials.

This examination of randomised trials includes several errors and misunderstandings. It is by no means the first such contribution from a non-trialist – see [2,3].

[1] Krauss A. Why all randomised controlled trials produce biased results. Ann Med 2018 https://doi.org/10.1080/07853890.2018.1453233
[2] Senn S. Seven myths of randomisation in clinical trials. Stat Med 2013;32:1439-1450.
[3] https://errorstatistics.com/2018/01/13/s-senn-being-a-statistician-means...

Competing interests: No competing interests

13 April 2018
Douglas G Altman
Professor of statistics in medicine
Stephen J Senn (Competence Center for Methodology and Statistics, Luxembourg Institute of Health, Luxembourg)
Centre for Statistics in Medicine,
Nuffield Department of Orthopaedics, Rheumatology & Musculoskeletal Sciences, University of Oxford, Oxford