Intended for healthcare professionals

News

Randomised controlled trials may have many unrecognised potential biases

BMJ 2018; 361 doi: https://doi.org/10.1136/bmj.k1561 (Published 05 April 2018) Cite this as: BMJ 2018;361:k1561

Re: Randomised controlled trials may have many unrecognised potential biases

Addressing the main points made in response to this study [1]:

Why do ‘all’ randomised controlled trials produce some degree of bias (and not only the 10 most cited trials)? This is often because some share of recruited people refuse to participate in any trial (which leads to sample bias), some degree of partial blinding or unblinding of the various trial persons generally arises in any trial (which leads to selection bias), participants generally take treatment for different lengths of time and different dosages in any trial (which leads to measurement bias), among many other issues. The ten most cited RCTs assessed in this study [1] are thus affected by such general issues. But other issues can also influence their estimated outcomes: participants’ characteristics (like age, health status, level of need for the treatment etc.) are often poorly distributed between trial groups, and trials often neglect alternative factors contributing to their main reported outcome, among others. Some of these issues cannot be avoided in trials. That trials face some degree of bias is simply the trade-off for studies to actually be conducted in the real world. A number of things do not always go as planned or designed. This is also related to the fact that the scientific process is a complex human process that involves many actors (trial designers, all participants, data collectors, practitioners/physicians, trial statisticians etc.) who must make many unique decisions at many different steps over time when designing, implementing and analysing any given study – and some degree of bias arises during this process.

Can ‘all participants be randomised at the same time’? In many – if not most – trials in fields like economics, agriculture and psychology we randomise the entire sample at the same time before conducting a trial; to better ensure a balanced distribution of background influencers between trial groups, and to do so over the same period of time and reduce other possible confounders, this approach would also be better for a number of relevant trials in medicine – including for example six of the ten most cited trials that tested treatments for common health conditions like diabetes and high cholesterol, lifestyle choices like increased exercise, and hormone use in postmenopausal women, as many potential participants exist at any time and there is no need to wait for participants.

Should ‘participants be fully representative of the wider population of such patients’? For many trials with the aim of later scaling up the treatment, samples need to be more (not fully) representative. In one of the top ten cited trials [2] participants were selected for insulin therapy in one surgical intensive care unit in Belgium; this implies that results cannot be applied to those in medical intensive care units or those with illnesses not present in the sample (which the study’s authors themselves acknowledge) but also to those with different demographic or clinical traits. In another trial [3], which provides most detail on their study’s applicability compared to other top 10 trials, the authors state that the results could apply to an estimated 3% of the US population, while acknowledging that: “The validity of generalizing the results of previous prevention studies is uncertain. Interventions that work in some societies may not work in others, because social, economic, and cultural forces influence [for example] diet and exercise”. The important issue is that researchers need to begin better outlining, in detail, the potential scope of their results for people outside the trial context – as outlined in the study [1].

The ‘remarkable observation’ made about baseline data is indeed remarkable, and found in Nigel Hawkes’ BMJ news article (not in the actual study). The study instead recommends that since participants’ background traits, and also clinic characteristics, can change over the course of a trial’s implementation, especially in longer trials, researchers need to control for changes in such background influencers during the trial by collecting endline data (not just baseline data) for them – which none of the top ten cited trials do.

Having worked at the World Bank for a team working on trials in developing countries, I became aware that not all researchers, practitioners as well as policymakers working with trials are familiar with a number of the issues facing RCTs. That is why the study’s main recommendation is that journals need to begin requiring researchers to outline the broader set of assumptions, biases and limitations in their trial studies – and that can only help improve trial quality and reduce the range of problems that affected the top ten cited trials. See the full study for further clarifications and a discussion of the wider range of issues facing trials and how to improve them [1].

[1] Krauss A. Why all randomised controlled trials produce biased results. Ann Med 2018 https://doi.org/10.1080/07853890.2018.1453233
[2] Van Den Berghe G, Wouters P, Weekers F, et al. Intensive insulin therapy in critically ill patients. N Engl J Med. 2001; 345:1359–1367.
[3] Knowler W, Barrett-Connor E, Fowler S, et al. Reduction in the incidence of type 2 diabetes with lifestyle intervention or metformin. N Engl J Med. 2002; 346:393–403.

Competing interests: No competing interests

03 May 2018
Alexander Krauss
Postdoctoral Research Fellow - London School of Economics
London