RoB 2: a revised tool for assessing risk of bias in randomised trialsBMJ 2019; 366 doi: https://doi.org/10.1136/bmj.l4898 (Published 28 August 2019) Cite this as: BMJ 2019;366:l4898
All rapid responses
Sterne et al. present RoB2, a revised tool for assessing risk of bias in randomized trials. The finalized version is rather complex. I would like to propose three changes that might help to clarify the new tool.
It seems that RoB2 makes it easier for trial results to be rated as low risk of bias for a particular domain although the overall risk of bias corresponds to the worst risk of bias in any of the domains. This delicate balance might be lost if reviewers are allowed to downplay the overall risk of bias. I would, therefore, recommend adding a requirement to use the overall risk of bias judgment in interpreting results and formulating a conclusion. Otherwise, RoB2 would simply lower the bar and rate trial results at a lower risk of bias than the previous assessment tool.  Without a focus on the overall risk of bias, the traffic light presentation would show more green lights than was previously the case. This could be misleading to readers unaware of how the overall judgment should be reached.
Secondly, question 4.5 seems redundant. After asking if outcome assessors were aware of the intervention received and if that knowledge could have influenced the assessment of the outcome, question 4.5 asks whether this was likely or not. The elaboration in the supplementary material does not give an example where such influence is not likely and it’s rather difficult to imagine a relevant scenario. A trial in which patients are not blinded and subjective outcomes are used, for example, is at high risk of bias.  Low expectations measured before the trial would change little to that, as patients’ hopes and expectations might alter during the treatment or following contact with (unblinded) therapists. As question 4.5 adds little to the risk of bias assessment and could easily be misused, I would recommend removing it from RoB2. Answers ‘yes’, ‘probably yes’, and ‘no information’ to question 4.4 would then directly result in a high risk of bias judgment.
Finally, there seems to be an issue with question 5.1 which asks if the analysis of the results were in accordance with a pre-specified analysis plan. The elaboration states that “changes to analysis plans that were made before unblinded outcome data were available (…) do not raise concerns about bias in selection of the reported result.” I suspect that in open-label trials, researchers can get a hint of the size and direction of the results before looking at the data. The previous Cochrane risk of bias tool required all relevant outcomes to be reported in a pre-specified way for a study to be rated as low risk of bias if a protocol was available.  I would recommend restoring this requirement as it is less ambiguous and failure to meet it would only result in ‘some concerns’, not a high risk of bias judgment.
 Higgins JPT, Altman DG, Gøtzsche PC, et al, Cochrane Bias Methods Group, Cochrane Statistical Methods Group. The Cochrane Collaboration’s tool for assessing risk of bias in randomised trials. BMJ 2011;343:d5928. doi:10.1136/bmj.d5928
 Savović J, Jones H, Altman D, Harris R, Jűni P, Pildal J, et al. Influence of reported study design characteristics on intervention effect estimates from randomised controlled trials: combined analysis of meta-epidemiological studies. Health Technol Assess. 2012 Sep;16(35):1-82. doi: 10.3310/hta16350
Competing interests: No competing interests
In the recent publication in BMJ of the finalised Risk of Bias 2 tool (RoB2), used in the context of the GRADE system by Cochrane for assessing quality of clinical trial evidence, the discussion claims that changes made are likely to reduce stringency of RoB assessment, especially in unblinded trials. This looks to be a retrograde step, since unblinded trials with subjective outcomes appear increasingly to be given more credence than deserved. What is also of concern is that the corresponding author, Jonathan Sterne, is an author on the report of the unblinded SMILE trial of the Lightning Process for myalgic encephalomyelitis/ chronic fatigue syndrome (ME/CFS) (Crawley et al. 2018). This report has been severely criticised for multiple methodological errors. Despite rewriting, it is still considered inadequate by over 50 academics and clinicians (Tuller, 2019).
The proposals the RoB2 document makes include handling of problems with bias due to beliefs held by the patient or treatment delivery team when outcome measures are subjective and also the changing of outcome measures after trial initiation but before data analysis. Problems with the SMILE trial include both severe risk of bias relating to subjective outcomes and outcome switching midstream.
The GRADE system takes the premise that randomised trials provide high quality evidence unless they suffer from one or more defects, including bias. The fragility of this can be illustrated by proposing a trial that randomises patients to being taught to think and say they feel better, whether they do or not, or to being told to say how they really feel, using how the patient says they feel as outcome. One might think such a trial would never be proposed. Yet this appears to be more or less what the SMILE trial was (at least from the little we know about the Lightning Process). Moreover, the report does not tell us much, and it looks as if according to RoB2 the risk of bias is not scored high if we do not know enough to have specific reasons for doing so!
RoB2 does mention the possibility that bias might arise from patient or therapist beliefs, citing a physiotherapist assessing her own treatment or a patient reporting their response to homeopathy, as if to indicate bias is restricted to such self-evident examples. There appears to be no recognition that expectation bias due to beliefs about the outcome of an intervention is ubiquitous in trials (and any experiment, including laboratory work). In ME/CFS we have seen major responses due to expectation bias with conventional drugs now known to have no therapeutic effect, such as rituximab. The RoB2 analysis seems either naïve or disingenuous.
The SMILE trial had a highly unsatisfactory structure, starting as a ‘feasibility study’, recruiting more than half of the patients, and then being registered after switching of outcomes in the knowledge of progress of early participants. This might fall under RoB2’s condition of before data analysis but is a classic cherry-picking scenario. ‘Feasibility’ and ‘pragmatic’ trials appear increasingly popular. These terms look like attempts to legitimise methods that break basic rules for gathering reliable evidence. It needs to be acknowledged that ‘methodological experts’ associated with clinical trials units and related departments may have a conflict of interest in terms of co-authorship on publications of such trials.
We appear to be moving towards acceptance of methodology that for decades we have known yields meaningless results. I am concerned that bodies like Cochrane and the BMJ are sleepwalking into a situation where they rubber stamp commercial ventures of no merit.
Crawley, E.M., Gaunt, D.M., Garfield, K., et al. (2018). Arch. Dis. Childhood 103,2. https://adc.bmj.com/content/103/2/155
Tuller, D (2019). http://www.virology.ws/2019/08/28/trial-by-error-an-open-letter-to-dr-go...
Competing interests: No competing interests