Intended for healthcare professionals


Commentary: does the selective inversion approach demonstrate bias in the results of studies using routinely collected data?

BMJ 2018; 362 doi: (Published 17 August 2018) Cite this as: BMJ 2018;362:k3259
  1. Jonathan Sterne, professor of medical statistics and epidemiology
  1. NIHR Bristol Biomedical Research Centre, Department of Population Health Sciences, Bristol Medical School, University of Bristol, Bristol BS8 2BN, UK
  1. jonathan.sterne{at}

In early 2016, The BMJ published a paper by Hemkens and colleagues entitled “Agreement of treatment effects for mortality from routinely collected data and subsequent randomized trials: meta-epidemiological survey” (doi:10.1136/bmj.i493).1 This study used a novel design: the authors identified observational studies based on routinely collected data (RCD) that were performed before any randomised controlled trial (RCT) on the same clinical question, reported on all cause mortality, and used a propensity score based analysis to control confounding. The authors identified 16 RCD studies for which results from RCTs were subsequently reported. The main result was: “Overall, RCD studies showed significantly more favorable mortality estimates by 31% than subsequent trials (summary relative odds ratio [ROR] 1.31 (95% CI 1.03 to 1.65; I2=0%)).” The authors also reported that “Confidence intervals in nine (56%) RCD studies did not include the RCT effect estimate.” They concluded that “studies using routinely collected health data do not agree with subsequent randomized trials” and that “the emerging pattern was that RCD studies systematically and substantially overestimated the mortality benefits of medical treatments compared with subsequent trials investigating the same question.”

In August 2016, Franklin and colleagues submitted to The BMJ a paper that criticised the methods used by the Hemkens article. After The BMJ invited these authors to submit their comments as a rapid response rather than a separate paper, the work was published in Epidemiologic Methods.2 In their article, Hemkens and colleagues had “inverted the RCD effect estimates where necessary so that each RCD study indicated an odds ratio less than 1”.1 Their aim was to overcome the problem that some of the pairs compared two active interventions, so that the choice of direction of comparison was arbitrary. Franklin and colleagues showed that, under reasonable assumptions, the selective inversion …

View Full Text

Log in

Log in through your institution


* For online subscription