Research News

Dramatic trial results are often too good to be true

BMJ 2012; 345 doi: (Published 24 October 2012) Cite this as: BMJ 2012;345:e7107


New interventions rarely make large differences to patient outcomes when tested in trials. Trials that report very large effects (odds ratios ≥5 or ≤0.2) are usually small and investigate surrogate outcomes, such as laboratory test results, rather than clinical outcomes, such as death, say researchers. In an analysis of 85 002 forest plots from reviews published in the Cochrane Database of Systematic Reviews, 9.7% of plots included a first trial with a very large effect size, and 6.1 % of plots included a subsequent trial with a very large effect size. These trials reported fewer than 20 outcomes each (median), and odds ratios shrank substantially when results were fed into meta-analyses.

Among all 3082 reviews studied, the authors found just one intervention with a reliable, significant, sustained, and large beneficial effect on mortality—extracorporeal membrane oxygenation for newborns with respiratory failure (summary odds ratio relative to control 0.19, 95% CI 0.08 to 0.50).

These findings remind us to be humble when conducting clinical research, says a linked editorial (p 1691). We are looking for modest advances on previous treatments, not dramatic leaps forward that are unlikely to be real. Evidence from trials must be collated and summarised, preferably by large collaborations that can direct the activity towards important unanswered questions that matter to patients. The research agenda currently goes wherever individual researchers choose to look, and gaping holes remain in the evidence for some interventions. There isn’t a single systematic review summarising the effect of screening on mortality from cervical cancer, writes the author.


Cite this as: BMJ 2012;345:e7107