Analysis

The knowledge system underpinning healthcare is not fit for purpose and must change

BMJ 2015; 350 doi: https://doi.org/10.1136/bmj.h2463 (Published 03 June 2015) Cite this as: BMJ 2015;350:h2463
  1. Ian Roberts, professor of epidemiology and public health,
  2. Katharine Ker, lecturer in epidemiology,
  3. Phil Edwards, statistician,
  4. Deirdre Beecher, information scientist,
  5. Daniela Manno, clinical lecturer,
  6. Emma Sydenham, managing editor
  1. 1Cochrane Injuries Group, London School of Hygiene and Tropical Medicine, Keppel Street, London WC1E 7HT, UK
  1. Correspondence to: I Roberts ian.roberts{at}lshtm.ac.uk
  • Accepted 25 March 2015

The medical literature is biased and inundated with poor quality trials. Ian Roberts and colleagues explain how these problems affect systematic reviews and how they might be overcome

Information on the effectiveness and safety of healthcare should be valid, precise, up to date, clear, and freely available. Currently none of these criteria are fully satisfied, and Cochrane systematic reviews are not the solution. In this article we explain why the knowledge system for healthcare is unfit for purpose and suggest how it should change.

The problem with systematic reviews

Because the medical literature contains a biased sample of trials, systematic reviews that are based on it are also biased.1 Despite decades of exhortation about trial publication, about half of all trials are unpublished, and even the most diligent efforts to synthesise the results from all (or an unbiased sample of) relevant trials are in vain.1 2 Even when trials are identified selective outcome reporting limits their validity.3 4 The literature is awash with low quality, underpowered, single centre trials and the trend is upwards. As Altman recognised over 20 years ago “much poor research arises because researchers feel compelled for career reasons to carry out research that they are ill equipped to perform, and nobody stops them.”5

The systematic review movement is renowned for its obsessive zeal to find every published trial, irrespective of size or quality. These efforts have increased the accessibility of many trials that should never have been conducted. Their citation in reviews perpetuates the problem by contributing to the impact factor of the journals that published them. In the United Kingdom, the funding provided to Cochrane review groups is proportional to the number of trials included in reviews, creating a financial incentive to find and include every trial regardless of its quality.

Most journal editors and systematic …

View Full Text

Sign in

Log in through your institution

Free trial

Register for a free trial to thebmj.com to receive unlimited access to all content on thebmj.com for 14 days.
Sign up for a free trial

Subscribe