Editorials

Which clinical studies provide the best evidence?

The best RCT still trumps the best observational study

A common question in clinical consultations is: "For this person, what are the likely effects of one treatment compared with another?" The central tenet of evidence based medicine is that this task is achieved by using the best evidence combined with consideration of that person's individual needs.¹ A further question then arises: "What is the best evidence?" Two recent studies in the New England Journal of Medicine have caused uproar in the research community by finding no difference in estimates of treatment effects between randomised controlled trials and non-randomised trials.

The randomised controlled trial and, especially, systematic reviews of several of these trials are traditionally the gold standards for judging the benefits of treatments, mainly because it is conceptually easier to attribute any observed effect to the treatments being compared. The role of non-randomised (observational) studies in evaluating treatments is contentious: deliberate choice of the treatment for each person implies that observed outcomes may be caused by differences among people being given the two treatments, rather than the treatments alone. Unrecognised confounding factors can always interfere with attempts to correct for identified differences between groups.

These considerations have supported a hierarchy of evidence, with randomised controlled trials and derivatives at the top, controlled observational studies in the middle, and uncontrolled studies and opinion at the bottom. The best evidence to use in decisions is then the evidence highest in the hierarchy. Evidence from a lower level should be used only if there is no good randomised controlled trial to answer a particular clinical question. This view was supported by two studies that found larger effects in observational studies than in randomised controlled trials of the same treatment comparisons. ^{2 3}

However, these findings were not confirmed by the two latest studies in the New England Journal of Medicine, which compared individual randomised controlled trials with observational studies in 19 therapeutic areas⁴ and meta-analyses of randomised controlled trials with meta-analyses of cohort and case-control studies in five therapeutic areas.⁵ No major differences were found between the estimates of treatment effects in the observational studies and randomised controlled trials.

Do these newer results overturn the idea of best evidence and mean that we should abandon the use of a hierarchy of evidence? The authors speculate that their latest comparisons of study designs failed to confirm older studies for two main reasons. Firstly, observational studies have improved (people who are given different treatments may be more comparable or researchers may be better at allowing for residual differences), and secondly, earlier comparisons used particularly poor observational designs (such as historical controls that use control data from a different set of people and from an earlier period than the one used for the treatment being studied).

However, an accompanying editorial⁶ found three additional problems with the latest comparisons of observational studies and randomised controlled trials. ^{4 5} Firstly, the search for corresponding randomised controlled trials and observational studies in well known journals selected a small, potentially atypical subgroup of available randomised controlled trials. Conclusions based on the selected therapies might not extend to other areas. Secondly, one observational study did not involve any treatment but explored risk factors in the general population. Thirdly, meta-analyses and randomised controlled trials published after the studies in the New England Journal of Medicine did not follow the same pattern and disagreed with results of corresponding observational studies. For example, a new meta-analysis of breast cancer screening that included weighting by quality of randomised controlled trial found no evidence of benefit, in contrast to results from observational studies,⁷ and a randomised controlled trial of hormone replacement therapy in menopausal women found no secondary prevention of coronary risk or reduced fracture risk, in contrast to numerous observational studies.⁸

Even before the papers in the New England Journal of Medicine an earlier systematic review also found no consistent difference between randomised controlled trials and observational studies in estimates of the effects of treatment in 22 areas.⁹ Differential quality of care, selection of people with a larger capacity to benefit, and publication bias against negative results from observational studies could explain larger treatment effects in either study design.

The issue is further confused by another systematic review published in JAMA that compared eight randomised controlled trials with non-randomised trials of the same intervention and found larger effects in five of the non-randomised trials.¹⁰

It is not surprising that high quality randomised controlled trials and high quality observational studies can sometimes produce similar answers. Not all observational studies are misleading. The hierarchy of evidence is merely a convenient rule of thumb that, all other things being equal, randomised controlled trials are more able to attribute effects to causes. Randomised controlled trials that are well conducted remain the gold standard for evidence of efficacy. However, small inadequate ones do not automatically trump any conflicting observational study. Identifying the best evidence for any question requires detailed appraisalfor example, relevance, allocation concealment (ensuring that the assignment of interventions are unpredictable by all involved in the trial until the point of allocation), intention to treat analysis, and relevant outcomes. If high quality randomised controlled trials exist for a clinical question then they trump any number of observational studies. Limited randomised controlled trials need other forms of evidence to be appraised and considered.

A similar debate took place centuries ago in English law. The legal "best evidence rule" initially created a rigid hierarchy of evidence (that original written documents took precedence over oral evidence). It was replaced by the flexible principle that the weight given to each bit of evidence should be determined by a detailed appraisal of the characteristics of that evidence.¹¹

The new studies do not justify a major revision of the hierarchy of evidence, but they do support a flexible approach in which randomised controlled trials and observational studies have complementary roles. High quality observational studies may extend evidence over a wider population and are likely to be dominant in the identification of harms and when randomised controlled trials would be unethical or impractical.

Stuart Barton, executive editor, Clinical Evidence. 

BMA House, Tavistock Square, London WC1H 9JR (sbarton{at}bmjgroup.com)

Acknowledgments

Competing interests: SB edits a journal that uses a flexible hierarchy of evidence.