Education And Debate

Analysis of quality of interventions in systematic reviews

BMJ 2005; 331 doi: (Published 01 September 2005) Cite this as: BMJ 2005;331:507
  1. Robert D Herbert, associate professor (,
  2. Kari B⊘, professor2
  1. 1 Centre for Evidence-Based Physiotherapy, University of Sydney, PO Box 170, Lidcombe, NSW 1825, Australia
  2. 2 Norwegian School of Sport Sciences, Department of Sport Medicine, Oslo, Norway
  1. Correspondence to: R Herbert
  • Accepted 17 May 2005

The quality of interventions can affect the results of clinical trials. Reviews of complex interventions need to take this into account


Complex health interventions, such as surgery or physiotherapy, can be administered well or badly. Variation in the quality of administration of interventions may explain some of the variability in estimates of effects between trials in systematic reviews. We argue that systematic reviews of complex interventions should assess the quality of interventions, and we suggest how to make such assessments.

Effect of training pelvic floor muscle on urinary incontinence in pregnancy. Pooled estimate for all trials with sufficient data (top) and excluding a trial without supervised exercise (bottom)

Best evidence

Randomised controlled trials provide the best test of the efficacy of preventive or therapeutic interventions because they can separate the effects of the intervention from those of extraneous factors such as natural recovery and statistical regression. When more than one trial has examined a particular intervention, systematic reviews potentially provide the best summaries of the available evidence.1

Systematic reviewers can summarise findings of randomised trials using an impressionistic approach (qualitative synthesis) or they can produce quantitative syntheses by statistically combining the results from several studies (meta-analysis). Regardless of the method used, most systematic reviewers seek to reduce data from clinical trials into simple statements about treatment. Systematic reviews that provide succinct statements about the effects of an intervention are particularly useful to clinicians. But simple summaries are possible only when the studies address similar questions in similar ways.


We can differentiate several types of heterogeneity between trials. Clinical heterogeneity can be identified before analysis of data. It may be due to variations across trials in the inclusion and exclusion criteria, the way in which the intervention is administered or the outcomes that are measured. Statistical heterogeneity occurs when estimates of effects …

Sign in

Free trial

Register for a free trial to to receive unlimited access to all content on for 14 days.
Sign up for a free trial