Education and debate

Why certain systematic reviews reach uncertain conclusions

BMJ 2003; 326 doi: 10.1136/bmj.326.7392.756 (Published 5 April 2003)
Cite this as: BMJ 2003;326:756.1

Access to the full text of this article requires a subscription or payment. Please log in or subscribe below.

  1. Mark Petticrew, associate director (mark@msoc.mrc.gla.ac.uk)
  1. MRC Social and Public Health Sciences Unit, University of Glasgow, Glasgow G12 8RZ

    The “stainless steel” law of evaluation states that the better designed the outcome evaluation, the less effective the intervention seems. This article explores how this law may be operating in relation to systematic reviews

    Research syntheses are essential for putting studies in their proper scientific context and are increasingly common in public health, education, crime, and social welfare. A key criticism of systematic reviews, however, is that they are often unable to provide specific guidance on effective (or even ineffective) interventions; instead, they often conclude that little evidence exists to allow the question to be answered. This problem has been recognised in reviews of healthcare interventions,1 and the electronic journal Bandolier recently lamented the absence of systematic reviews containing a solid take home message.2 However, the problem is even more common in reviews of social and public health interventions, and this paper explains why.

    Summary points

    Systematic reviews are often criticised for being unable to provide specific guidance

    This is often because the primary studies that they include contain few outcome evaluations

    A “stainless steel” law of systematic reviews may also be operating—namely, the more rigorous the review, the less evidence there will be that the intervention is effective

    Narrative review methods and narrative and meta-analytic approaches to reviewing observational data need to be improved

    Uncertainty will often remain, but systematic reviews help us to acknowledge this and to map the areas of doubt

    Sound systematic reviews may not guide practice

    In public health there are few trials to review and indeed few other types of outcome assessment.3 Unsurprisingly, research users often regard reviews of such a limited evidence base as unhelpful and find their conclusions confusing and frustrating.4 This is ironic, given that systematic reviews are intended (among other things) to reduce uncertainty (box 1). Systematic reviews are certainly capable of doing this, and …

    Access to the full text of this article requires a subscription or payment

    Subscribe

    Article access

    Article access for 1 day

    Purchase this article for £20 $30 €32*

    The PDF version can be downloaded as your personal record

    * Prices do not include VAT

    Rapid responses

    THIS WEEK'S POLL

    BMJ most popular