Why certain systematic reviews reach uncertain conclusions

doi:10.1136/bmj.326.7392.756

From trainee to consultant, BMJ Group offers doctors around the world tailored information, special events, learning resources and recruitment services at every step along their career path.

... by doctors, for doctors, for patients
We are open for entries!

Online learning

BMJ Learning

High-quality CME / CPD for doctors and other healthcare professionals. BMJ Learning features hundreds of accredited, peer-reviewed learning modules in text, video, and audio formats. Find out more

Courses and Qualifications

BMJ Masterclasses

BMJ Masterclasses, led by experts, help clinicians to use the latest evidence and recent guidelines in practice and meet their CPD/CME requirements. Find out more

Exam Preparation

The leading provider of online exam preparation, helping over 167,000 healthcare professionals to pass their exams. Find out more

onExamination

Search all BMJ education articles:

From

Limit by

Access to the full text of this article requires a subscription or payment. Please log in or subscribe below.

Mark Petticrew, associate director (mark@msoc.mrc.gla.ac.uk)

MRC Social and Public Health Sciences Unit, University of Glasgow, Glasgow G12 8RZ

The “stainless steel” law of evaluation states that the better designed the outcome evaluation, the less effective the intervention seems. This article explores how this law may be operating in relation to systematic reviews

Research syntheses are essential for putting studies in their proper scientific context and are increasingly common in public health, education, crime, and social welfare. A key criticism of systematic reviews, however, is that they are often unable to provide specific guidance on effective (or even ineffective) interventions; instead, they often conclude that little evidence exists to allow the question to be answered. This problem has been recognised in reviews of healthcare interventions,1 and the electronic journal Bandolier recently lamented the absence of systematic reviews containing a solid take home message.2 However, the problem is even more common in reviews of social and public health interventions, and this paper explains why.

Summary points

Systematic reviews are often criticised for being unable to provide specific guidance

This is often because the primary studies that they include contain few outcome evaluations

A “stainless steel” law of systematic reviews may also be operating—namely, the more rigorous the review, the less evidence there will be that the intervention is effective

Narrative review methods and narrative and meta-analytic approaches to reviewing observational data need to be improved

Uncertainty will often remain, but systematic reviews help us to acknowledge this and to map the areas of doubt

Sound systematic reviews may not guide practice

In public health there are few trials to review and indeed few other types of outcome assessment.3 Unsurprisingly, research users often regard reviews of such a limited evidence base as unhelpful and find their conclusions confusing and frustrating.4 This is ironic, given that systematic reviews are intended (among other things) to reduce uncertainty (box 1). Systematic reviews are certainly capable of doing this, and …