Intended for healthcare professionals

Education And Debate

The need for caution in interpreting high quality systematic reviews

BMJ 2001; 323 doi: (Published 22 September 2001) Cite this as: BMJ 2001;323:681
  1. Kevork Hopayian, general practitioner (
  1. Seahills, Leiston Road, Aldeburgh IP15 5PL
  • Accepted 25 June 2001

The emergence of systematic reviews raised hopes of a new era for the objective appraisal of evidence available on a given topic. Such reviews promised a synthesis of trial results, which could be conflicting, and an escape from the personal bias inherent in traditional reviews and expert opinion.1 As the discipline of systematic reviews has evolved, however, two new problems have arisen: the quality of reviews is variable 2 3; and two or more systematic reviews on the same topic may arrive at different conclusions, raising questions on the validity4-7 or the relevance8 of the conclusions. Moreover, adherence to a “checklist” system when appraising trials may overlook important clinical details in the original trials and so reduce the validity of the review. I uncovered this last shortcoming when I recently conducted a study of three systematic reviews; the study is reported here.

Summary points

The discipline of systematic reviews has given clinicians a valuable tool with which to synthesise evidence

As the methodology of systematic reviews has evolved, the quality of reviews has improved

Nevertheless, high quality systematic reviews may overlook important clinical details in the papers reviewed, thereby diminishing their validity

This shortcoming might be avoided if trials were assessed from a clinician's viewpoint as well as from a reviewer's viewpoint


Guidelines have been drawn up to improve the quality of reviews.9 Differences in the quality of reviews, however, do not always explain discordance. Jadad and McQuay4 identified six sets of reviews covering six topics in pain research; despite similar quality scores for reviews in each set, four of the sets contained discordant reviews. Jadad et al8 identified six generic differences between reviews that might lead to discordance: the clinical question asked; the selection and inclusion of studies; data extraction; assessment of study quality; …

View Full Text

Log in

Log in through your institution


* For online subscription