Quality of Cochrane reviews: assessment of sample from 1998BMJ 2001; 323 doi: https://doi.org/10.1136/bmj.323.7317.829 (Published 13 October 2001) Cite this as: BMJ 2001;323:829
- Ole Olsen (), senior researchera,
- Philippa Middleton, assistant directorb,
- Jeanette Ezzo, systematic reviews coordinatorc,
- Peter C Gøtzsche, directora,
- Victoria Hadhazy, research associatec,
- Andrew Herxheimer, emeritus fellowd,
- Jos Kleijnen, directore,
- Heather McIntosh, lecturerb
- a See also editorial by Clarke and Langhorne Nordic Cochrane Centre, Rigshospitalet, Dept 7112, Blegdamsvej 9, DK-2100 Copenhagen Ø, Denmark
- b Australasian Cochrane Centre, Department of General Practice, Flinders Medical Centre, Adelaide, South Australia, Australia 5042
- c Cochrane Complementary Medicine Field, Complementary and Alternative Program, University of Maryland, School of Medicine, Baltimore, MD, USA
- d UK Cochrane Centre, NHS R&D Programme, Oxford OX2 7LG
- e NHS Centre for Reviews and Dissemination, University of York, York YO1 5DD
- Correspondence to: O Olsen
- Accepted 29 August 2001
Objective: To assess the quality of Cochrane reviews.
Design: Ten methodologists affiliated with the Cochrane Collaboration independently examined, in a semistructured way, the quality of reviews first published in 1998. Each review was assessed by two people; if one of them noted any major problems, they agreed on a common assessment. Predominant types of problem were categorised.
Setting: Cyberspace collaboration coordinated from the Nordic Cochrane Centre.
Studies: All 53 reviews first published in issue 4 of the Cochrane Library in 1998.
Main outcome measure: Proportion of reviews with various types of major problem.
Results: No problems or only minor ones were found in most reviews. Major problems were identified in 15 reviews (29%). The evidence did not fully support the conclusion in nine reviews (17%), the conduct or reporting was unsatisfactory in 12 reviews (23%), and stylistic problems were identified in 12 reviews (23%). The problematic conclusions all gave too favourable a picture of the experimental intervention.
Conclusions: Cochrane reviews have previously been shown to be of higher quality and less biased on average than other systematic reviews, but improvement is always possible. The Cochrane Collaboration has taken steps to improve editorial processes and the quality of its reviews. Meanwhile, the Cochrane Library remains a key source of evidence about the effects of healthcare interventions. Its users should interpret reviews cautiously, particularly those with conclusions favouring experimental interventions and those with many typographical errors.
What is already known on this topic
What is already known on this topic Cochrane reviews are, on average, more systematic and less biased than systematic reviews published in paper journals
Errors and biases also occur in Cochrane reviews
What this study adds
What this study adds Too often, reviewers' conclusions over-rated the benefits of new interventions
Readers of Cochrane reviews should remain cautious, especially regarding conclusions that favour new interventions
The Cochrane Collaboration has taken steps to improve the quality of reviews
Funding OO was funded by the Danish Institute for Health Technology Assessment.
Competing interests All assessors are associated with the Cochrane Collaboration.
Additional references plus full text of the submitted comments and criticisms are available on the BMJ's website
- Accepted 29 August 2001