BMJ 2001;323:829-832 ( 13 October )

Papers

Quality of Cochrane reviews: assessment of sample from 1998

Ole Olsen, senior researcher, ^a Philippa Middleton, assistant director, ^b Jeanette Ezzo, systematic reviews coordinator, ^c Peter C Gøtzsche, director, ^a Victoria Hadhazy, research associate, ^c Andrew Herxheimer, emeritus fellow, ^d Jos Kleijnen, director, ^e Heather McIntosh, lecturer. ^b

^a See also editorial by Clarke and Langhorne Nordic Cochrane Centre, Rigshospitalet, Dept 7112, Blegdamsvej 9, DK-2100 Copenhagen Ø, Denmark, ^b Australasian Cochrane Centre, Department of General Practice, Flinders Medical Centre, Adelaide, South Australia, Australia 5042, ^c Cochrane Complementary Medicine Field, Complementary and Alternative Program, University of Maryland, School of Medicine, Baltimore, MD, USA, ^d UK Cochrane Centre, NHS R&D Programme, Oxford OX2 7LG, ^e NHS Centre for Reviews and Dissemination, University of York, York YO1 5DD

Correspondence to: O Olsen o.olsen{at}cochrane.dk

Objective: To assess the quality of Cochrane reviews.
Design: Ten methodologists affiliated with the Cochrane Collaboration independently examined, in a semistructured way, the quality of reviews first published in 1998. Each review was assessed by two people; if one of them noted any major problems, they agreed on a common assessment. Predominant types of problem were categorised.
Setting: Cyberspace collaboration coordinated from the Nordic Cochrane Centre.
Studies: All 53 reviews first published in issue 4 of the Cochrane Library in 1998.
Main outcome measure: Proportion of reviews with various types of major problem.
Results: No problems or only minor ones were found in most reviews. Major problems were identified in 15 reviews (29%). The evidence did not fully support the conclusion in nine reviews (17%), the conduct or reporting was unsatisfactory in 12 reviews (23%), and stylistic problems were identified in 12 reviews (23%). The problematic conclusions all gave too favourable a picture of the experimental intervention.
Conclusions: Cochrane reviews have previously been shown to be of higher quality and less biased on average than other systematic reviews, but
improvement is always possible. The Cochrane Collaboration has taken steps to improve editorial processes and the quality of its reviews. Meanwhile, the Cochrane Library remains a key source of evidence about the effects of healthcare interventions. Its users should interpret reviews cautiously, particularly those with conclusions favouring experimental interventions and those with many typographical errors.

What is already known on this topic Cochrane reviews are, on average, more systematic and less biased than systematic reviews published in paper journals Errors and biases also occur in Cochrane reviews What this study adds Too often, reviewers' conclusions over-rated the benefits of new interventions Readers of Cochrane reviews should remain cautious, especially regarding conclusions that favour new interventions The Cochrane Collaboration has taken steps to improve the quality of reviews

---

© BMJ 2001
CiteULike    Complore    Connotea    Del.icio.us    Digg    Reddit    Technorati    What's this?

Related Articles

Cochrane reviews compared with industry supported meta-analyses and other meta-analyses of the same drugs: systematic review

Anders W Jørgensen, Jørgen Hilden, and Peter C Gøtzsche
BMJ 2006 333: 782. [Abstract] [Full Text] [PDF]

Quality of Cochrane reviews

Mark Petticrew, Paul Wilson, Kath Wright, Fujian Song, Helen Handoll, and Rajan Madhok
BMJ 2002 324: 545. [Extract] [Full Text]

War, normality, and a time for women
BMJ 2001 323: 0. [Full Text] [PDF]

War, normality, and a time for women
BMJ 2001 323: 0. [Full Text] [PDF]

Some caution is needed in interpreting Cochrane reviews
BMJ 2001 323: 0. [Full Text]

Revisiting the Cochrane Collaboration

Mike Clarke and Peter Langhorne
BMJ 2001 323: 821. [Extract] [Full Text] [PDF]

This article has been cited by other articles:

• Jorgensen, A. W, Hilden, J., Gotzsche, P. C (2006). Cochrane reviews compared with industry supported meta-analyses and other meta-analyses of the same drugs: systematic review. BMJ 333: 782- [Abstract] [Full text]  
• Waters, E., Doyle, J., Jackson, N., Howes, F., Brunton, G., Oakley, A. (2006). Evaluating the effectiveness of public health interventions: the role and activities of the Cochrane Collaboration.. J. Epidemiol. Community Health 60: 285-289 [Abstract] [Full text]  
• Gilligan, P, Law, H, Lumsden, G, Brenchley, J, Kitching, G, Taylor, A, Khan, A, Shepherd, M, Jones, J, Hegarty, D (2005). SOCRATES 3 (synopsis of Cochrane reviews applicable to emergency services). Emerg. Med. J. 22: 50-52 [Full text]  
• Bigby, M., Williams, H. (2003). Appraising Systematic Reviews and Meta-analyses. Arch Dermatol 139: 795-798 [Abstract] [Full text]  
• Poikolainen, K. (2002). A NICE TRY THAT FAILS: THE SWEDISH COUNCIL ON TECHNOLOGY ASSESSMENT IN HEALTH CARE (SBU) EVALUATION OF THE EFFECT OF TREATMENT OF ALCOHOL AND DRUG PROBLEMS: THE EPIDEMIOLOGIST'S VIEW. Alcohol Alcohol 37: 416-418 [Abstract] [Full text]  
• Petticrew, M., Wilson, P., Wright, K., Song, F., Handoll, H., Madhok, R. (2002). Quality of Cochrane reviews. BMJ 324: 545-545 [Full text]  
• Ober, M., Shohara, R., Rennie, D., Yank, V., Bero, L. A. (2002). The Criticism Management System for the Cochrane Library. Eval Health Prof 25: 98-115 [Abstract]  
• Shea, B., Moher, D., Graham, I., Pham, B., Tugwell, P. (2002). A Comparison of the Quality of Cochrane Reviews and Systematic Reviews Published in Paper-Based Journals. Eval Health Prof 25: 116-129 [Abstract]  
• Laursen, L. C. (2002). Article from Clinical Evidence. BMJ 324: 428-428 [Full text]  
• (2001). Cochrane Reviews: Good, but Could Be Better. JWatch General 2001: 7-7 [Full text]  

Rapid Responses:

Read all Rapid Responses

Further observations on Cochrane reviews

Helen Handoll
bmj.com, 18 Oct 2001 [Full text]

Quality of Cochrane and other systematic reviews

Mark Petticrew
bmj.com, 19 Oct 2001 [Full text]

Cochrane Reviews: are they reliable?

Alan M Edwards, et al.
bmj.com, 3 Jun 2005 [Full text]