Rapid Responses to:
|
|
Rapid Responses: Rapid Responses published:
-
![[Read Rapid Response]](/icons/misc/sectionDOWN.gif)
Further observations on Cochrane reviews
- Helen Handoll (18 October 2001)
-
Quality of Cochrane and other systematic reviews
- Mark Petticrew (19 October 2001)
-
Cochrane Reviews: are they reliable?
- Alan M Edwards, Jack B L, Howell, Michael T, Stevens, Brendan Callaghan, Simon Godfrey, Akihiro Morikawa, Stephen T. Holgate, Peter König, Nils Åberg, Brita Stenuis-Aarniala, John O, Warner (3 June 2005)
|
|
|||
|
Helen Handoll, research fellow Hull University
Send response to journal:
|
EDITOR, We write to endorse and extend Olsen et al.'s observations on Cochrane reviews.1 Last year we undertook a study of the utility of the Cochrane Database of Systematic Reviews in informing health policy and practice.2 We produced summary documents listing the conditions or diseases reviewed, the statements of evidence / effect and, where available, conclusions for policy and practice for the reviews from Collaborative Review Groups (CRGs) that covered cancer (including tobacco addiction), vascular disease and fractures. In assessing Cochrane reviews, we scrutinised high profile sections (review title, abstract, objectives, conclusions, synopsis); just as a busy healthcare professional would do. Where necessary, we inspected other sections of the review. Although we did not critically appraise review methodology, we recorded any errors, discrepancies, including discordance between the conclusions of effect and the available evidence, and other items needing clarification. We reported such information directly to the relevant CRG co-ordinators. We sent specific comments on 62 out of the 159 reviews processed in Issue 2, 2000 of The Cochrane Library. Although most comments were of minor nature, the possibility of inappropriate interpretation of results leading to spurious conclusions was considered likely in two reviews, disregard of unit of analysis problems likely in four reviews, and there were missing sections in two reviews. Failure to collect outcome data on adverse effects of treatment and quality of life / function was also commented on in several reviews. Our experience confirms that the majority of Cochrane reviews are of good standard. This is a considerable achievement, especially given the unpaid and voluntary nature of this work. However, there is scope for improvement. We believe that the regularly updated electronic publication, and comments and criticisms facility, offers great advantages in this regard. For instance, in cases where reviews with serious defects cannot be remedied speedily, their temporary removal is an important and laudable action. And, like Oslen et al., we stress the importance of feedback from users of the Cochrane Library. Finally, Olsen et al. conclude that readers should themselves assess the reliability of individual Cochrane reviews and they emphasise the need to learn the skills of critical appraisal. Whilst we support their recommendation, we are concerned that this may seem like 'let the buyer beware' advice. Given the broad readership, including lay people, of Cochrane reviews, the main emphasis must be on good quality and reliable reviews that people can trust. Helen Handoll, Research Fellow
Rajan Madhok, Professor
1. Olsen O, Middleton P, Ezzo J, Gøtzsche PC, Hadhazy V, Herxheimer A, Kleijnen J, McIntosh H. Quality of Cochrane reviews: assessment of sample from 1998. BMJ 2001; 323: 829-32 (13 October) 2. Handoll H, Madhok R. Utility of the Cochrane Database of Systematic Reviews for evidence-based health policy and practice: a case study. J Clin Excel 2001: 3; 59-68. Potential competing interest Both authors are affiliated to the Cochrane Musculoskeletal Injuries Group: Rajan Madhok is an editor, Helen Handoll was the previous CRG co- ordinator and both are currently active reviewers. The views expressed in the letter are the responsibility of the authors. |
|||
|
|
|||
|
Mark Petticrew, Associate Director MRC Social and Public Health Sciences Unit, Glasgow
Send response to journal:
|
Olsen and colleagues' recent paper [1] assesses a sample of Cochrane reviews from 1998 and highlights some areas where improvement is possible. They found that 29% of reviews had major problems, including inappropriate methods and conclusions. While improvement is always possible, this still represents a major improvement on the quality of non-Cochrane reviews. We recently carried out a review of the methods of 480 systematic reviews on the Database of Abstracts of Reviews of Effectiveness (DARE) at the University of York.[2,3] Methodological details of 480 systematic reviews were coded and checked by 2 reviewers working independently. We found that only about half (52%) the reviews had systematically assessed the validity of the included studies, that most systematic reviews were unlikely to be comprehensive (had searched either one or two databases) and overall only about a quarter (26%) of reviews met 3 key methodological criteria (relating to a thorough search, assessment of the validity of the included studies and investigation of heterogeneity). Narrative reviews were less likely to meet all three criteria (20% vs 30%, p=0.02), and were more likely to be coded by raters as inconclusive. It would therefore seem that up to 50% of non-Cochrane reviews are potentially misleading, against which Olsen and colleagues' estimate of 29% for Cochrane reviews compares favourably. While more recent research syntheses are likely to be of higher quality, particularly if reviewers follow current guidelines [4,5], it is likely that problems with the reliability of systematic reviews will remain. Since our study was conducted, the criteria for the inclusion of systematic reviews on the DARE database have been revised (from October 2000 onwards) to ensure that only reviews of potentially high methodological quality are included. We would support Olsen and colleagues' suggestion that users of any systematic review should assess its reliability, and recommend that for a critical assessment of the quality of non-Cochrane reviews, the DARE database should be the user's first port of call. Mark Petticrew
Paul Wilson, Kath Wright
Fujian Song
References 1. Olsen O, Middleton P, Ezzo J, Gøtzsche PC, Hadhazy V, Herxheimer A, Kleijnen J, McIntosh H. Quality of Cochrane reviews: assessment of sample from 1998 BMJ 2001; 323: 829-832. 2. DARE database: http://nhscrd.york.ac.uk/welcome.htm 3. Petticrew M, Song F, Wilson P, Wright K. The DARE database of abstracts of systematic reviews: a summary and analysis. International Journal of Technology Assessment in Health Care 2000; 15(4)671-8. 4. Moher D, Cook DJ, Eastwood S, Olkin I, Rennie D, Stroup DF, for the QUOROM Group. Improving the quality of reports of meta-analyses of randomised controlled trials: the QUOROM statement. Lancet 1999; 354:1896- 1900 5. NHS Centre for Reviews and Dissemination. Undertaking systematic reviews of research on effectiveness: CRD's guidance for those carrying out or commissioning reviews. York: CRD, 2001. Report number 4 (2nd ed). |
|||
|
|
|||
|
Alan M Edwards, Clinical assistant (Allergy) David Hide Asthma and Allergy Research Centre, St Mary's Hospital, Newport. Isle of Wight. PO305TG, Jack B L, Howell, Michael T, Stevens, Brendan Callaghan, Simon Godfrey, Akihiro Morikawa, Stephen T. Holgate, Peter König, Nils Åberg, Brita Stenuis-Aarniala, John O, Warner
Send response to journal:
|
Cochrane Reviews are considered to be among the most reliable sources of evidence relating to clinical trials. The Cochrane Collaboration which describes itself as “The most reliable source of evidence healthcare” states that Cochrane Reviews are prepared by health professionals “with editorial teams overseeing the preparation and maintenance of the reviews, as well as the application of the rigorous quality standards for which Cochrane Reviews have become known” (1). Our recent experience calls this judgement into question. The British Guideline on the Management of Asthma, 2003 (2) contains the statement, “sodium cromoglycate is ineffective in children” (para 4.2.4). This statement was based on the conclusion of a systematic review by Tasche et al. in 2000 (3). We had criticised this review and its conclusions when first published (4, 5) on the basis that the review did provide statistically significant evidence in favour of the efficacy of sodium cromoglycate compared to placebo. We conveyed our concerns to the authors of the guideline and although they were not persuaded to withdraw the statement, they did alter it in the 2004 update published on the website (6) to “The evidence of benefits of sodium cromoglicate is contentious” We were informed that a newly published Cochrane Review. “Inhaled sodium cromoglycate for asthma in children” (7) had come to the same conclusion as the Tasche et al 2000 review: not surprising given that both reviews came from the same group with virtually the same authors. We found this review also to be seriously flawed. We published detailed criticisms on the Cochrane website (8). These included inappropriate statistical analyses and incorrect interpretation of analyses presented; lack of homogeneity in the patient groups, which included the whole age range from new born babies to adolescents; the choice of a primary outcome measure for which data was available in only 4 of the 24 studies; and exclusion of trials without justification. Their reply (9) failed to address the majority of our criticisms but undertook to correct some of the points in a future update. They stated they would welcome large, methodologically strong trials which “would enable us to incorporate these new results in a future update of our review.” We responded (10) that this reply seemed to be a tacit admission that they were unable to justify their conclusion with the evidence reviewed. Over six months have elapsed and we have not seen an answer to our second comment. It is ironic that even with the flaws that we identify in their review, our analysis of their data provides statistically significant evidence in favour of SCG compared with placebo. This is not just an academic disagreement: the unjustified conclusion has led to an effective drug with an unrivalled safety record being denied a place in the treatment of asthma in children. There is even a risk that a clinician continuing to prescribe SCG could be charged with negligence. We are also aware that, based upon the two reviews from this group, another major European country has considered whether it too should remove from its guidelines the recommendation that SCG be used in the treatment of asthma in children. The implications for the reputation of Cochrane Reviews are serious. Criticisms of Cochrane reviews have been made previously (11, 12). Given its role in the promotion of Evidence-Based Medicine including providing evidence for the preparation of guidelines, the Cochrane Collaboration needs to clarify [explain] how it discharges its responsibilities for the quality of reviews published under its imprimatur, and how it responds when they are shown to have come to unjustified conclusions. Dr A M Edwards Clinical Assistant The David Hide Asthma and Allergy Research Centre St Mary’s Hospital Newport Isle of Wight. UK Professor JBL Howell Emeritus Professor of Medicine University of Southampton Southampton, UK M T Stevens Consultant Statistician EMStat Limited Leicester, UK Associate Professor N Åberg Head of Department Asthma and Allergy Department Queen Silvia Children’s Hospital Gothenburg, Sweden Dr. B Callaghan Consultant Physician (Retd.) Dublin, Ireland Professor Simon Godfrey Professor of Pediatrics, Institute of Pulmonology Hadassah University Hospital Jerusalem, Israel Professor Stephen T Holgate MRC Clinical Professor of Immunopharmacology Southampton General Hospital Southampton UK Professor Peter König Professor of Child Health Director, Division of Pediatric Pulmonary/Allergy School of Medicine Columbia, Missouri. USA Professor Akihiro Morikawa Professor of Paediatrics Department of Pediatrics and Developmental Medicine Gunma University Graduate School of Medicine Maebashi, Gunma Japan Professor Brita Stenius-Aarniala Professor (Emerita) of Respiratory Medicine University of Helsinki Helsinki Finland Professor J O Warner Professor of Child Health University Child Health Southampton General Hospital Southampton, UK References: 1. The Cochrane Collaboration www.cochrane.org 2. British Guideline on the Management of Asthma, Thorax 58, Supplement 1, 2003. 3. Tasche MJA, Uijen JHJM, Bersen RMD, et al. Inhaled sodium cromoglycate (DSCG) as maintenance therapy in children with asthma: a systematic review Thorax 2000; 55: 913-920. 4. Edwards A, Holgate S, Howell J, et al. Sodium cromoglycate in childhood asthma Thorax 2001; 56(4): 331-332. 5. Edwards A, Stevens M, Holgate S, et al. Inhaled sodium cromoglycate in children with asthma. Thorax 2002; 57(3): 282. . 6. http://www.brit-thoracic.org.uk/asthma-guideline-download.html 7. van der Wouden JC, Tasche MJA, Bernsen RMD et al. Inhaled sodium cromoglycate for asthma in children. (Cochrane Review). In The Cochrane Library, Issue 2, 2004, Chichester, UK: John Wiley & Sons Ltd. 8. http://www.update-software.com/ccng/ccng.exe?SourceID=CD002173#Content 1620 9. http://wwwe.update-software.com/ccng/ccng.exe?SourceID=CD002173#Content 1621 10. http://www.update- software.com/ccng/ccvng.exe?SourceID=CD002173#Content1654 11. Olsen O, Middleton P, Erzo J, et al. Quality of Cochrane reviews: assessment of sample from 1998. BMJ 2001; 323: 829-832. 12. Horsey P. Albumin and Hypovolaemia: is the Cochrane evidence to be trusted? Lancet, 2002; 359: 70-72. . Competing interests: None declared |
|||