Cochrane Reviews are considered to be among the most reliable sources
of evidence relating to clinical trials. The Cochrane Collaboration which
describes itself as “The most reliable source of evidence healthcare”
states that Cochrane Reviews are prepared by health professionals “with
editorial teams overseeing the preparation and maintenance of the reviews,
as well as the application of the rigorous quality standards for which
Cochrane Reviews have become known” (1). Our recent experience calls this
judgement into question.
The British Guideline on the Management of Asthma, 2003 (2) contains
the statement, “sodium cromoglycate is ineffective in children” (para
4.2.4). This statement was based on the conclusion of a systematic review
by Tasche et al. in 2000 (3). We had criticised this review and its
conclusions when first published (4, 5) on the basis that the review did
provide statistically significant evidence in favour of the efficacy of
sodium cromoglycate compared to placebo.
We conveyed our concerns to the authors of the guideline and although
they were not persuaded to withdraw the statement, they did alter it in
the 2004 update published on the website (6) to “The evidence of benefits
of sodium cromoglicate is contentious” We were informed that a newly
published Cochrane Review. “Inhaled sodium cromoglycate for asthma in
children” (7) had come to the same conclusion as the Tasche et al 2000
review: not surprising given that both reviews came from the same group
with virtually the same authors.
We found this review also to be seriously flawed. We published
detailed criticisms on the Cochrane website (8). These included
inappropriate statistical analyses and incorrect interpretation of
analyses presented; lack of homogeneity in the patient groups, which
included the whole age range from new born babies to adolescents; the
choice of a primary outcome measure for which data was available in only 4
of the 24 studies; and exclusion of trials without justification.
Their reply (9) failed to address the majority of our criticisms but
undertook to correct some of the points in a future update. They stated
they would welcome large, methodologically strong trials which “would
enable us to incorporate these new results in a future update of our
review.” We responded (10) that this reply seemed to be a tacit admission
that they were unable to justify their conclusion with the evidence
reviewed. Over six months have elapsed and we have not seen an answer to
our second comment.
It is ironic that even with the flaws that we identify in their
review, our analysis of their data provides statistically significant
evidence in favour of SCG compared with placebo. This is not just an
academic disagreement: the unjustified conclusion has led to an effective
drug with an unrivalled safety record being denied a place in the
treatment of asthma in children. There is even a risk that a clinician
continuing to prescribe SCG could be charged with negligence. We are also
aware that, based upon the two reviews from this group, another major
European country has considered whether it too should remove from its
guidelines the recommendation that SCG be used in the treatment of asthma
in children.
The implications for the reputation of Cochrane Reviews are serious.
Criticisms of Cochrane reviews have been made previously (11, 12). Given
its role in the promotion of Evidence-Based Medicine including providing
evidence for the preparation of guidelines, the Cochrane Collaboration
needs to clarify [explain] how it discharges its responsibilities for the
quality of reviews published under its imprimatur, and how it responds
when they are shown to have come to unjustified conclusions.
Dr A M Edwards
Clinical Assistant
The David Hide Asthma and Allergy Research Centre
St Mary’s Hospital
Newport
Isle of Wight. UK
Professor JBL Howell
Emeritus Professor of Medicine
University of Southampton
Southampton, UK
M T Stevens
Consultant Statistician
EMStat Limited
Leicester, UK
Associate Professor N Åberg
Head of Department
Asthma and Allergy Department
Queen Silvia Children’s Hospital
Gothenburg, Sweden
Dr. B Callaghan
Consultant Physician (Retd.)
Dublin, Ireland
Professor Simon Godfrey
Professor of Pediatrics,
Institute of Pulmonology
Hadassah University Hospital
Jerusalem, Israel
Professor Stephen T Holgate
MRC Clinical Professor of Immunopharmacology
Southampton General Hospital
Southampton
UK
Professor Peter König
Professor of Child Health
Director, Division of Pediatric Pulmonary/Allergy
School of Medicine
Columbia, Missouri. USA
Professor Akihiro Morikawa
Professor of Paediatrics
Department of Pediatrics and Developmental Medicine
Gunma University
Graduate School of Medicine
Maebashi, Gunma
Japan
Professor Brita Stenius-Aarniala
Professor (Emerita) of Respiratory Medicine
University of Helsinki
Helsinki
Finland
Professor J O Warner
Professor of Child Health
University Child Health
Southampton General Hospital
Southampton, UK
References:
1. The Cochrane Collaboration www.cochrane.org
2. British Guideline on the Management of Asthma, Thorax 58, Supplement 1,
2003.
3. Tasche MJA, Uijen JHJM, Bersen RMD, et al. Inhaled sodium cromoglycate
(DSCG) as maintenance therapy in children with asthma: a systematic review
Thorax 2000; 55: 913-920.
4. Edwards A, Holgate S, Howell J, et al. Sodium cromoglycate in childhood
asthma Thorax 2001; 56(4): 331-332.
5. Edwards A, Stevens M, Holgate S, et al. Inhaled sodium cromoglycate in
children with asthma. Thorax 2002; 57(3): 282. .
6. http://www.brit-thoracic.org.uk/asthma-guideline-download.html
7. van der Wouden JC, Tasche MJA, Bernsen RMD et al. Inhaled sodium
cromoglycate for asthma in children. (Cochrane Review). In The Cochrane
Library, Issue 2, 2004, Chichester, UK: John Wiley & Sons Ltd.
8. http://www.update-software.com/ccng/ccng.exe?SourceID=CD002173#Content
1620
9. http://wwwe.update-software.com/ccng/ccng.exe?SourceID=CD002173#Content
1621
10. http://www.update-
software.com/ccng/ccvng.exe?SourceID=CD002173#Content1654
11. Olsen O, Middleton P, Erzo J, et al. Quality of Cochrane reviews:
assessment of sample from 1998. BMJ 2001; 323: 829-832.
12. Horsey P. Albumin and Hypovolaemia: is the Cochrane evidence to be
trusted? Lancet, 2002; 359: 70-72. .
Competing interests:
None declared
Competing interests:
No competing interests
03 June 2005
Alan M Edwards
Clinical assistant (Allergy)
Jack B L, Howell, Michael T, Stevens, Brendan Callaghan, Simon Godfrey, Akihiro Morikawa, Stephen T. Holgate, Peter König, Nils Åberg, Brita Stenuis-Aarniala, John O, Warner
David Hide Asthma and Allergy Research Centre, St Mary's Hospital, Newport. Isle of Wight. PO305TG
Rapid Response:
Cochrane Reviews: are they reliable?
Cochrane Reviews are considered to be among the most reliable sources
of evidence relating to clinical trials. The Cochrane Collaboration which
describes itself as “The most reliable source of evidence healthcare”
states that Cochrane Reviews are prepared by health professionals “with
editorial teams overseeing the preparation and maintenance of the reviews,
as well as the application of the rigorous quality standards for which
Cochrane Reviews have become known” (1). Our recent experience calls this
judgement into question.
The British Guideline on the Management of Asthma, 2003 (2) contains
the statement, “sodium cromoglycate is ineffective in children” (para
4.2.4). This statement was based on the conclusion of a systematic review
by Tasche et al. in 2000 (3). We had criticised this review and its
conclusions when first published (4, 5) on the basis that the review did
provide statistically significant evidence in favour of the efficacy of
sodium cromoglycate compared to placebo.
We conveyed our concerns to the authors of the guideline and although
they were not persuaded to withdraw the statement, they did alter it in
the 2004 update published on the website (6) to “The evidence of benefits
of sodium cromoglicate is contentious” We were informed that a newly
published Cochrane Review. “Inhaled sodium cromoglycate for asthma in
children” (7) had come to the same conclusion as the Tasche et al 2000
review: not surprising given that both reviews came from the same group
with virtually the same authors.
We found this review also to be seriously flawed. We published
detailed criticisms on the Cochrane website (8). These included
inappropriate statistical analyses and incorrect interpretation of
analyses presented; lack of homogeneity in the patient groups, which
included the whole age range from new born babies to adolescents; the
choice of a primary outcome measure for which data was available in only 4
of the 24 studies; and exclusion of trials without justification.
Their reply (9) failed to address the majority of our criticisms but
undertook to correct some of the points in a future update. They stated
they would welcome large, methodologically strong trials which “would
enable us to incorporate these new results in a future update of our
review.” We responded (10) that this reply seemed to be a tacit admission
that they were unable to justify their conclusion with the evidence
reviewed. Over six months have elapsed and we have not seen an answer to
our second comment.
It is ironic that even with the flaws that we identify in their
review, our analysis of their data provides statistically significant
evidence in favour of SCG compared with placebo. This is not just an
academic disagreement: the unjustified conclusion has led to an effective
drug with an unrivalled safety record being denied a place in the
treatment of asthma in children. There is even a risk that a clinician
continuing to prescribe SCG could be charged with negligence. We are also
aware that, based upon the two reviews from this group, another major
European country has considered whether it too should remove from its
guidelines the recommendation that SCG be used in the treatment of asthma
in children.
The implications for the reputation of Cochrane Reviews are serious.
Criticisms of Cochrane reviews have been made previously (11, 12). Given
its role in the promotion of Evidence-Based Medicine including providing
evidence for the preparation of guidelines, the Cochrane Collaboration
needs to clarify [explain] how it discharges its responsibilities for the
quality of reviews published under its imprimatur, and how it responds
when they are shown to have come to unjustified conclusions.
Dr A M Edwards
Clinical Assistant
The David Hide Asthma and Allergy Research Centre
St Mary’s Hospital
Newport
Isle of Wight. UK
Professor JBL Howell
Emeritus Professor of Medicine
University of Southampton
Southampton, UK
M T Stevens
Consultant Statistician
EMStat Limited
Leicester, UK
Associate Professor N Åberg
Head of Department
Asthma and Allergy Department
Queen Silvia Children’s Hospital
Gothenburg, Sweden
Dr. B Callaghan
Consultant Physician (Retd.)
Dublin, Ireland
Professor Simon Godfrey
Professor of Pediatrics,
Institute of Pulmonology
Hadassah University Hospital
Jerusalem, Israel
Professor Stephen T Holgate
MRC Clinical Professor of Immunopharmacology
Southampton General Hospital
Southampton
UK
Professor Peter König
Professor of Child Health
Director, Division of Pediatric Pulmonary/Allergy
School of Medicine
Columbia, Missouri. USA
Professor Akihiro Morikawa
Professor of Paediatrics
Department of Pediatrics and Developmental Medicine
Gunma University
Graduate School of Medicine
Maebashi, Gunma
Japan
Professor Brita Stenius-Aarniala
Professor (Emerita) of Respiratory Medicine
University of Helsinki
Helsinki
Finland
Professor J O Warner
Professor of Child Health
University Child Health
Southampton General Hospital
Southampton, UK
References:
1. The Cochrane Collaboration www.cochrane.org
2. British Guideline on the Management of Asthma, Thorax 58, Supplement 1,
2003.
3. Tasche MJA, Uijen JHJM, Bersen RMD, et al. Inhaled sodium cromoglycate
(DSCG) as maintenance therapy in children with asthma: a systematic review
Thorax 2000; 55: 913-920.
4. Edwards A, Holgate S, Howell J, et al. Sodium cromoglycate in childhood
asthma Thorax 2001; 56(4): 331-332.
5. Edwards A, Stevens M, Holgate S, et al. Inhaled sodium cromoglycate in
children with asthma. Thorax 2002; 57(3): 282. .
6. http://www.brit-thoracic.org.uk/asthma-guideline-download.html
7. van der Wouden JC, Tasche MJA, Bernsen RMD et al. Inhaled sodium
cromoglycate for asthma in children. (Cochrane Review). In The Cochrane
Library, Issue 2, 2004, Chichester, UK: John Wiley & Sons Ltd.
8. http://www.update-software.com/ccng/ccng.exe?SourceID=CD002173#Content
1620
9. http://wwwe.update-software.com/ccng/ccng.exe?SourceID=CD002173#Content
1621
10. http://www.update-
software.com/ccng/ccvng.exe?SourceID=CD002173#Content1654
11. Olsen O, Middleton P, Erzo J, et al. Quality of Cochrane reviews:
assessment of sample from 1998. BMJ 2001; 323: 829-832.
12. Horsey P. Albumin and Hypovolaemia: is the Cochrane evidence to be
trusted? Lancet, 2002; 359: 70-72. .
Competing interests:
None declared
Competing interests: No competing interests