Before We Put Too Much Stock In Cochrane Systematic Reviews…Or Anyone’s…
The demonstration by JØrgensen et al. (1) that industry sponsored
meta-analyses differ in their conclusions and recommendations from non-
industry sponsored meta-analyses should surprise no one. Yet, this
demonstration should not lull us into believing that industry sponsorship
is the only source of bias or that the Cochrane reviews to which the
industry sponsored ones were compared should be in all cases uncritically
It is well established that investigator allegiance is a strong
determinant of the outcome of the evaluation of interventions, even in the
absence of industry ties (2). Allegiance of the authors of meta-analyses
have also been associated with selective attention to relevant studies,
less critical evaluation of favorable studies, and more positive
conclusions (3,4). This suggests a general need to scrutinize the
backgrounds of authors, not just their declarations of conflict of
interest. It also invites skepticism about a meta-analysis co-authored by
the director of Cochrane Centre that puts the centre in such a favorable
Cochrane reviews sometimes are conducted for literatures that are not
yet ready for meta-analysis. As a case in point, a recent Cochrane meta-
analysis concluded that there that couples therapy was not significantly
better than individual therapy for depression (5). Whether couples therapy
is offered should be a matter of “patient preference and availability of
specific resources.” Such a conclusion is unlikely to prompt the
redistribution of scarce resources to having marital therapists trained
and available and may serve to discourage commitment of resources to an
adequate comparison between the two forms of therapy because the
authoritative Cochrane Collaboration has already spoken. Lack of evidence
can easily be confused as evidence of a lack of differences. Yet, the
studies reviewed were all seriously flawed, and none had close to the
minimal cell size deemed necessary for inclusion in a meta-analysis (6),
much less for a noninferiority or equivalence trial ( 7). Sometimes, the
most appropriate conclusion is that a literature is not ready for meta-
analysis (8. The Cochrane Collaborative has a tendency to err in the other
direction, and with important implications for clinical practice and
Whether the output of the Cochrane Collaborative is better, less
biased, or less tainted by conflict of interest than reviews from other
sources should be evaluated by someone other than a member of the
collaborative. One of the founding members of the collaborative who still
lists the collaborative as his institutional affiliation sometimes
discloses his activities as an expert witness in product liability suits
which his articles appear to benefit (9) and sometimes he fails to do so
(10, 11). More to the point, Bjordal and colleagues (12) recently did a
troubling analysis of a Cochrane report on low level laser therapy in
osteoarthritis. They showed that only investigators who had performed
trials with negative findings had been recruited to the review group.
Furthermore, when asessed with a standard checklist for evaluating
systematic reviews, the Cochrane review was found to be deficient in terms
of trials not being included and omission of data, including subgroup
analyses. Deficiencies were consistently in the direction of supporting
the negative conclusions of the review. More such critical scrutiny of the
output of Cochrane Collaborative is thus in order.
At its website, the Cochrane Collaborative modestly describes itself
as “the gold standard in evidence-based healthcare.”
(http://www3.interscience.wiley.com/cgi-bin/mrwhome/106568753/HOME) In a
headline in the This Week in BMJ accompanying the JØrgensen et al. paper
admonished us to “Read industry supported drug reviews with caution.”
This reasonable caution should be expanded to all reviews, including those
of the Cochrane Collaborative.
1. JØrgensen AW, Hilden J, GØtzsche PC. Cochrane reviews compared
with industry supported meta-analyses and other meta-analyses of the same
drugs: systematic reviews. BMJ 2006:332:782-6.
2. Luborsky L, Diguer L, Seligman DA, Rosenthal R, Krause ED, Johnson
S, Halperin G, Bishop M, Berman JS, Schweizer E.The researcher's own
therapy allegiances: A "wild card" in comparisons of treatment efficacy
Clinical Psychology-Science and Practice. 1999 6 (1): 95-106.
3. Klein DF Flawed meta-analyses comparing psychotherapy with
pharmacotherapy.American Journal of Psychiatry, 2000. 157 (8): 1204-1211.
4. Parker G, Roy K, Eyers K Cognitive behavior therapy for
depression? Choose horses for courses American Journal of Psychiatry.
2003. 160 (5): 825-834.
5. Barbato A, D'Avanzo B. Marital therapy for depression. Cochrane
Database Of Systematic Reviews. 20006 (2): Art. No. CD004188.pub2.
6. Kraemer, H. C., Gardner, C., Brooks, J. O., & Yesavage, J. A.
Advantages of excluding underpowered studies in meta-analysis:
Inclusionists versus exclusionists viewpoints. Psychological Methods,1998.
7. Le Henanff A, Giraudeau B, Baron G, Ravaud P. Quality of reporting
of noninferiority and equivalence randomized trials JAMA-Journal of the
American Medical Association. 2006. 295(10): 1147-1151
8. Egger M, Smith GD, Phillips AN Meta-analysis: Principles and
procedures British Medical Journal. 1997. 315 (7121): 1533-1537.
9. Herxheimer A, Healy, D., Menkes,D.B. Antidepressants and Violence:
Problems at the Interface of Medicine and Law. PLoS Medicine 2006. 3,
(9), DOI: 10.1371/journal.pmed.0030372
10. Herxheimer A, Mintzes B Antidepressants and adverse effects in
young patients: uncovering the evidence. Canadian Medical Association
Journal. 2004. 170 (4): 487-489.
11.Herxheimer A, Mintzes B. SSRI treatment for under-18s - Response.
Canadian Medical Association Journal.2004. 170 (12): 1771-1772.
12. Bjordal JM, Lopes-Martins RAB, Klovning A Is quality control of
Cochrane reviews in controversial areas sufficient? Journal of Alternative
and Complementary Medicine. 2006. 12 (2): 181-183.
Competing interests: No competing interests