Intended for healthcare professionals

Research

Cochrane reviews compared with industry supported meta-analyses and other meta-analyses of the same drugs: systematic review

BMJ 2006; 333 doi: https://doi.org/10.1136/bmj.38973.444699.0B (Published 12 October 2006) Cite this as: BMJ 2006;333:782

Journals must share the blame - word restrictions, not the funder, best explain these findings

Dear Editor

Whilst there is little doubt that some industry funded systematic
reviews use poor methodology and may misrepresent their findings,
Jørgensen and colleagues’ study has overestimated this potential bias and
misattributed differences in methodology and reporting to it [1].

First, the median quality score for the Cochrane reviews assessed was
7, whereas the scores for industry funding, undeclared funding and non-
profit/no funding journal articles were 2, 2 and 3 respectively. These
results are best explained by whether the review was published in The
Cochrane Library or in another journal, not by the source of financial
support. This links to word limits, which restrict reporting of
methodological detail, and apply to journal articles but not Cochrane
reviews. For example, the paper BMJ version of the industry funded review
of celecoxib for RA in their sample had 2211 words [2], the online BMJ
version had 3425 words, whilst the matching Cochrane review has 6002 words
[3].

Second, at least for the celecoxib reviews, the reliability of
unblinded quality assessments raises concern. For example, for assessment
of allocation concealment, the Cochrane review states “concealment
allocation was assessed and rated as A (blind randomisation), B (unclear
methods of randomisation), or C (quasi-randomisation)”. The industry
funded review stated “to assess the potential for bias we considered the
method of randomisation, concealment of allocation …”. The first was
deemed adequate by Jørgensen and colleagues, the second not. It is not
clear why – both say that this assessment was undertaken but neither
define adequate concealment.

Similarly, Jørgensen and colleagues state “no industry reviews made
reservations about their recommendations”. This is not consistent with the
discussion section of the BMJ celecoxib review where 1) the importance of
assessing serious GI events rather than endoscopically detected ulcers, 2)
the lack of data to reliably determine upper gastrointestinal safety
beyond six months, 3) the urgent need to assess cardiovascular safety of
COX 2 inhibitors, and 4) the need for more data to assess the impact of
concomitant aspirin therapy, are all raised as issues of concern.
Interestingly it was the BMJ editors who choose to delete paragraphs
expressing reservations 1) and 2) when preparing the abridged version for
the BMJ paper journal.

Blinded assessment would be possible if the study were repeated and
only included published articles constrained by the restrictions on length
imposed by traditional journals. This would allow a fairer comparison
between funding source and reporting to be made that controlled for these
word limits.

The BMJ celecoxib review was produced as an experimental
collaboration between industry and a respected academic unit with an
international track record in systematic review methodology and
substantial involvement in The Cochrane Collaboration. Industry provided
details of all trials undertaken, gave access to the full trial reports
and some funding. The academic unit produced a protocol which was agreed
by all, then undertook data extraction, data analysis and interpretation.
The BMJ paper was produced by the academic unit through a clause included
in the contract allowing freedom to publish, including presentation of all
unpublished trials and their findings. One author from industry was
included on the paper to recognize their contribution to the review, which
did not include any rights of veto over its content. We believe that this
collaboration produced a robust, transparent and methodologically sound
review. Having access to full study reports avoided problems experienced
by the Cochrane reviewers in extracting detail from abridged journal
articles, and is a move which should be applauded by those that campaign
for fuller access to information. This included providing the company
report for the CLASS trial, which gave substantially more detail than the
corresponding JAMA article [4].

Industry has rightly been criticized for its failures to provide
access to all available data [5]. Now, on the basis of very weak
evidence, it stands accused of bias and interference when it tries to
assist academic work by providing these documents. This will not
encourage campgains for greater openness. The assessment of the likelihood
of bias in systematic reviews, including Cochrane reviews, should always
be made from assessing the reports of methods and the completeness of
results of the review, and not prejudices about the organisations from
which they emanate - this is the ethos of EBM. The lesson from Jørgensen
and colleagues’ paper may be more that the opportunity to explain methods
in the detail afforded by the Cochrane review format should be available
for all systematic reviews.

[1] Jørgensen AW, Hilden J, Gøtzsche PC. Cochrane reviews compared
with industry supported meta-analyses and other meta-analyses of the same
drugs: systematic reviews. BMJ 2006:332:782.

[2] Deeks JJ, Smith LA, Bradley MD. Efficacy, tolerability, and upper
gastrointestinal safety of celecoxib for treatment of osteoarthritis and
rheumatoid arthritis: systematic review of randomised controlled trials
BMJ 2002;325:619.

[3] Garner SE, Fidan DD, Frankish RR, Judd MG, Shea BJ, Towheed TE,
Tugwell P, Wells G. Celecoxib for rheumatoid arthritis. Cochrane Database
of Systematic Reviews 2002, Issue 4. Art. No.: CD003831. DOI:
10.1002/14651858.CD003831.

[4] Silverstein FE, Faich G, Goldstein JL, Simon LS, Pincus T,
Whelton A, Makuch R, Eisen G, Agrawal NM, Stenson WF, Burr AM, Zhao WW,
Kent JD, Lefkowith JB, Verburg KM, Geis GS. Gastrointestinal toxicity with
celecoxib vs nonsteroidal anti-inflammatory drugs for osteoarthritis and
rheumatoid arthritis: the CLASS study: A randomized controlled trial.
Celecoxib Long-term Arthritis Safety Study. JAMA. 2000; 284:1247-55.

[5] House of Commons Health Committee. The Influence of the
Pharmaceutical Industry. Fourth Report of the Session 2004-2005. London:
Stationery Office, 2005.

Competing interests:
JJD has contributed to The Cochrane Collaboration for over a decade, is currently a member of the Steering Group of the Cochrane Collaboration and Treasurer. He has co-authored 14 Cochrane reviews, and is the lead editor of the Statistical Section of the Cochrane Handbook. The views expressed here are his own and not necessarily those of The Cochrane Collaboration. His previous unit (Centre for Statistics in Medicine, Oxford) received funding from Pfizer and Searle to undertake the review mentioned in [2] and JJD received consultancy fees from Pfizer from 2001-2. Both his previous and current units have received royalties from the BMJ on sales of reprints of reference [2].

Competing interests: No competing interests

19 October 2006
Jonathan J Deeks
Professor of Health Statistics
Department of Public Health and Epidemiology, University of Birmingham, Birmingham B15 2TT