Users' guide to detecting misleading claims in clinical research reports
BMJ 2004; 329 doi: https://doi.org/10.1136/bmj.329.7474.1093 (Published 04 November 2004) Cite this as: BMJ 2004;329:1093
All rapid responses
Rapid responses are electronic comments to the editor. They enable our users to debate issues raised in articles published on bmj.com. A rapid response is first posted online. If you need the URL (web address) of an individual response, simply click on the response headline and copy the URL from the browser window. A proportion of responses will, after editing, be published online and in the print journal as letters, which are indexed in PubMed. Rapid responses are not indexed in PubMed and they are not journal articles. The BMJ reserves the right to remove responses which are being wilfully misrepresented as published articles or when it is brought to our attention that a response spreads misinformation.
From March 2022, the word limit for rapid responses will be 600 words not including references and author details. We will no longer post responses that exceed this limit.
The word limit for letters selected from posted responses remains 300 words.
The tone and scope of the rapid responses to our Users Guides confirm
the central assumption of our Guides: that there are emotional and
economic interests that affect the objectivity of science. We think that
these guides will help readers form their own opinions about the evidence.
While we appreciate all of the responses thus far, some require comment.
Roberts outlines the complexities of conflict of interest surrounding
the Cochrane Albumin review (the NHS funded the review group, the NHS is
the major producer of albumin in the UK, the NHS did not release
unpublished data to the reviewers, the NHS ultimately pays for albumin use
in the UK, and the NHS was embarrassed by the conclusions of the review) –
one wonders how this situation would have played out if a commercial
interest funded the review. Wilkes and Navickis, authors of one such
review and who alert readers of our effort to market our editorial
products through this paper, emphasize methodological differences between
the Cochrane review and theirs as the explanation for their different
conclusions. We recognize that the evidence that each review summarizes
was somewhat different. We also recognize that the studies that each
summarized were small, the results heterogeneous, and continue to believe
that this evidence did not warrant strong statements of either danger or
safety.
Penston pointed out an inconsistency between our title and our first
guide – how can one detect misleading claims if one is not exposed to the
Discussion section where those claims are often made? Perhaps our original
title (Guides for clinicians to defend themselves from misleading claims…)
was more appropriate. Others have seen in our guides validation of their
concerns about specific studies and scientists’ behaviors, and
justification for a broader evaluation of research protocols in ethics
boards. It seems that the process our guides are intended to foster -
frank dialog on more effective ways for decision makers to deal with data
that comes wrapped in conflict – is ongoing.
Competing interests:
VMM, RJ, HJS, PJD, and GHG are associate editors of the ACP Journal Club and Evidence-Based Medicine. JLB and RJ edit an evidence based medicine journal in Poland. MB is editor of the evidence based orthopaedic trauma section in the Journal of Orthopaedic Trauma.
Competing interests: No competing interests
Montori and colleagues cite the systematic review of albumin
administration in critically ill patients that was conducted by the
Cochrane Injuries Group as an example of a review that provided results
that were consistent with the interests of the funders of the review. They
claim that our review was funded by the NHS who they say pays for albumin
use, and go on to claim that our assertion that albumin should only be
used in the context of rigorously conducted randomised controlled trials
was therefore in the interest of the funders. This is wrong on both
counts. The systematic review of human albumin administration was not
funded by the NHS. Indeed, none of the authors received any external
funding for the work undertaken on this review. In common with most UK
based Cochrane Collaborative Review Groups, the infrastructure of the
Cochrane Injuries Group is supported by the NHS Research and Development
Programme but the decision to conduct this review was entirely independent
of the NHS who were in fact deeply embarrassed by its findings. The NHS
was embarrassed by the results because the NHS is the largest manufacturer
of human albumin in England and they had declined to provide the authors
of the review with details of any unpublished trials. This was covered
extensively in the British media. However, it is not surprising that
Montori and colleagues have rehearsed this slur on the Cochrane Injuries
Group Albumin review because considerable sums of money have been spent by
the Plasma Proteins Therapeutic Association on just this activity. The
systematic review conducted by Wilkes and Navickis from Hygeia Associates
in California was indeed funded by the Plasma Proteins Therapeutic
Association who have now published nine papers promoting the use of
albumin many of which were co-authored by leading university based
intensive care physicians. Although the Wilkes review was presented as a
response to the Cochrane review, their review used different inclusion
criteria, a fact also overlooked by Montori and colleagues, in which case
it is not surprising that the results were different. As Montori and
colleagues point out there is compelling evidence that funding influences
the conclusions from research but in this case the links are somewhat more
complicated than Montori and colleagues appreciate.
Competing interests:
None declared
Competing interests: No competing interests
Another example of a meta-analysis coming up with a wrong result is
the Salpeter meta-analysis on HRT mortality.1 After 40 years of
unjustified claims of health benefits major randomised, double-blind
trials of HRT including the US WHI studies of combined and oestrogen-only
HRT, the Million Women Study, and the HABITS and Stockholm studies of HRT
in breast cancer patients, were terminated early. Increased health risks
include breast and endometrial cancer, myocardial infarction, stroke,
venous thrombosis, and dementia and no overall improvement in the quality
of life. How then has Salpeter managed to claim HRT reduces mortality?
Not one of the 30 short-term randomised controlled trials pooled for
the meta-analysis produced a statistically significant effect of HRT on
mortality. The claimed reduction in mortality for women with a mean age
under 60 was achieved by statistical manipulations. Trials of sick women
with most deaths during the study follow-up were given disproportionate
weighting. For example one small open-label study of 68 controls / 62 HRT
treated ovarian cancer “survivors” was allocated a weight of 41% because
the majority of the women, 41 and 32 women respectively, died during the
study follow-up period. The fact that many studies have found HRT
increases in the risk of fatal ovarian cancer was not mentioned. In
contrast, the largest study of 701 previously healthy women found a 75%
increased mortality risk with HRT use but was weighted at only 2.3%
If only double-blind randomised controls studies with more than a
hundred HRT- treated younger women had been pooled, a completely different
result would be achieved as seven of the deaths occurred in 1450 treated
women and none in 479 control women.
This seems to be an attempt to validate the myth that hormones are
safer for younger women. Breast cancer incidence increases match the
changes, mostly huge increases, in hormone use in hormone-taking countries
since the 1960s. Women diagnosed with breast cancer are still likely to
die of the disease 20 years after diagnosis. Most modern epidemiological
studies fail to clarify which women have never taken hormones, and
therefore underestimate the real mortality risks.
1 Salpeter SR, Walsh JME, Greyber E, Ormiston TM, Salpeter EE.
Mortality associated with hormone replacement therapy in younger and older
women. A meta-analysis. J Gen Intern Med 2004; 19: 791-804.
Competing interests:
None declared
Competing interests: No competing interests
Dear Editor
I read with interest Montori et al’s paper on ‘Users' guide to
detecting misleading claims in clinical research reports’.1 The advice to
read only the methods and results section and to bypass the discussion
section, though may be ideal did not seem to be the sine qua non. Penston
J in his response, has already objected to this ‘radical proposal’, from a
reader point of view.
The implications of this to some authors of clinical research papers
are unnerving. Every research paper that is produced is not perfect.
Authors are more often than not fully aware of the bias and pitfalls that
occur in their paper, some of which are inevitable. The discussion
section gives the researcher an opportunity to address these issues and
explain their obligatory position. The researcher has spent valuable time
and resources on the work and is versatile with the methodology and
results. In contrast to this, a reader who scans the methods and results
section can easily overlook important information as critical appraisal of
a clinical trial is a truly absorbing and daunting task. Whilst it may be
true that some research papers can mislead the reader towards believing
the result, it is debatable if the mentioned tip warrants a number one
slot in the users’ guide.
1 Victor M Montori, Roman Jaeschke, Holger J Schünemann, et al.
Users' guide to detecting misleading claims in clinical research reports.
BMJ 2004;329:1093-1096.
Competing interests:
None declared
Competing interests: No competing interests
Sir,
The paper by Montori and his colleagues cites the Cochrane albumin
review (1) in the table. This review is an egregious example of a review
which made a misleading claim - in this case a 6 per cent additional
mortality when albumin was compared with saline in a meta-analysis.
The users` guide given by Montori is designed for the assessment of a
single randomised controlled trial (RCT); when many RCTs are amalgamated
into a review the compilers demand a degree of trust from those who use
their review for guidance. In dealing with a complex diverse subject,
such as critical illness, no ordinary reader, editor, or peer reviewer,
will examine critically all the RCTs from which the review`s authors`
conclusions were derived.
In the Cochrane albumin review 7 RCTs were adduced to support the
mistaken contention that albumin, used in the treatment of hypovolaemia,
would result in one additional death for every 17 patients when compared
with saline. Cogent criticism of each of these RCTs was documented (2)
and the author of the review did not answer any of them (3).
Reviews such as this are normally assessed by the Centre for Research
and Development (CRD) whose aim is to raise the general standard of
reviews in the NHS. The comment of the CRD on this review was "Cochrane
reviews are of a high standard and are not evaluated by a CRD reviewer".
It has taken 6 years for the SAFE study (4) to disprove the erroneous
allegation of the albumin review and so far there has been no retraction
by any member of the Cochrane Collaboration.
Peter J Horsey,
The Old Manor,
Ashley, Stockbridge,
Hampshire SO20 6RH.
UK.
Tel 01794 388 256
References
1 Cochrane Injuries Group Albumin Revuers.
Human albumin administration in critically ill patients: systematic review
of randomised controlled trials. BMJ 1998; 317:235-40.
2 Horsey P Albumin and hypovolaemia: is the Cochrane evidence to be
trusted? Lancet 2002; 359:70-72.
3 Roberts I Albumin and hypovolaemia: time to move on and generate
new evidence. Lancet 2002; 359: 72-73.
4 Saline versus albumin study evaluation.
A comparison of saline and albumin for fluid resuscitation in the
intensive care unit. N Engl J Med 2004; 350:2247-56.
Competing interests:
None declared
Competing interests: No competing interests
Randomised studies which have failed to consider the confounding
influences of meme mutations and the branching meme complexes that have
evolved from them. The study of the relative benefits of balanced salt
solutions and albumin in the resucitation of patients is a good example
(1). In this case a beneficial effect of albumin might have been missed
for these very reasons (2).
The prostacyclin study published in the NEJM (3), addressed in this
eLetter to the CMAJ, is a example of what I believe to have been
unappreciated risks in the best designed and most acclaimed of prospective
randomised studis in the critically ill.
Evidence-informed medicine (4)would indeed seem to be a more
appropriate term than evidence-based medicine and certainly preferable to
meme-BM a term which by definition would have to include memes and meme
complexes that have evolved from mutated memes.
1. Misleading claims about albumin
Mahlon M. Wilkes, et al. (10 November 2004) .
2. Might saline have caused unappreciated harm in this study?
Richard G Fiddian-Green
eCMAJ, 14 Aug 2004 eLetter re: Roy Ilan and Robert A. Fowler
Should we use albumin or saline for fluid resuscitation of critically ill
patients?
CMAJ 2004; 171: 232
3. Bihari D, Smithies M, Gimson A, Tinker J. The effects of
vasodilation with prostacyclin on oxygen delivery and uptake in critically
ill patients.
N Engl J Med. 1987 Aug 13;317(7):397-403.
4. Evidence-Informed Practice not Meme-BM
Paul P Glasziou (4 November 2004) eLetter re: John Gabbay and Andrée le
May
Evidence based guidelines or collectively constructed "mindlines?"
Ethnographic study of knowledge management in primary care
BMJ 2004; 329: 1013-0
Competing interests:
None declared
Competing interests: No competing interests
Sir,
In their table and narrative text Montori et al make misleading
claims regarding the meta-analyses of the Cochrane group [1] indicating
excess mortality due to albumin administration and of Wilkes and Navickis
[2] detecting no difference in mortality. They argue that the two meta-
analyses generated similar results but differed only in "spin" by the
authors. The premise is factually erroneous. The differences between the
two meta-analyses were both large (51%) and statistically significant, as
we have elsewhere detailed [3]. Contrary to the claim of Montori et al
that the trials included in both meta-analyses were heterogeneous, no
significant heterogeneity could be detected in either meta-analysis by
standard statistical test.
The table misleadingly suggests approximate parity in the scope of
the two meta-analyses (42 vs 31 trials included) and thus equal claim to
validity. The total of 42 included by Wilkes and Navickis was for only
those trials with one or more deaths that contributed to the calculation
of relative risk, whereas the total in the table for the Cochrane meta-
analysis included trials with no deaths. The apples-to-apples comparison
would be 42 trials to 24. It should also be noted that Wilkes and Navickis
included all 24 trials of the Cochrane meta-analysis plus an additional 18
trials. Furthermore, the median size of the trials included by Wilkes and
Navickis was much larger (56 vs 36 patients per trial). Consequently, the
total weight of meta-analytic evidence assembled by Wilkes and Navickis
was approximately three times that of the Cochrane group [4]. Thus, the
disparate results were due to reliance by the Cochrane group on a small,
biased subset of the relevant evidence.
Left uncited was decisive extrinsic evidence that the results reached
by Wilkes and Navickis were correct and those of the Cochrane group
incorrect. The blinded randomized Saline versus Albumin Fluid Evaluation
(SAFE) trial demonstrated no excess mortality due to albumin among nearly
7,000 hypovolemic patients [5].
Albumin is, of course, used only as the primary example for the theme
that bias due to financial interest is everywhere palpable, and that
readers need help sorting it all out. Such help, according to Montori et
al, is to be found in the pages of unbiased publications such as ACP
Journal Club and Evidence-Based Medicine, of which they are editors. Also
recommended is a book edited by one of the co-authors. Most of the
subsections in their paper begin with the word "beware". We would add:
beware those who claim to be unbiased and have journals and books to sell.
Mahlon M. Wilkes and Roberta J. Navickis
1. Cochrane Injuries Group Albumin Reviewers. Human albumin
administration in critically ill patients: systematic review of randomised
controlled trials. BMJ 1998;317:235-40.
2. Wilkes MM, Navickis RJ. Patient survival after human albumin
administration: a meta-analysis of randomized, controlled trials. Ann
Intern Med 2001;135:149-164.
3. Wilkes MM, Navickis RJ. Colloid use in the critically ill. Ann
Intern Med 2002;137:370-1; discussion 370-1.
4. Wilkes MM, Navickis RJ. Does albumin infusion affect survival?
Review of meta-analytic findings. In: Vincent J-L, editor. 2002 Yearbook
of Intensive Care and Emergency Medicine. Berlin: Springer-Verlag,
2002:454-64.
5. SAFE Study Investigators. A comparison of albumin and saline for
fluid resuscitation in the intensive care unit. N Engl J Med 2004;350:2247
-56.
Competing interests:
None declared
Competing interests: No competing interests
This article is a neat and timely warning about the need for rapid
readers bias assessment before change in practice.
Meanwhile the commercialisation of all that is medical and academic
continues: I received this in my email in box today:
________________________________________
From: owner-smd-all-staff@qmul.ac.uk [mailto:owner-smd-all-
staff@qmul.ac.uk]
Sent: 09 November 2004 11:12
To: smd-staff mailing list
Subject: Reminder: BBSRC Workshop
The School of Medicine and Dentistry together with the Innovation and
Enterprise Unit is hosting a workshop on:
Commercialising Academic Research (sponsored by BBSRC)
on: Wednesday 24th November 2004, 2 - 4pm
at: Paul Garrod Lecture Theatre, Robin Brook Centre, St
Bartholomew's
Hospital
The programme will cover issues including: Managing your intellectual
property; Commercialising academic research; The patent process;
Licensing;
Spin-out companies; Funding opportunities; QM's IP policy
NEWS!!
LEAP 2005 is Simfonec's annual £50k business plan competition which
provides
aspiring entrepreneurs with a great opportunity to turn their ideas into
successful business ventures. (See www.leap2005.com) This competition is
open to both QMUL academics and students. Come to the workshop to find
out
more!
ALL WELCOME
Yours faithfully
Anna Livingstone
Competing interests:
none
Competing interests: No competing interests
Sir,
Montori et al. draw attention to misleading claims in published
clinical trials. [1] In this regard, the authors’ views are to be
welcomed. Their examples of this practice, as well as those documented
elsewhere, [2,3] suggest that it is not uncommon. Much of their advice –
including the encouragement to use independently produced abstracts,
together with the cautions concerning faulty comparators, composite
endpoints, small treatment effects and subgroup analyses - seem to be
entirely reasonable. There are, though, objections to their more radical
proposal.
Montori et al. recommend that only the methods and results sections
should be read, while the remainder of the paper – and, in particular, the
discussion – should be ignored. Yet, anyone capable of critically
appraising a clinical trial solely on the basis of the methods and results
sections – a challenging task, given the complexity of publications
related to clinical research - is unlikely to be fooled by misleading
claims in the discussion.
The advice to omit the discussion section, however, raises further
problems. The title of the paper by Montori et al. is: “Users’ guide to
detecting misleading claims in clinical research reports”. [1] But how are
these misleading claims to be identified if they are present in the
discussion and yet this section is to be omitted? There appears to be some
confusion. But, more importantly, if the parts of the paper containing the
misleading claims were to be omitted, then the discerning reader would
lose the opportunity of witnessing the conflict between the results of the
study and the unwarranted conclusions. This has implications beyond the
recognition of unjustifiable claims. If the researchers involved in a
clinical trial are willing to disseminate misleading claims, then we
should surely question their integrity. But if their integrity is under
suspicion, then all aspects of the clinical trial – including the contents
of the methods and results sections – are also under suspicion. Misleading
claims should be identified and then broadcast loudly throughout the
medical research community, for they signal doubts about the rest of the
study.
As a footnote to these remarks, I would add that much of the problem
of misleading claims has its origin in the flawed nature of large-scale
randomised trials. The complex and opaque data, the trivial treatment
effects and the absence of any means to verify the findings are ideal
conditions for those intent on manipulating the results for their own
advantage. There is, of course, a simply remedy. In future, when reading
the latest clinical trial, ignore everything except the number of patients
recruited to the study. If this is large, then go no further – the study
is unlikely to be of any value to your patients. [3]
[1] Montori VM, Jaeschke R, Schunemann HJ, et al. User’s guide to
detecting misleading claims in clinical research reports. BMJ
2004;329;1093-6.
[2] Wennberg R & Zimmermann C. The PROGRESS trial three years later:
time for a balanced report of effectiveness. BMJ 2004;329;968-70.
[3] Penston J. Ficition and Fantasy in Medical Research: the Large-Scale
Randomised Trial. The London Press. London, 2003. Chap V.
Competing interests:
None declared
Competing interests: No competing interests
No economic interest?
Roberts characterizes as a "slur" the idea that the results of the
Cochrane albumin meta-analysis were aligned with the economic interests of
the NHS, which provided funding for the meta-analysis in the form of
infrastructure support. This denial is contrary to a 1999 presentation on
the topic of securing funding for Cochrane reviews, in which Iain
Chalmers, the head of the UK Cochrane Collaboration at that time, stated:
"Estimates suggest that annual savings to the NHS resulting from
reduced use of human albumin solution alone are three times greater than
the total investment in Cochrane work in the UK" [1].
To this day the Cochrane Collaboration continues to make known the
savings on albumin costs effectuated by their meta-analysis
(http://www.cochranechildhealth.ualberta.ca/pdf%20n%20ppt%20files/may2_pr...).
The Cochrane albumin meta-analysis has been controverted both by a
more comprehensive meta-analysis of Wilkes and Navickis [2] and the large-
scale SAFE randomized trial [3]. Roberts contends that the discrepancies
between the two meta-analyses can be ascribed to different inclusion
criteria but makes no mention of the SAFE trial results. In any case, we
dispute the claimed explanation for the disparities between the two meta-
analyses. In reality the contrasts in inclusion criteria were minor and
their impact nearly nihil. Specifically, the differences were the
inclusion of paracentesis and hemodilution trials by Wilkes and Navickis
but not the Cochrane investigators. If the single included paracentesis
trial [4] and single included hemodilution trial [5] are excluded from the
Wilkes and Navickis meta-analysis, the pooled relative risk for death is
1.09 (CI, 0.93-1.27) compared with 1.10 (CI, 0.95-1.28) without the
exclusions. The true reason for the discordant results was the reliance by
the Cochrane investigators on a small, biased subset of the pertinent
evidence.
Mahlon M. Wilkes and Roberta J. Navickis
1, Chalmers I: Funding support for ensuring that Cochrane Reviews are
up to date and correct. Presented at the 7th Annual Cochrane Colloquium,
Rome, October 9, 1999
2. Wilkes MM, Navickis RJ: Patient survival after human albumin
administration: a meta-analysis of randomized, controlled trials. Ann
Intern Med 135:149-164, 2001
3. SAFE Study Investigators: A comparison of albumin and saline for
fluid resuscitation in the intensive care unit. N Engl J Med 350:2247-
2256, 2004
4. Ginès P, Titó L, Arroyo V, Planas R, Panés J, Viver J, Torres M,
Humbert P, Rimola A, Llach J, Badalamenti S, Jiménez W, Gaya J, Rodés J:
Randomized comparative study of therapeutic paracentesis with and without
intravenous albumin in cirrhosis. Gastroenterology 94:1493-1502, 1988
5. Hallowell P, Bland JH, Dalton BC, Erdmann AJd, Lappas DG, Laver
MB, Philbin D, Thomas S, Lowenstein E: The effect of hemodilution with
albumin or Ringer's lactate on water balance and blood use in open-heart
surgery. Ann Thorac Surg 25:22-29, 1978
Competing interests:
None declared
Competing interests: No competing interests