Press repeats journal’s hype over acupuncture
BMJ 2011; 343 doi: https://doi.org/10.1136/bmj.d4606 (Published 20 July 2011) Cite this as: BMJ 2011;343:d4606
All rapid responses
Rapid responses are electronic comments to the editor. They enable our users to debate issues raised in articles published on bmj.com. A rapid response is first posted online. If you need the URL (web address) of an individual response, simply click on the response headline and copy the URL from the browser window. A proportion of responses will, after editing, be published online and in the print journal as letters, which are indexed in PubMed. Rapid responses are not indexed in PubMed and they are not journal articles. The BMJ reserves the right to remove responses which are being wilfully misrepresented as published articles or when it is brought to our attention that a response spreads misinformation.
From March 2022, the word limit for rapid responses will be 600 words not including references and author details. We will no longer post responses that exceed this limit.
The word limit for letters selected from posted responses remains 300 words.
It is interesting that the CACTUS study debate is taking place in the
BMJ correspondence columns and blogs, but not the BJGP (which has not
published my letter on the study).
The authors say that "The trial used standard statistical methods
that underwent rigorous peer-review."
Unfortunately, the study's statistics were naively interpreted by the
authors, their rigorous peer reviewers, and the BJGP's editors.
The differences in outcome measures are small and the confidence
intervals wide. This suggests (before taking the risks of bias into
account) that the results are unlikely to be clinically important, even if
they are statistically significant.
For example, the primary outcome measure in CACTUS was the "measure
yourself medical outcome profile" (MYMOP) at 26 weeks. The mean difference
between the treated and untreated groups was only -0.6 on a 7 point scale,
and the 95% confidence interval was wide (-1.1 to 0.0). If the confidence
intervals had been shown in the graphs, it would have been clear just how
small and clinically unimportant the differences are.
Statistical significance means that the result is unlikely to have
occurred by chance. However, this does not mean that the result is likely
to be a specific effect of the intervention. It could also be caused by
bias. And, the onus is on authors, especially when results are
controversial, to rigorously assess the risks of bias.
CACTUS was an open (unblinded) trial, and the outcome measures were
subjective assessments by the patients. In this situation there are high
risks of several bias effects, and the size of the overall bias effect is
likely to be larger than the effect specific to acupuncture (1).
The untreated control group were likely to have resented not
receiving acupuncture, and therefore to negatively bias their assessments.
This would increase the difference between them and the acupuncture group,
thus making a harm look like a benefit!
To say that the statistician was blinded is a rhetorical trick to
give the appearance of rigor in the analysis.
The qualitative study regrettably missed an opportunity to document
the experience of those who had been denied acupuncture. This would have
provided useful information on the likely importance of the negative
placebo effects.
Until the advocates of acupuncture honestly and rigorously address
the risks of bias in their studies, science-based medicine will regard it
as no more than a particularly theatrical form of social grooming.
Reference
(1) Power M, Hopayian K. Exposing the evidence gap for complementary
and alternative medicine to be integrated into science-based medicine. J R
Soc Med. 2011 Apr;104(4):155-61. DOI 10.1258/jrsm.2011.100271
Competing interests: No competing interests
McCartney's BMJ article entitled 'Press repeats journal's hype over
acupuncture'(1)makes 'observations' about a published randomised pragmatic
trial which compares traditional acupuncture to usual care as a treatment
for people who consult frequently with medically unexplained symptoms(2).
The trial used standard statistical methods that underwent rigorous peer-
review. McCartney's article bases its arguments on a number of
inaccuracies (reproduced in "inverted commas" below) which BMJ readers
should be made aware of:
1. "The "wellbeing score" was better in the control group than in the
acupuncture group". This is untrue: an adjusted mean difference, in favour
of acupuncture, was seen with wellbeing (Wellbeing Questionnaire, W-BQ12):
4.4 (1.6 to 7.2) p=0.002 and this remained significant after missing
values were imputed (3.4 (0.5 to 6.3) p=0.02.
2. "The graphical information behind the paywall showed the
difference between the scores in the two groups over time. These were
almost identical." The scores in the two groups were presented in tabular
form with the results of statistical significance tests which showed a
statistical difference for the primary outcome Measure Yourself Medical
Outcome Profile in favour of acupuncture (adjusted mean difference:
acupuncture v control -0.6 (-1.1 to 0) P=0.05) and for the wellbeing
score, as above. The scores cannot therefore be said to be 'almost
identical'. In the longer version, graphs were provided for readers who
prefer to see findings depicted in this way.
3. "The "measure yourself medical outcome profile," is a self
administered questionnaire used mainly in alternative medicine.........
There were no blinded functional assessments." The Measure Yourself
Medical Outcome Profile (MYMOP) has been validated in several studies that
included conventional general practice patients (3,4) the first of which
was published in the BMJ in 19963,. MYMOP is an individualised outcome
questionnaire and is therefore particularly suited to study populations
who have a variety of chronic symptoms - such as these patients with
'medically unexplained symptoms'. This variety of symptoms, most of which
were primarily chronic pain, are best measured using a range of well
validated self-report questionnaires, as used in this study, and cannot be
adequately measured by 'functional assessments'.
4. "There was no sham acupuncture group, which is a big problem". A
pragmatic design, comparing to 'usual care' instead of a placebo or sham
intervention, is an accepted method of investigating complex non-
pharmaceutical interventions and is the appropriate design to 'measure
effectiveness - the benefit the treatment produces in routine clinical
practice.'(5). Our previous work, published in the BMJ in 2005 (6), has
led us to question whether 'needling' can be isolated from the other
aspects of the consultation without changing it dramatically (imagine
trying to isolate a set of physiotherapy exercises from the talking that
accompanies their delivery). The paper makes it clear that we are testing
the effect of adding a series of acupuncture consultations to usual care,
rather than testing the effect of needling alone, and hence our published
conclusion states 'The addition of 12 sessions of five-element acupuncture
to usual care resulted in improved health status and wellbeing ......'.
5. "The energy is the main thing I have noticed. You know, yeah, it's
marvellous!" This is a quote from the linked qualitative study and is
taken from one of the patient interviews. Rather than have this presented
as a meaningless remark, BMJ readers may prefer to be informed that an
analysis of this interview data indicated that many patients who were
treated with five element acupuncture not only perceived a range of
positive effects but also appeared to take on a more active role in
consultations and in self-care (7).
In our paper we acknowledged and discussed the limitations of a
pragmatic design and of blinding the statistician but not the patients. We
did this with reference to scholarly peer reviewed papers rather than
McCarthy's approach of referring readers to blog sites with names such as
'The Quackometer'. We believe the statistical findings of the randomised
trial, together with the qualitative analysis of the patients'
perspectives, provides doctors and patients with robust and useful
information for treatment decision making as well as providing a firm
basis for a future cost-effectiveness study.
References
1. McCartney, M. Press repeats journal's hype over acupuncture, BMJ
2011;343:d4606
2. Paterson C, Taylor R, Griffiths P, Britten N, Rugg S, Bridges J,
McCallum B, Kite G. Acupuncture for 'frequent attenders' with medically
unexplained symptoms: a randomised controlled trial (CACTUS Study). Br J
Gen Pract 2011;DOI: 10.3399/bjgp11X572689
3. Paterson C. Measuring outcome in primary care: a patient-generated
measure, MYMOP, compared to the SF-36 health survey. BMJ 1996;312:1016-20.
4. Paterson C, Langan C.E, McKaig G.A, Anderson G.D.H, Maclaine
G.D.H, Rose L.H. Assessing patient outcomes in acute exacerbations of
chronic bronchitis: the measure yourself medical outcome profile ( MYMOP),
medical outcomes study 6-item general health survey ( MOS-6) and EuroQol (
EQ-5D). Quality of Life Research 2000; 9: 521-527.
5. Roland M, Torgerson DJ. Understanding controlled trials: what are
pragmatic trials? BMJ 1998;316:285-286.
6. Paterson C,.Dieppe P. Characteristic and incidental (placebo)
effects in complex interventions such as acupuncture. BMJ 2005;330:1202-5.
7. Rugg S , Paterson C, Britten N, Bridges J, Griffiths P (in press).
Traditional Acupuncture for people with medically unexplained symptoms: a
longitudinal qualitative study of patients' experiences. Br J Gen Pract
2011; DOI: 10.3399/bjgp11X577972.
Competing interests: No competing interests
The fact that none of the authors of the paper or the editor of BJGP
have bothered to try and defend themselves speaks volumes.
Like many people I glanced at the report before throwing it away with
an incredulous guffaw. You bothered to look into it and refute it - in a
real journal. That last comment shows part of the problem with them
publishing, and promoting, such drivel. It makes you wonder whether
anything they publish is any good, and that should be a worry for all GPs.
Competing interests: No competing interests
The problem with acupuncture trials which have no sham component.
I thank Charlotte Paterson and her colleagues
for their reply.
The overreaching problem is that sham acupuncture has shown to be as
effective as 'real' acupuncture. This trial was not capable of discerning
the magnitude of the placebo effect because it did not contain a sham
acupuncture arm. This means that the press releases, editorial in the BJGP
and conclusions of the study are wrong because we cannot suggest that it
is the acupuncture which is making a difference (the size of which I think
is clinically insignificant and which I suspect we shall continue to
disagree on.) Sham trials are important where there is a large placebo
effect. This is important because unless we realise this we will remain
'stuck' with the idea acupuncture is important when it is not.
Regarding the benefit of acupuncture in their study, I think it would
be useful to look at the graphical information about the variations in the
various scores measured before and after acupuncture.
I apologise for stating that the wellbeing stores improved in the
control group more when they rose both in the control and acupuncture
group; perhaps the marginal difference in outcomes is expressed
graphically:
Statistical significance is not a replacement
or equivalent to clinical significance.
It is wrong to suggest that functional assessments cannot be useful
in measuring effectiveness of clinical interventions. These are
particularly important when there has been no blinding to the
intervention.
Many of the most robust and detailed peer reviews now take place
online post-publication rather than behind the closed doors of medical
journals. These have highlighted the flaws in papers when many journals
have neglected to do so. The failures of 'scholarly peer reviewed papers'
is to the detriment of science and, in the end, good patient care. I have
no qualms about quoting from freely available online sites such as the
Quackometer who construct useful peer reviews and I will continue to do
so. In the same spirit it would be useful if the peer reviews from the
BJGP could be made available. Perhaps we could compare them.
Yours sincerely
Margaret McCartney
Competing interests: No competing interests