When are randomised trials unnecessary? Picking signal from noise
BMJ 2007; 334 doi: https://doi.org/10.1136/bmj.39070.527986.68 (Published 15 February 2007) Cite this as: BMJ 2007;334:349
All rapid responses
Rapid responses are electronic comments to the editor. They enable our users to debate issues raised in articles published on bmj.com. A rapid response is first posted online. If you need the URL (web address) of an individual response, simply click on the response headline and copy the URL from the browser window. A proportion of responses will, after editing, be published online and in the print journal as letters, which are indexed in PubMed. Rapid responses are not indexed in PubMed and they are not journal articles. The BMJ reserves the right to remove responses which are being wilfully misrepresented as published articles or when it is brought to our attention that a response spreads misinformation.
From March 2022, the word limit for rapid responses will be 600 words not including references and author details. We will no longer post responses that exceed this limit.
The word limit for letters selected from posted responses remains 300 words.
We are glad to see the general agreement with the principles and
examples we set out. However we agree that further work and refinement are
needed if we are to avoid erroneous conclusions about non-randomised
evidence. Further careful examination of examples and the development of
methods will help us tease out which methods apply when. While some may
find the examples simple and plain “common-sense”, they are also important
bridges to more complex areas of evidence. We need to find the harmony
between uncommon sense and common sense (the sense that tells us the world
is flat, but also keeps us alive).
In this regard, the work of Aronson and the development of case-
crossover designs (mentioned by Hocine) are both important and clearly
closely related to the methods we set out. However as Goodman suggests,
though a case or case series or other non-randomised evidence may
sometimes be sufficient, careful Methods are still important if we are to
avoid being caught by Texas sharpshooters and other biases.
Finally like Polowetzky, a number of folk have expressed concerns
about the possible safety of the Mother's Kiss method. To date we have
found no problems reported in the case series, the case reports, and the
experience of folk we have spoken to who have used it, but cleary there is
a need for sufficient cumulative experience. This is analogous to the
safety element of phase IV pharmaco-epidemiological studies after
randomised trials. We plan to study this further, but would appreciate
hearing of anyone's experience, adverse or not, with the Mother's Kiss
method.
Competing interests:
None declared
Competing interests: No competing interests
Although efficacy of the "Mother's Kiss" technique for removal of a
foreign body lodged in a nostril may be demonstrated sufficiently by a
report of a case series in which it was successful in 15 out of 19
cases,and thereby, precludes futher randomized trials for demonstration of
treatment effect, no mention of safety is made.
The authors in fact go so far as to recommend the technique:
A search yields only one report of a case series, in which the
mother's kiss was successful in 15 out of 19 children. We think this is
sufficient evidence to recommend use in practice without randomised
trials.
On first blush, the technique does not seem obviously safe, as
compared to visualizing the object and removing it with forceps. It sounds
like performing the pulmonary portion of CPR on a living, breathing
person. Is it not possible that the increased airway pressure may not be
such a good idea?
Competing interests:
None declared
Competing interests: No competing interests
We very much agree with Glaziou et al [1] that in some situations
evidence from case series or non-randomized cohorts render randomised
trials unnecessary. The authors discuss a rate ratio obtained by comparing
the rate of progression between treated and untreated periods for a single
case. This is similar to the relative incidence estimated from case series
using the self-controlled case series approach [2].
The authors recommend their approach when effects are large, but
acknowledge that their approach may yield biased estimates in the presence
of time trends. Indeed such methods have been used inappropriately, as in
investigating a possible association between DTP immunisation and sudden
infant death syndrome (SIDS) (see Example 1 in [3]).
The self-controlled case series method allows for full adjustment for
underlying trends, most flexibly using a semi-parametric approach [4].
Furthermore, it controls implicitly for all fixed confounders and random
individual effects, whether measured or not. Thus, in this respect, the
method achieves a degree of confounder control usually only available in
randomized trials.
Under suitable conditions, the method can be used to evaluate
treatment effects of small and moderate size for both acute and
progressive conditions. It has been applied in a wide range of situations
in pharmaco-epidemiology, including MMR vaccine and autism and
antidepressants and hip fracture (see [5] for details), and in other
contexts such as bacterial resistance to antibiotics [6].
References
1. Glasziou P., Chalmers I., Rawlins M. and McCulloch P. When are
randomized trials necessary? Picking signal from noise. British Medical
Journal 334: 349-351.
2. Farrington, C.P., 1995. Relative incidence estimation from case
series for vaccine safety evaluation. Biometrics, 51 228-235.
3. Farrington, C.P., 2004. Control without separate controls:
Evaluation of vaccine safety using case-only methods. Vaccine, 22 2064-
2070.
4. Farrington, C.P. and Whitaker, H.J., 2006. Semiparametric analysis
of case series data (with Discussion). Journal of Royal Statistical
Society, Series C, 55 553-594.
5. Whitaker, H.J., Farrington, C.P., Spiessens, B. and Musonda, P.,
2006. Tutorial in Biostatistics: The self-controlled case series method.
Statistics in Medicine, 25 1768-1797.
6. Hocine M, Guillemot D, Tubert-Bitter P, Moreau T. 2005. Testing
independence between two Poisson-generated multinomial variables in case-
series and cohort studies. Statistics in Medicine, 24 4035-4044.
Competing interests:
None declared
Competing interests: No competing interests
Glasziou et al’s method of calculating rate ratios of progression
(stable unchanging condition before vs. change shortly after the
intervention) is appealing, but in applying it we need to be wary of a
‘Texas sharp shooter’ effect. This effect is usually associated in
epidemiology with the problem of interpreting apparent ‘clusters’ of
disease in space, where the geographical unit of analysis - a town, a
borough, a few streets, the top half of one street etc - may have been
chosen post hoc in such a way as to maximise the apparent density of cases
(the sharp shooter metaphor comes from a joke about a Texan firing bullets
into the wall of a barn and then drawing the targets around the bullet
holes as a demonstration of his shooting prowess).
There is the potential for an analogous problem when trying to
calculate rate ratios in the manner described in this article, although
here the sharp shooting is in time not in space. To take the authors’
example of the mother’s kiss, the time period used is 10 seconds which
gives a rate ratio of progression of 1440. Perhaps, however, the bead
dislodged after only 8 seconds – if a somewhat sharper shooter had used 8
seconds as a time frame, this would have given a rate ratio of 1440/0.8 =
1800. Alternatively, if the bead had taken 15 seconds to dislodge, the
GP, nurse and mother might still reasonably have felt that they should
take the credit for the this happy outcome. The authors’ might
correspondingly have stretched their time interval to 15 seconds
(associated rate ratio 960) or included two 10 second intervals, just as
they included 3 (rather than one) month in their portwine stain example
(associated rate ratio 720). The point is that one needs to make an a
priori decision about the post-intervention time frame you will use –
presumably based on the maximum length of time after the event during
which, if improvement occurs, you are prepared to attribute it to your
intervention.
Competing interests:
None declared
Competing interests: No competing interests
Sir,
Reading the article by Glasziou et al.[1], we might be forgiven for
believing that they had discovered some hitherto unknown method of causal
inference. Instead, of course, they have merely stumbled across the way in
which causes have been identified in everyday life and science throughout
history. [2]
The “Mother’s kiss” technique for removing a bead lodged in a nostril
is known to be an effective treatment not only because it has been shown
to work in case reports but also because it is grounded in elementary
principles of physics familiar to every child who has played with a pea-
shooter. It does not need statistical analysis. Yet, the authors - unable
to free themselves of the urge to season the data with a sprinkle of
relative risks or P-values - neglect the fact that the many examples they
provide of treatments with clearly observable effects are widely accepted
without the need for statistical tricks.
The obsession with both randomised controlled trials and the
statistical approach to causation has clouded the thinking of a generation
or more of medical researchers. So much so, that the common sense notion
of causation has been relegated to little more than an afterthought. And
this accounts for the dismissive approach to any data not derived from
randomised trials. Perhaps, after their damascene conversion, Glasziou et
al. will campaign for a change in the hierarchy of evidence in favour of
data from non-randomised sources.
References
[1] Glasziou P, Chalmers I, Rawlins M, McCulloch P. When are
randomised trials unnecessary? Picking signal from noise. BMJ 2007;334;349
-351.
[2] Penston J. Fiction and fantasy in medical research: the large-
scale randomised trial. The London Press. London, 2003.
Competing interests:
None declared
Competing interests: No competing interests
Aronson and Hauben recently described circumstances where anecdotal
reports can prove definitive adverse reactions without further formal
verification [1]. Glasziou and colleagues now outline circumstances where
even methodologically weak study designs can produce strong evidence
supporting a clinical intervention.
They refine and extend the concept of "all or none studies",
introduced in the reference book on evidence-based medicine [2]. All or
none studies are not explicitly mentioned in this paper but most
historical references to treatments with dramatic effects relate to
conditions with rapidly fatal outcome, like diabetic ketoacidosis. All
patients died before the treatment, insulin, became available and some
then survived on it, proving its effectiveness.
The authors remind us that quality of evidence is not an absolute
notion solely based on methodological criteria. Its assessment must be
adjusted with clinical knowledge of the condition under study.
1. Aronson JK, Hauben M. Anecdotes that provide definitive evidence.
BMJ 2006;333:1267-1269.
2. Straus SE, Richardson WS, Glasziou P. Evidence-Based Medicine: How
To Practice And Teach EBM. 3rd ed. Churchill Livingstone, 2005
Competing interests:
None declared
Competing interests: No competing interests
75 years ago: 24. December 1932. Streptococci kill, antibiotics help to survive!
Dear Sir,
only 75 years ago a miracle occurred.
"On 24th December, 1932, it was found that in an experiment begun on
20th December, 1932, all the controls had died, whereas all the mice which
had been given Prontosil were alive and well.
This was the basis of the discovery which was destined to bring
undreamed-of advances in chemotherapy." (1)
Now we even forgot the furor epidemicus of these streptococcal
diseases.
Everybody talks against antibiotics.
It is a shame.
Merry Christmas
Yours Friedrich Flachsbart
1. N. Svartz: Presentation Speech.
The Nobel Prize in Physiology or Medicine 1939.
http://nobelprize.org/nobel_prizes/medicine/laureates/1939/press.html
Competing interests:
None declared
Competing interests: No competing interests