Survey of claims of no effect in abstracts of Cochrane reviews
BMJ 2003; 326 doi: https://doi.org/10.1136/bmj.326.7387.475 (Published 01 March 2003) Cite this as: BMJ 2003;326:475All rapid responses
Rapid responses are electronic comments to the editor. They enable our users to debate issues raised in articles published on bmj.com. A rapid response is first posted online. If you need the URL (web address) of an individual response, simply click on the response headline and copy the URL from the browser window. A proportion of responses will, after editing, be published online and in the print journal as letters, which are indexed in PubMed. Rapid responses are not indexed in PubMed and they are not journal articles. The BMJ reserves the right to remove responses which are being wilfully misrepresented as published articles or when it is brought to our attention that a response spreads misinformation.
From March 2022, the word limit for rapid responses will be 600 words not including references and author details. We will no longer post responses that exceed this limit.
The word limit for letters selected from posted responses remains 300 words.
Alderson and Chalmers are rightly critical of those who make
inappropriate claims (1), but they are themselves guilty of this. Compare
these statements: “It is never correct to claim that treatments have no
effect or that there is no difference in the effects of treatments”, and
“Absence of evidence is not evidence of absence”. The first is the opening
sentence of their paper, the second is the title of a paper by Altman and
Bland that they cite in its support (2). The two are mutually exclusive,
because if the first were true then there could never be “evidence of
absence”.
In describing their method, Alderson and Chalmers specify that they
only classified statements as claiming no effect or no difference if they
were without qualification about clinical or statistical significance.
While this could have been described more clearly, it is reassuring, as it
shows that they do in fact accept that it is possible to have evidence of
absence, even if this contradicts their eye-catching but grossly
overstated first sentence.
Clearly, one can have evidence for the absence of an effect or of a
difference. If enough large, well-designed studies were to show that a
medical treatment or an exposure were unassociated with outcome, this
would be as clear as our knowledge about cigarettes and lung cancer, or
statins and heart disease. In practice, it would be unlikely for this to
happen where an association is thought not to exist, as hypotheses
suggesting the absence of an effect do not generate large research
programmes, but this does not affect the principle.
1. Alderson P, Chalmers I. Survey of claims of no effect in abstracts
of Cochrane reviews. BMJ 2003;326:475.
2. Altman DG, Bland JM. Absence of evidence is not evidence of
absence. BMJ 1995;311:485.
Competing interests:
None declared
Competing interests: No competing interests
Alderson & Chalmers state that ""It is never correct to claim that treatments have no effect".1 What nonsense! Of course, certain treatments are ineffective. What is more difficult is to know whether the claim is true from clinical trial evidence, as the probability of a type II error is unknown.
The limitations of statistical hypothesis testing mean that the probability of an inferential error is known if one is rejecting the null hypothesis but not if one is accepting it.2 The conclusion that a difference exists is therefore made on a clearer basis in placebo controlled trials. Equivalence trials are rarely large enough to give the same degree of certainty, and the practical difficulties of obtaining sufficiently large sample sizes are inhibiting. Alderson & Chalmers may therefore be correct to express caution about claims for no effect or difference, but they should not seek to impose a bias of language in favour of positive treatment effects. A positive result demonstrating effectiveness by rejecting the null hypothesis may be incorrect because it is a type I error. Should all Cochrane reviews therefore make clear that "a significant difference was detected" in such circumstances rather than conclude that there was an effect? The semantics may be ambiguous,3 but I think we should know when we read such statements that they are made within the limits of hypothesis testing. We can look for the confidence limits if we wish.
It is a pity if the Cochrane Collaboration seeks to impose a hegemony in favour of significant rather than non-significant clinical trial results. The Collaboration has done much to improve the quality ofsystematic reviews and the results appear to be less prone to bias.4 Moreover, it has highlighted methodological bias, such as allocation concealment.5 Too much of the data in clinical trials is interpreted as favourable to the intervention because of the methodological failure to eliminate bias. We must be allowed to make sceptical statements about effectiveness. It is true that trial sizes are often inadequate for the interpretation of non-significant results, but nonetheless non-significance may reflect a real lack of effect or difference.
- Alderson P. Chalmers I. Survey of claims of no effect in abstracts of Cochrane revies. BMJ 2003; 326: 475 [Full text]
- Rozeboom AW. The fallacy of the null-hypothesis significance test. Psychol Bull 1960; 57: 416-28.
- Parker NR. Semantics. http://bmj.com/cgi/eletters/326/7387/475#30039 (28 February 2003)
- Egger M, Davey Smith G, Schneider M, Minder M.Bias in meta-analysis detected by a simple, graphical test. BMJ 1997;315:629-634 [Full text]
- Schultz K. Randomised trials, human nature, and reporting guidelines. Lancet 1996; 348: 596-98
Competing interests:
None declared
Competing interests: No competing interests
The difficulty with the semantics of the topic are illustrated in the
article by the statement "no effect or difference had been shown". The
statement is ambiguous.
It may mean a failure to show a difference,
or it may mean an assertion that there is no difference, and that
fact has been demonstrated.
In other words, is it a failure to demonstrate, or a demonstration of
failure?
Competing interests:
None declared
Competing interests: No competing interests
Misinterpretations of significant and non-significant results
Alderson & Chalmers present evidence that inappropriate claims
of no effect or difference occured in about a fifth of abstracts of
Cochrane reviews.1 Indeed it is encouraging if we realize that the
likelihood of this error decreased to the level of the type II error
(b), which is 1:5 with a power of the test (1-b) of 80%.
Nevertheless, the authors failed to emphasize that significant
difference may be misinterpreted as evidence of effect as pointed out by
Double.2 It should be remembered that the tendency to avoid false negative
results by decreasing type II error increases the risk of false positive
results (type I error). To ignore this, would be allowing an ineffective
drug onto the market, or condemning an exposure that actually is safe.
The risk of overestimating drug effect is closely related to the
fact that most clinical activities are aimed at showing that one agent or
method is better than another. In fact, reporting of clinical trials
appears to be biased toward an exaggeration of treatment effects.3
Further, the lack of emphasis on clinical importance has led to a tendency
to equate statistical significance with evidence of clinically important
effects. Conversely, increasing number of reports are aimed at showing
that the investigated entities are "equivalent".4
It is well to remember that claims of "similarity" or "difference"
are often made by tests of statistical significance that are often
misapplied or accompanied by methodologically flawed experimental design.
Confidence intervals provide a useful addition to significance tests.
Nevertheless, identifying the best evidence requires detailed appraisal
of several additional items.5 This point seems to be more important
than concerns for careless wording
Michal R Pijak, Consultant rheumatologist
pijak@upkm.sk
Frantisek Gazdik, Research fellow
Department of Clinical Immunology, Institute of Preventive and
Clinical Medicine, 833 01 Bratislava, Slovakia
1. Anderson P, Chalmers I. Survey of claims of no effect in abstracts
of Cochrane reviews. BMJ 2003;326:475.
2. Double DB. Non-significant results are as valid as significant
findings. http://bmj.com/cgi/eletters/326/7387/475#30078 (1 march 2003)
3. Pocock SJ, Hughes MD, Lee RJ. Statistical problems in the
reporting of clinical trials. A survey of three medical journals. N Engl J
Med 1987;317:429-32.
4. Greene WL, Concato J, Feinstein AR. Claims of equivalence in
medical research: Are they supported by the evidence? Ann Intern Med
2000;132:715-22.
5. Barton S. Which clinical studies provide the best evidence? The
best RCT still trumps the best observational study. BMJ 2000;321:255-
256.
Competing interests:
Dr. Pijak has received speaker fees from Pharmacia and Fournier.
Competing interests: No competing interests