Intended for healthcare professionals

Rapid response to:

Papers Research pointers

Survey of claims of no effect in abstracts of Cochrane reviews

BMJ 2003; 326 doi: (Published 01 March 2003) Cite this as: BMJ 2003;326:475

Rapid Response:

Misinterpretations of significant and non-significant results

Alderson & Chalmers present evidence that inappropriate claims
of no effect or difference occured in about a fifth of abstracts of
Cochrane reviews.1 Indeed it is encouraging if we realize that the
likelihood of this error decreased to the level of the type II error
(b), which is 1:5 with a power of the test (1-b) of 80%.

Nevertheless, the authors failed to emphasize that significant
difference may be misinterpreted as evidence of effect as pointed out by
Double.2 It should be remembered that the tendency to avoid false negative
results by decreasing type II error increases the risk of false positive
results (type I error). To ignore this, would be allowing an ineffective
drug onto the market, or condemning an exposure that actually is safe.

The risk of overestimating drug effect is closely related to the
fact that most clinical activities are aimed at showing that one agent or
method is better than another. In fact, reporting of clinical trials
appears to be biased toward an exaggeration of treatment effects.3
Further, the lack of emphasis on clinical importance has led to a tendency
to equate statistical significance with evidence of clinically important
effects. Conversely, increasing number of reports are aimed at showing
that the investigated entities are "equivalent".4

It is well to remember that claims of "similarity" or "difference"
are often made by tests of statistical significance that are often
misapplied or accompanied by methodologically flawed experimental design.
Confidence intervals provide a useful addition to significance tests.
Nevertheless, identifying the best evidence requires detailed appraisal
of several additional items.5 This point seems to be more important
than concerns for careless wording

Michal R Pijak, Consultant rheumatologist

Frantisek Gazdik, Research fellow

Department of Clinical Immunology, Institute of Preventive and
Clinical Medicine, 833 01 Bratislava, Slovakia

1. Anderson P, Chalmers I. Survey of claims of no effect in abstracts
of Cochrane reviews. BMJ 2003;326:475.

2. Double DB. Non-significant results are as valid as significant
findings. (1 march 2003)

3. Pocock SJ, Hughes MD, Lee RJ. Statistical problems in the
reporting of clinical trials. A survey of three medical journals. N Engl J
Med 1987;317:429-32.

4. Greene WL, Concato J, Feinstein AR. Claims of equivalence in
medical research: Are they supported by the evidence? Ann Intern Med

5. Barton S. Which clinical studies provide the best evidence? The
best RCT still trumps the best observational study. BMJ 2000;321:255-

Competing interests:  
Dr. Pijak has received speaker fees from Pharmacia and Fournier.

Competing interests: No competing interests

05 May 2003
Michal R Pijak
Consultant Rheumatologist
Frantisek Gazdik
Institute of Preventive and Clinical Medicine, Bratislava, Slovakia