Absence of evidence is not evidence of absenceBMJ 2011; 342 doi: https://doi.org/10.1136/bmj.d3126 (Published 25 May 2011) Cite this as: BMJ 2011;342:d3126
- Philip Sedgwick, senior lecturer in medical statistics
- 1Section of Medical and Healthcare Education, St George’s, University of London, Tooting, London, UK
Last week’s question described a trial that assessed the effectiveness of topical intranasal corticosteroids in children aged 4-11 years who had bilateral otitis media with effusion.1 A randomised double blind, placebo controlled superiority trial was performed.2 The primary end point was proportion of children cured of bilateral otitis media with effusion in one or both ears at one month. In total, 105 children were allocated to receive nasal mometasone furoate 50 μg, given once daily into each nostril for three months, and 112 to placebo spray.
At one month, 40.6% of the topical steroid group and 44.9% of the placebo group were cured. The difference between the steroid and placebo groups in proportion cured was not statistically significant (difference favouring placebo 4.3%, 95% confidence interval −9.3% to 18.1%; P=0.55).
Which one of the following statements best describes the result of the statistical hypothesis test for comparison of steroid spray with placebo in the primary end point for the population of children studied?
a) No evidence exists of a difference in cure rates between the steroid and placebo sprays
b) The steroid spray was statistically inferior to the placebo spray
c) There is no difference in cure rates between the steroid and placebo sprays
d) The placebo spray was statistically superior to the steroid spray
Answer a is the best description. As described in last week’s question, the aim of the superiority trial was to establish whether one treatment was superior to the other in effectiveness. Superiority in cure rates between treatments was investigated using statistical hypothesis testing that incorporated a null and alternative hypothesis.1
The resulting P value for the statistical hypothesis test for the comparison of steroid spray with placebo in the primary end point was P=0.55. The difference in cure rates between the treatments was therefore not statistically significant at the 5% level of significance. The null hypothesis was not rejected in favour of the alternative, and it is concluded that for the population of children aged 4-11 years with bilateral otitis media with effusion there was no evidence of a difference between treatments in the primary outcome. This does not mean that in the population no difference existed between the effectiveness of the treatments. These are very different statements. Although the trial did not find a difference between treatments, this does not mean that one does not exist. The trial participants were a single sample from the population, and it is not obvious how representative they were. Another sample may give different results. This concept has led to the phrase, “Absence of evidence is not evidence of absence.”3
Even if a statistically significant difference in cure rates had been seen, under the alternative hypothesis it cannot be concluded that a difference definitively exists in the population, only that there was evidence of one. As a result of statistical hypothesis testing we can say only whether or not the trial results lend support to the alternative hypothesis. It is not possible to conclude that a difference definitively does or does not exist in the population on the basis of a single sample and statistical hypothesis test.
Cite this as: BMJ 2011;342:d3126
Competing interests: None declared.