Endgames Statistical Question

Confidence intervals and statistical significance

BMJ 2012; 344 doi: http://dx.doi.org/10.1136/bmj.e2238 (Published 28 March 2012) Cite this as: BMJ 2012;344:e2238
  1. Philip Sedgwick, senior lecturer in medical statistics
  1. 1Centre for Medical and Healthcare Education, St George’s, University of London, Tooting, London, UK
  1. p.sedgwick{at}sgul.ac.uk

Researchers evaluated the effect of percentage compliance with the Surviving Sepsis Campaign’s resuscitation and management bundles on mortality in patients admitted with severe sepsis to intensive care units in Asia. The resuscitation bundle delivered within six hours includes early blood cultures and antibiotics plus various aspects of early goal directed treatment for haemodynamic derangements. The management bundle delivered within 24 hours includes low dose steroids, drotrecogin alfa (activated), glucose control, and guidelines on ventilator support.1

In total, 1285 adult patients with severe sepsis were admitted to 150 intensive care units in July 2009. There were 713 survivors and 572 deaths. Compliance with the resuscitation bundle (six hours) was lower for non-survivors (4.2% (95% confidence interval 2.6% to 5.8%) v 10.4% (8.1% to 12.6%). Compliance with the management bundle (24 hours) was also lower for non-survivors (2.1% (0.9% to 3.3%) v 4.6% (3.1% to 6.2%)).

Which of the following statements, if any, regarding the comparison of non-survivors with survivors are true?

  • a) The difference in percentage compliance with the management bundle (24 hours) was not significant at the 5% level because the 95% confidence intervals for the two groups overlapped

  • b) The difference in percentage compliance with the resuscitation bundle (six hours) was significant at the 5% level because the 95% confidence intervals for the two groups did not overlap

Answers

Statement b is true, whereas a is false.

The percentage compliance with the management bundle (24 hours) for non-survivors was 2.1% (0.9% to 3.3%) compared with 4.6% (3.1% to 6.2%) for survivors. Although the 95% confidence intervals for the two groups overlapped, it cannot be inferred that the percentage difference between survivors and non-survivors was not significant at the 5% level (a is false). It is a common misinterpretation that overlapping 95% confidence intervals between groups imply a lack of significance at the 5% level. The researchers reported a significant difference (P=0.01) between groups in percentage compliance. Generally, when 95% confidence intervals for two sample percentages or means overlap, inferences cannot be made about the presence or absence of statistical significance at the 5% level.

The percentage compliance with the resuscitation bundle (six hours) for non-survivors was 4.2% (2.6% to 5.8%) compared with 10.4% (8.1% to 12.6%) for survivors. Because the 95% confidence intervals for the two groups did not overlap it can be inferred that the percentage difference between groups was statistically significant at the 5% level (b is true). The researchers reported a significant difference (P<0.001) between groups in percentage compliance. Generally, when the 95% confidence intervals for two sample percentages or means do not overlap, it can always be inferred that the two groups differ significantly at the 5% level.

When comparing the groups of non-survivors and survivors in percentage compliance with the resuscitation and management bundles, it would have been good practice and more informative to present a confidence interval for the percentage difference between groups for each bundle rather than for the individual group percentages. The 95% confidence interval for the percentage difference between groups would quantify the inaccuracy of the sample difference between groups as an estimate of the population difference. Furthermore, if the 95% confidence interval for the percentage difference did not include zero, then it could have been inferred that the difference between groups was significant at the 5% level.

Notes

Cite this as: BMJ 2012;344:e2238

Footnotes

  • Competing interests: None declared.

References