Letters

Quoting intermediate analyses can only mislead

BMJ 1997; 314 doi: https://doi.org/10.1136/bmj.314.7098.1907a (Published 28 June 1997) Cite this as: BMJ 1997;314:1907
  1. J M Bland, Professor of medical statisticsa
  1. a St George's Hospital Medical School, London SW17 0RE

    Editor—Marion E T McMurdo and colleagues report a trial of exercise in relation to bone density and falls.1 They state that “The difference between the groups in the number of women falling during the whole two year period was not significant (P=0.158), but between 12 months and 18 months into the study the difference was significant (P=0.011).” If in a clinical trial we carry out repeated significance tests as data are accumulated, and if the null hypothesis is true, then the probability of a spurious significant difference is increased. This is why we avoid intermediate analyses except in specially designed sequential trials, or if they are done by separate data monitoring committees which keep their finding from the investigators. I sympathise with the authors, who saw their “significant” difference melt away as more data were collected, but quoting intermediate analyses to give more weight to non-significant findings can only mislead. If the authors were disappointed with their non-significant difference why did they not give a confidence interval instead?2 The difference in the proportions of women who had falls (exercise group minus control group) was -14 percentage points (95% confidence interval -34 to 5), and the ratio of these proportions (exercise group over control group) was 0.68 (0.38 to 1.18). Either way, the data suggest that, while at worst exercise could be associated with a small increase in falls, at best it could be associated with a substantial decrease.

    The report would also benefit from a confidence interval for the difference between the groups in table 1,1 which shows mean percentage change in bone density. Here separate confidence intervals are presented for the control and exercise groups, with a P value from a two sample t test. How much better it would have been to present the confidence interval for the difference between the means for the two treatment groups. It is this difference in which we are interested in a clinical trial, not the mean for each group separately.

    References

    1. 1.
    2. 2.
    View Abstract

    Sign in

    Log in through your institution

    Free trial

    Register for a free trial to thebmj.com to receive unlimited access to all content on thebmj.com for 14 days.
    Sign up for a free trial

    Subscribe