What is the problem?

New interventions used to be compared with minimal or no treatment, so researchers were looking for and finding large treatment effects. Clear recommendations were made because the P values were usually small (eg, P<0.001). However, modern interventions are usually compared with the existing standard treatment, so that the effects are often expected to be smaller than before, and it is no longer as easy to get small P values. The cut-off used to indicate a real effect is widely taken as P=0.05 (called statistically significant). The problem is that although P=0.05 is an arbitrary figure, many researchers still adhere strictly to it when making conclusions about an intervention, and often use it as the sole basis for this. Researchers and journals sometimes conclude that there is no effect.

The P=0.05 cut-off was first proposed by R A Fisher in 1925 as being low enough to make decisions, and over time has become widely adopted. However, examining interventions with P values just above 0.05 is difficult, especially if the trial is unique. It is incorrect to regard, for example, a relative risk of 0.75 with …