Statistical interpretation can also bias research evidence

doi:10.1136/bmj.327.7417.752-a

BMJ 2003;327:752 (27 September), doi:10.1136/bmj.327.7417.752-a

Letter

Statistical interpretation can also bias research evidence

EDITOR—Kaptchuk discussed the effect of interpretive biason research evidence.¹ Let me add one more example. Studiesare designed to determine whether "a statistically significantdifference" exists between the outcomes of two alternative treatments.If no difference is discovered the temptation for authors isto conclude that the treatment under investigation is "justas good" as the gold standard. To make such a statement, thestudy needs to have adequate statistical power, ensuring thechance of a type II error (incorrectly accepting the null hypothesis)is sufficiently small.

Since power can generally be increased by enlarging the samplesize, it has become popular for researchers who do not havesufficient power to speculate in a way that makes the actualpower meaningless. For example, such a typical speculative statementmight read: "While the study failed to have sufficient powerto confirm the findings that the drugs were not different, hadthe sample size been increased from 10 to 180, then the powerwould have been sufficient to conclude no difference exists."In this way, the researcher implies that it's only a statisticalconvention is preventing him or her from stating that no differenceexists between the two drugs. In reality, had the sample sizebeen so increased, there is no guarantee as to what the researchersmay have found.

For those who cannot resist such hypothetical conclusions mayI suggest (tongue in cheek) you skip the study, examine onepatient, report the results, and speculate that whatever youfind could be of greatest statistical significance if only thestudy had been conducted with more people.

Lorne Basskin, president

Trinka Medical Education and Publications, 11100 Minneapolis Drive, Cooper City, FL 33026, USA lbasskin{at}hotmail.com

Competing interests: None declared.