Medical researchers, the executives of pharmaceutical companies and
members of special interest groups will be dancing in the streets. The
paper by Hackshaw and Kirkwood [1] is nothing but a license to make
unfounded claims about the efficacy of treatment from the results of
randomised trials. Data manipulation is already rife in the research
literature [2] and the proposed re-interpretation of the statistical
analysis will only make matters worse.
The authors' stated aim is to avoid the situation in which the
results of a study are ignored because the data fail to achieve
conventional statistical significance. "If a clinically important effect
is observed with a P-value of just above 0.05 (or an upper or lower
confidence limit close to the no effect value) it is incorrect to conclude
no effect and not consider further what is likely to be an effective
intervention..." [1] Thus, provided that there is a "clinically important"
effect - an arbitrary judgment - and that the P-value or confidence
interval is borderline - another arbitrary judgement - then we should
claim that there is evidence of an effect. But the authors want it both
ways: they criticise the use of the cut-off level for the P-value at 0.05
while, at the same time, they propose changes based on equally arbitrary
criteria.
It would be naive to believe that this re-interpretation will not be
seized upon by those with a vested interest in the outcome of medical
research in order to distort the findings, just as it would be naive to
believe that the authors' call for the use of "moderate words" would lead
to anything but an abuse of language.
Statistics-based research in medicine is fundamentally flawed [2] and
attempting to repair individual defects is a waste of time. We should stop
pretending that large RCTs and epidemiological studies produce anything
that is reliable or valuable.
Rapid Response:
A license to manipulate research findings?
Sir,
Medical researchers, the executives of pharmaceutical companies and
members of special interest groups will be dancing in the streets. The
paper by Hackshaw and Kirkwood [1] is nothing but a license to make
unfounded claims about the efficacy of treatment from the results of
randomised trials. Data manipulation is already rife in the research
literature [2] and the proposed re-interpretation of the statistical
analysis will only make matters worse.
The authors' stated aim is to avoid the situation in which the
results of a study are ignored because the data fail to achieve
conventional statistical significance. "If a clinically important effect
is observed with a P-value of just above 0.05 (or an upper or lower
confidence limit close to the no effect value) it is incorrect to conclude
no effect and not consider further what is likely to be an effective
intervention..." [1] Thus, provided that there is a "clinically important"
effect - an arbitrary judgment - and that the P-value or confidence
interval is borderline - another arbitrary judgement - then we should
claim that there is evidence of an effect. But the authors want it both
ways: they criticise the use of the cut-off level for the P-value at 0.05
while, at the same time, they propose changes based on equally arbitrary
criteria.
It would be naive to believe that this re-interpretation will not be
seized upon by those with a vested interest in the outcome of medical
research in order to distort the findings, just as it would be naive to
believe that the authors' call for the use of "moderate words" would lead
to anything but an abuse of language.
Statistics-based research in medicine is fundamentally flawed [2] and
attempting to repair individual defects is a waste of time. We should stop
pretending that large RCTs and epidemiological studies produce anything
that is reliable or valuable.
James Penston
e-mail: james.penston@nhs.net
References
1. Hackshaw A, Kirkwood A. Interpreting and reporting clinical trials
with results of borderline significance. BMJ 2011;343;d3340
2. Penston J. Stats.con - How we've been fooled by statistics-based
research in medicine. The London Press, November 2010.
Competing interests: No competing interests