BMJ  2008;336:23-25 (5 January), doi:10.1136/bmj.39379.359560.AD

Analysis

Listen to the data when results are not significant

¹ York Trials Unit, Department of Health Sciences, University of York, York YO10 5DD

Unexpected non-significant results from randomised trials can be difficult to accept. Catherine Hewitt, Natasha Mitchell, and David Torgerson find that some authors continue to support interventions despite evidence that they might be harmful

The first 150 words of the full text of this article appear below.

When randomised controlled trials show a difference that is not statistically significant there is a risk of interpretive bias.¹ Interpretive bias occurs when authors and readers overemphasise or underemphasise results. For example, authors may claim that the non-significant result is due to lack of power rather than lack of effect, using terms such as borderline significance² or stating that no firm conclusions can be drawn because of the modest sample size.³ In contrast, if the study shows a non-significant effect that opposes the study hypothesis, it may be downplayed by emphasising the results are not statistically significant. We investigated the problem of interpretive bias in a sample of recently published trials with findings that did not support the study hypothesis.

Why interpretive bias occurs

A non-significant difference between two groups in a randomised controlled trial may have several explanations. The observed difference may be real and the study is underpowered or the observed difference . . . [Full text of this article]

Extent of problem

Sex education programme for 13-15 year olds
Providing free child safety equipment to prevent injuries
Oral misoprostol for induction of labour
Lidocaine spray to reduce pain during vaginal delivery

Acting on evidence

Summary points

CiteULike    Complore    Connotea    Del.icio.us    Digg    Reddit    Technorati    What's this?

Related Articles

Impact of a theoretically based sex education programme (SHARE) delivered by teachers on NHS registered conceptions and terminations: final results of cluster randomised trial

M Henderson, D Wight, G M Raab, C Abraham, A Parkes, S Scott, and G Hart
BMJ 2007 334: 133. [Abstract] [Full Text] [PDF]

Effectiveness and acceptability of lidocaine spray in reducing perineal pain during spontaneous vaginal delivery: randomised controlled trial

Julia Sanders, Rona Campbell, and Tim J Peters
BMJ 2006 333: 117. [Abstract] [Full Text] [PDF]

Oral misoprostol for induction of labour at term: randomised controlled trial

Jodie M Dodd, Caroline A Crowther, and Jeffrey S Robinson
BMJ 2006 332: 509-513. [Abstract] [Full Text] [PDF]

Randomised placebo controlled multicentre trial to assess short term clarithromycin for patients with stable coronary heart disease: CLARICOR trial

Christian M Jespersen, Bodil Als-Nielsen, Morten Damgaard, Jørgen Fischer Hansen, Stig Hansen, Olav H Helø, Per Hildebrandt, Jørgen Hilden, Gorm B Jensen, Jens Kastrup, Hans Jørn Kolmos, Erik Kjøller, Inga Lind, Henrik Nielsen, Lars Petersen, Christian Gluud CLARICOR Trial Group
BMJ 2006 332: 22-27. [Abstract] [Full Text] [PDF]

Antioxidant supplementation for the prevention of kwashiorkor in Malawian children: randomised, double blind, placebo controlled trial

Heather Ciliberto, Michael Ciliberto, Andreé Briend, Per Ashorn, Dennis Bier, and Mark Manary
BMJ 2005 330: 1109. [Abstract] [Full Text] [PDF]

Providing child safety equipment to prevent injuries: randomised controlled trial

Michael Watson, Denise Kendrick, Carol Coupland, Amanda Woods, Deb Futers, and Jean Robinson
BMJ 2005 330: 178. [Abstract] [Full Text] [PDF]

The Lambeth Early Onset (LEO) Team: randomised controlled trial of the effectiveness of specialised care for early psychosis

Tom K J Craig, Philippa Garety, Paddy Power, Nikola Rahaman, Susannah Colbert, Miriam Fornells-Ambrojo, and Graham Dunn
BMJ 2004 329: 1067. [Abstract] [Full Text] [PDF]

Impact of nurse practitioners on workload of general practitioners: randomised controlled trial

Miranda G H Laurant, Rosella P M G Hermens, Jozé C C Braspenning, Bonnie Sibbald, and Richard P T M Grol
BMJ 2004 328: 927. [Abstract] [Full Text] [PDF]

Effect of interpretive bias on research evidence

Ted J Kaptchuk
BMJ 2003 326: 1453-1455. [Extract] [Full Text] [PDF]

Effect on hip fractures of increased use of hip protectors in nursing homes: cluster randomised controlled trial

Gabriele Meyer, Andrea Warnke, R Bender, and I Mühlhauser
BMJ 2003 326: 76. [Abstract] [Full Text] [PDF]

Sifting the evidencewhat's wrong with significance tests? • Another comment on the role of statistical methods

Jonathan A C Sterne, George Davey Smith, and D R Cox
BMJ 2001 322: 226-231. [Extract] [Full Text] [PDF]

This article has been cited by other articles:

• Adams, J., Burridge, J., Mullee, M., Hammond, A., Cooper, C. (2008). The clinical effectiveness of static resting splints in early rheumatoid arthritis: a randomized controlled trial. Rheumatology (Oxford) 0: ken292v1-ken292 [Abstract] [Full text]  

Rapid Responses:

Read all Rapid Responses

Nature's data

David JR Hutchon
bmj.com, 5 Jan 2008 [Full text]

Cholesterol - a case where the data was not heeded

Evan L Lloyd
bmj.com, 9 Jan 2008 [Full text]

Re: Cholesterol - a case where the data was not heeded

Raymond G Holder
bmj.com, 10 Jan 2008 [Full text]

About p-values and confidence intervals: habitual misconceptions.

Luis C Silva
bmj.com, 12 Jan 2008 [Full text]

Listen to all the evidence when results are not significant

Michael C. Watson, et al.
bmj.com, 16 Jan 2008 [Full text]