Listen to the data when results are not significant

doi:10.1136/bmj.39379.359560.AD

From trainee to consultant, BMJ Group offers doctors around the world tailored information, special events, learning resources and recruitment services at every step along their career path.

... by doctors, for doctors, for patients
We are open for entries!

Online learning

BMJ Learning

High-quality CME / CPD for doctors and other healthcare professionals. BMJ Learning features hundreds of accredited, peer-reviewed learning modules in text, video, and audio formats. Find out more

Courses and Qualifications

BMJ Masterclasses

BMJ Masterclasses, led by experts, help clinicians to use the latest evidence and recent guidelines in practice and meet their CPD/CME requirements. Find out more

Exam Preparation

The leading provider of online exam preparation, helping over 167,000 healthcare professionals to pass their exams. Find out more

onExamination

Search all BMJ comment articles:

From

Limit by

Our online table of contents is updated at least twice each day. Read all articles published in the last 7 days.
Keep up to date with cardiology: Access the latest cardiovascular medicine resources from across BMJ Group.
The BMJ iPad app brings you the best of print and online, including live links to the latest news, blogs, video, and podcasts. Get the BMJ iPad app.
OPEN ACCESS: All research articles are freely available online, with no word limit. Find out more about the BMJ's open access policy. Submit your paper.
Countdown to London 2012: BMJ Group's Olympics portal highlights latest Olympics and sports medicine-themed research, comment and learning

Access to the full text of this article requires a subscription or payment. Please log in or subscribe below.

Catherine E Hewitt, research fellow,
Natasha Mitchell, research fellow,
David J Torgerson, director

¹York Trials Unit, Department of Health Sciences, University of York, York YO10 5DD

Correspondence to: D J Torgerson djt6{at}york.ac.uk

Accepted 16 September 2007

Unexpected non-significant results from randomised trials can be difficult to accept. Catherine Hewitt, Natasha Mitchell, and David Torgerson find that some authors continue to support interventions despite evidence that they might be harmful

When randomised controlled trials show a difference that is not statistically significant there is a risk of interpretive bias.1 Interpretive bias occurs when authors and readers overemphasise or underemphasise results. For example, authors may claim that the non-significant result is due to lack of power rather than lack of effect, using terms such as borderline significance2 or stating that no firm conclusions can be drawn because of the modest sample size.3 In contrast, if the study shows a non-significant effect that opposes the study hypothesis, it may be downplayed by emphasising the results are not statistically significant. We investigated the problem of interpretive bias in a sample of recently published trials with findings that did not support the study hypothesis.

Why interpretive bias occurs

A non-significant difference between two groups in a randomised controlled trial may have several explanations. The observed difference may be real and the study is underpowered or the observed difference may occur simply by chance. Bias can also produce a non-significant difference, but we will not include this in our discussion below.

Trialists are rarely neutral about their research. If they are testing a novel intervention they usually suspect that it is effective otherwise they could not convince themselves, their peers, or research funders that it is worth evaluating. This lack of equipoise, however, can affect the way they interpret negative results. They have often invested a large amount of intellectual capital in developing the treatment under evaluation. Naturally, therefore, it is difficult to accept that it may be ineffective.

A trial with statistically significant negative results should, generally, overwhelm any preconceptions and prejudices …