BMJ  2004;328:476-477 (28 February), doi:10.1136/bmj.328.7438.476

Editorial

Absence of evidence is not evidence of absence

We need to report uncertain results and do it clearly

The title of this editorial is not new. For example, it was used nearly a decade ago for an article in the BMJ's Statistics Notes series.¹ Altman and Bland considered the dangers of misinterpreting differences that do not reach significance, criticising use of the term "negative" to describe studies that had not found statistically significant differences. Such studies may not have been large enough to exclude important differences. To leave the impression that they have proved that no effect or no difference exists is misleading.

As an example, a randomised trial of behavioural and specific sexually transmitted infection interventions for reducing transmission of HIV-1 was published in the Lancet.² The incidence rate ratios for the outcome of HIV-1 infection were 0.94 (95% confidence interval 0.60 to 1.45) and 1.00 (0.63 to 1.58) for two intervention groups compared with control. In the abstract, the interpretation is: "The interventions we used were insufficient to reduce HIV-1 incidence..." But, looking again at the confidence intervals, the results in both treatment arms are compatible with a wide range of effects, from a 40% reduction in incidence of HIV-1 to a 50% increase. So, to give a summary of the results that gives the impression that this study has shown that these interventions are not capable of reducing HIV-1 incidence is misleading. What might be the implications for people at risk of HIV-1 infection? It could be that an intervention that does in fact protect against infection is not widely used. It could also be that an intervention that actually harms people by increasing HIV-1 infection is viewed as an intervention which has "no effect." The truth of these situations can be established only by collecting more evidence, and statements implying that an intervention has no effect might actually discourage further studies by giving the impression that the question has been answered.

When is it reasonable to claim that a study has proved that no effect or no difference exists? The correct answer is "never," because some uncertainty will always exist. However, we need to have some rules for deciding when we are fairly sure that we have excluded an important benefit or harm. This implies that some threshold must be decided, in advance, for what size of effect is clinically important in that situation. This concept is not new and is used in designing equivalence studies, which set out to show whether one intervention is as good as another.³ Thresholds, often called limits of equivalence, are set between which an effect is designated as being too small to be important. Outcomes of, for example, studies of effectiveness can then be related to these thresholds. This is shown in the figure, where the confidence interval from a study is interpreted in the context of predefined limits of equivalence.

View larger version (49K): [in this window] [in a new window]   Relation between confidence interval, line of no effect, and thresholds for important differences (adapted from Armitage, Berry, and Matthews4)

 

Of course, setting such thresholds is not straightforward. How big a reduction in the incidence of HIV-1 infection is important? How large an increase in incidence is important? Who should decide? How different should the thresholds be for different groups of patients and different outcomes? These are difficult questions, and although we may not be able to find easy answers to them, we can at least be more explicit in reporting what we have found in our research. Wording such as "our results are compatible with a decrease of this much or an increase of this much" would be more informative.

What can we do to help ensure that in another decade we will be closer to heeding the advice of Altman and Bland? Firstly, considering results of a particular study in the context of all available research which considers the same question can increase statistical power, reduce uncertainty, and thus reduce the confusing reporting of underpowered studies. Such an approach might have clarified the implications of a recent study of passive smoking published in the BMJ.⁵ Secondly, researchers need to be precise in their interpretation and language and avoid the temptation to save words by reducing the summary of the study to such an extent that the correct meaning is lost. Thirdly, journals need to be willing to publish uncertain results and thus reduce the pressure on researchers to report their results as definitive.⁶ We need to create a culture that is comfortable with estimating and discussing uncertainty.

UK Cochrane Centre, Oxford OX2 7LG (palderson{at}cochrane.co.uk)

---

I thank Iain Chalmers and Mike Clarke for comments on draft versions.

Competing interests: None declared.

References

1. Altman DG, Bland JM. Absence of evidence is not evidence of absence. BMJ 1995;311: 485.[Free Full Text]
2. Kamali A, Quigley M, Nakiyingi J, Kinsman J, Kengeya-Kayondo J, Gopal R, et al. Syndomic management of sexually-transmitted infections and behaviour change interventions on transmission of HIV-1 in rural Uganda: a community randomised trial. Lancet 2003;361: 645-52.[CrossRef][ISI][Medline]
3. Greene WL, Concato J, Feinstein AR. Claims of equivalence in medical research: are they supported by the evidence? Ann Intern Med 2000;132: 715-22.[Abstract/Free Full Text]
4. Armitage P, Berry G, Matthews JNS. Statistical methods in medical research. 4th ed. Oxford: Blackwell Science, 2002.
5. Enstrom JE, Kabat GC. Environmental tobacco smoke and tobacco related mortality in a prospective study of Californians, 1960-98. BMJ 2003;326: 1057-60.[Abstract/Free Full Text]
6. Alderson P, Roberts I. Should journals publish systematic reviews that find no evidence to guide practice? Examples from injury research. BMJ 2000;320: 376-7.[Free Full Text]

Related Articles

Treatment of hepatic encephalopathy: Authors' reply

Bodil Als-Nielsen, Lise L Gluud, and Christian Gluud
BMJ 2004 329: 112. [Extract] [Full Text]

Confidence intervals illuminate absence of evidence

Doug Altman and J Martin Bland
BMJ 2004 328: 1016-1017. [Extract] [Full Text]

This article has been cited by other articles:

• Meshinchi, S., Arceci, R. J., Sanders, J. E., Smith, F. O., Woods, W. B., Radich, J. P., Alonzo, T. A., Gale, R. E., Hills, R., Wheatley, K., Burnett, A. K., Linch, D. C. (2006). Role of allogeneic stem cell transplantation in FLT3/ITD-positive AML.. Blood 108: 400-401 [Full text]  
• Taddio, A., Lee, C., Yip, A., Parvez, B., McNamara, P. J., Shah, V. (2006). Intravenous Morphine and Topical Tetracaine for Treatment of Pain in Preterm Neonates Undergoing Central Line Placement. JAMA 295: 793-800 [Abstract] [Full text]  
• Kumar, A., Soares, H., Djulbegovic, B. (2005). Are Statistically Non-Significant Findings Necessarily Negative? A Review of All Phase III Randomized Controlled Trials in Hematology Conducted by NCI Sponsored Cooperative Groups.. ASH ANNUAL MEETING ABSTRACTS 106: 293-293 [Abstract]  
• Novik, B. (2005). Randomized Trial of Fixation vs Nonfixation of Mesh in Total Extraperitoneal Inguinal Hernioplasty. Arch Surg 140: 811-812 [Full text]  
• Degraeuwe, P. L. J., Blanco, C. E. (2005). Rapid Feeding Volume Advancements: Uncertainty About the Effect on Necrotizing Enterocolitis Incidence. Pediatrics 115: 1439-1439 [Full text]  
• Soares, H. P., Kumar, A., Daniels, S., Swann, S., Cantor, A., Hozo, I., Clark, M., Serdarevic, F., Gwede, C., Trotti, A., Djulbegovic, B. (2005). Evaluation of New Treatments in Radiation Oncology: Are They Better Than Standard Treatments?. JAMA 293: 970-978 [Abstract] [Full text]  
• Bhardwaj, S. S., Camacho, F., Derrow, A., Fleischer, A. B. Jr, Feldman, S. R. (2004). Statistical Significance and Clinical Relevance: The Importance of Power in Clinical Trials in Dermatology. Arch Dermatol 140: 1520-1523 [Abstract] [Full text]  
• Cummings, P., Rivara, F. P., Koepsell, T. D. (2004). Writing Informative Abstracts for Journal Articles. Arch Pediatr Adolesc Med 158: 1086-1088 [Full text]  
• Almedom, A. M (2004). Evidence from samples of one. BMJ 329: 1052-1052 [Full text]  
• Ogilvie, D., Egan, M., Hamilton, V., Petticrew, M. (2004). Promoting walking and cycling as an alternative to using cars: systematic review. BMJ 329: 763- [Abstract] [Full text]  
• Als-Nielsen, B., Gluud, L. L, Gluud, C. (2004). Treatment of hepatic encephalopathy: Authors' reply. BMJ 329: 112-112 [Full text]  
• Altman, D., Bland, J M. (2004). Confidence intervals illuminate absence of evidence. BMJ 328: 1016-1017 [Full text]  
• (2004). Hit Parade. BMJ 328: 962-962 [Full text]  

Rapid Responses:

Read all Rapid Responses

Splenic infarcts as a manifestation of fever of unknown origin (FUO)

Daniel Cuevas-Ramos, et al.
bmj.com, 1 Mar 2004 [Full text]

Claims of no harm: a call for improvement of medical reporting in medical journals

Michal R Pijak, et al.
bmj.com, 1 Mar 2004 [Full text]

Absence of evidence and the importance of confidence intervals

Douglas G Altman, et al.
bmj.com, 3 Mar 2004 [Full text]

Interpreting the epidemiologic evidence concerning environmental tobacco smoke and coronary heart disease

Geoffrey C. Kabat, et al.
bmj.com, 5 Mar 2004 [Full text]

Re: Interpreting the epidemiologic evidence concerning environmental tobacco smoke and coronary heart disease

Jon P. Krueger
bmj.com, 6 Mar 2004 [Full text]

A Laymans View

Robert Feal-Martinez
bmj.com, 7 Mar 2004 [Full text]

Re: Re: Interpreting the epidemiologic evidence concerning environmental tobacco smoke and coronary heart disease

Wiel M Maessen
bmj.com, 8 Mar 2004 [Full text]

Re: Re: Interpreting the epidemiologic evidence concerning environmental tobacco smoke and coronary heart disease

James W Austin
bmj.com, 11 Mar 2004 [Full text]

Re: Re: Re: Interpreting the epidemiologic evidence concerning environmental tobacco smoke and coronary heart disease

Adam Jacobs
bmj.com, 13 Mar 2004 [Full text]

Secondary Motives....

Michael J. McFadden
bmj.com, 14 Mar 2004 [Full text]

UNRELIABILITY OF SCIENTIFIC PAPERS AS EVIDENCE

Clifford G. Miller
bmj.com, 16 Apr 2004 [Full text]

In response to Clifford Miller's opinions on medical science

Jeffrey Mann
bmj.com, 17 Apr 2004 [Full text]

Re: In response to Clifford Miller's opinions on medical science

Clifford G. Miller
bmj.com, 18 Apr 2004 [Full text]

Absence of evidence is not evidence of absence is a false statement

Ron Law
bmj.com, 11 Aug 2004 [Full text]

Re: Absence of evidence is not evidence of absence is a false statement

John P Heptonstall
bmj.com, 13 Aug 2004 [Full text]

Re: Absence of evidence is not evidence of absence is a false statement

Clifford G. Miller
bmj.com, 13 Aug 2004 [Full text]

Where do you get these people?

Richard M Lindley
bmj.com, 15 Aug 2004 [Full text]

Absence of evidence is not THE SAME AS evidence of absence is a TRUE statement

Sam Lewis
bmj.com, 15 Aug 2004 [Full text]

Chiasmus: entertaining and ambiguous

Robert A. Da Prato
bmj.com, 17 Aug 2004 [Full text]