BMJ  2006;333:231-234 (29 July), doi:10.1136/bmj.38895.410451.79 (published 19 July 2006)

Research

Believability of relative risks and odds ratios in abstracts: cross sectional study

¹ Nordic Cochrane Centre, H:S Rigshospitalet, DK-2100 Copenhagen Ø, Denmark, pcg{at}cochrane.dk

Abstract

Objective To compare the distribution of P values in abstracts of randomised controlled trials with that in observational studies, and to check P values between 0.04 and 0.06.

Design Cross sectional study of all 260 abstracts in PubMed of articles published in 2003 that contained "relative risk" or "odds ratio" and reported results from a randomised trial, and random samples of 130 abstracts from cohort studies and 130 from case-control studies. P values were noted or calculated if unreported.

Main outcome measures Prevalence of significant P values in abstracts and distribution of P values between 0.04 and 0.06.

Results The first result in the abstract was statistically significant in 70% of the trials, 84% of cohort studies, and 84% of case-control studies. Although many of these results were derived from subgroup or secondary analyses, or biased selection of results, they were presented without reservations in 98% of the trials. P values were more extreme in observational studies (P < 0.001) and in cohort studies than in case-control studies (P = 0.04). The distribution of P values around P = 0.05 was extremely skewed. Only five trials had 0.05 P < 0.06, whereas 29 trials had 0.04 P < 0.05. I could check the calculations for 27 of these trials. One of four non-significant results was significant. Four of the 23 significant results were wrong, five were doubtful, and four could be discussed. Nine cohort studies and eight case-control studies reported P values between 0.04 and 0.06, but in all 17 cases P < 0.05. Because the analyses had been adjusted for confounders, these results could not be checked.

Conclusions Significant results in abstracts are common but should generally be disbelieved.

CiteULike    Complore    Connotea    Del.icio.us    Digg    Reddit    StumbleUpon    Technorati    What's this?

Relevant Articles

The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate healthcare interventions: explanation and elaboration

Alessandro Liberati, Douglas G Altman, Jennifer Tetzlaff, Cynthia Mulrow, Peter C Gøtzsche, John P A Ioannidis, Mike Clarke, P J Devereaux, Jos Kleijnen, and David Moher
BMJ 2009 339: b2700. [Abstract] [Full Text]

Don't believe the relative risks and odds ratios in abstracts
BMJ 2006 333: 0. [Full Text]

Identifying outcome reporting bias in randomised trials on PubMed: review of publications and survey of authors

An-Wen Chan and Douglas G Altman
BMJ 2005 330: 753. [Abstract] [Full Text] [PDF]

Issues in the reporting of epidemiological studies: a survey of recent practice

Stuart J Pocock, Timothy J Collier, Kimberley J Dandreo, Bianca L de Stavola, Marlene B Goldman, Leslie A Kalish, Linda E Kasten, and Valerie A McCormack
BMJ 2004 329: 883. [Abstract] [Full Text] [PDF]

Data dredging, bias, or confounding

George Davey Smith and Shah Ebrahim
BMJ 2002 325: 1437-1438. [Extract] [Full Text] [PDF]

Sifting the evidencewhat's wrong with significance tests? • Another comment on the role of statistical methods

Jonathan A C Sterne, George Davey Smith, and D R Cox
BMJ 2001 322: 226-231. [Extract] [Full Text] [PDF]

Statistics Notes: The use of transformation when comparing two means

J Martin Bland and Douglas G Altman
BMJ 1996 312: 1153. [Extract] [Full Text] [PDF]

This article has been cited by other articles:

• Liberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gotzsche, P. C., Ioannidis, J. P.A., Clarke, M., Devereaux, P. J., Kleijnen, J., Moher, D. (2009). The PRISMA Statement for Reporting Systematic Reviews and Meta-Analyses of Studies That Evaluate Health Care Interventions: Explanation and Elaboration. ANN INTERN MED 151: W-65-W-94 [Abstract] [Full text]  
• Liberati, A., Altman, D. G, Tetzlaff, J., Mulrow, C., Gotzsche, P. C, Ioannidis, J. P A, Clarke, M., Devereaux, P J, Kleijnen, J., Moher, D. (2009). The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate healthcare interventions: explanation and elaboration. BMJ 339: b2700-b2700 [Abstract] [Full text]  
• Boffetta, P., McLaughlin, J. K., La Vecchia, C., Tarone, R. E., Lipworth, L., Blot, W. J. (2008). False-Positive Results in Cancer Epidemiology: A Plea for Epistemological Modesty. JNCI J Natl Cancer Inst 100: 988-995 [Abstract] [Full text]  
• Hopewell, S., Eisinga, A., Clarke, M. (2008). Better reporting of randomized trials in biomedical journal and conference abstracts. Journal of Information Science 34: 162-173 [Abstract]  
• Vandenbroucke, J. P., Elm, E. v., Altman, D. G., Gotzsche, P. C., Mulrow, C. D., Pocock, S. J., Poole, C., Schlesselman, J. J., Egger, M., for the STROBE initiative, (2007). Strengthening the Reporting of Observational Studies in Epidemiology (STROBE): Explanation and Elaboration. ANN INTERN MED 147: W-163-W-194 [Abstract] [Full text]  
• Gotzsche, P. C., Hrobjartsson, A., Maric, K., Tendal, B. (2007). Data Extraction Errors in Meta-analyses That Use Standardized Mean Differences. JAMA 298: 430-437 [Abstract] [Full text]  
• Milbrandt, E. B., Ishizaka, A., Angus, D. C. (2007). Update in Critical Care 2006. Am. J. Respir. Crit. Care Med. 175: 638-648 [Full text]  

Rapid Responses:

Read all Rapid Responses

Time to slow down and accentuate the negative

Trevor Lane, DPhil
bmj.com, 28 Jul 2006 [Full text]

Beware abstracts

Tom Hughes-Davies
bmj.com, 30 Jul 2006 [Full text]

Transparency is a must

Jeevan P Marasinghe, et al.
bmj.com, 31 Jul 2006 [Full text]

P-values without value?

James Penston
bmj.com, 4 Aug 2006 [Full text]

A statistical quibble

Frank E Harrell
bmj.com, 23 Aug 2006 [Full text]

Online poll

Find out more on doc2doc >>