Believability of relative risks and odds ratios in abstracts: cross sectional studyBMJ 2006; 333 doi: https://doi.org/10.1136/bmj.38895.410451.79 (Published 27 July 2006) Cite this as: BMJ 2006;333:231
- Peter C Gøtzsche (), director1
- Accepted 1 April 2006
Objective To compare the distribution of P values in abstracts of randomised controlled trials with that in observational studies, and to check P values between 0.04 and 0.06.
Design Cross sectional study of all 260 abstracts in PubMed of articles published in 2003 that contained “relative risk” or “odds ratio” and reported results from a randomised trial, and random samples of 130 abstracts from cohort studies and 130 from case-control studies. P values were noted or calculated if unreported.
Main outcome measures Prevalence of significant P values in abstracts and distribution of P values between 0.04 and 0.06.
Results The first result in the abstract was statistically significant in 70% of the trials, 84% of cohort studies, and 84% of case-control studies. Although many of these results were derived from subgroup or secondary analyses, or biased selection of results, they were presented without reservations in 98% of the trials. P values were more extreme in observational studies (P < 0.001) and in cohort studies than in case-control studies (P = 0.04). The distribution of P values around P = 0.05 was extremely skewed. Only five trials had 0.05 ≤ P < 0.06, whereas 29 trials had 0.04 ≤ P < 0.05. I could check the calculations for 27 of these trials. One of four non-significant results was significant. Four of the 23 significant results were wrong, five were doubtful, and four could be discussed. Nine cohort studies and eight case-control studies reported P values between 0.04 and 0.06, but in all 17 cases P < 0.05. Because the analyses had been adjusted for confounders, these results could not be checked.
Conclusions Significant results in abstracts are common but should generally be disbelieved.
References w1-w19 and a table giving the recalculations for P values are on bmj.com
This article was posted on bmj.com on 19 July 2006: http://bmj.com/cgi/doi/10.1136/bmj.38895.410451.79
I thank S H Arshad, B H Auestad, C Baker, D Bishai, P Callas, J T Connor, H Fjærtoft, T P George, K-T Khaw, L Korsholm, L C Mion, F-J Neumann, Y Sato, B-S Sheu, and P Talmud for clarifications on their studies.
Contributors PCG is the sole contributor and is guarantor.
Competing interest None declared.
Ethical approval Not needed.