BMJ  2004;329:868-869 (16 October), doi:10.1136/bmj.329.7471.868

Editorial

The scandal of poor epidemiological research

Reporting guidelines are needed for observational epidemiology

Something surely must be wrong with epidemiology when the new editors of a leading journal in the field entitle their inaugural offering, "Epidemiology—is it time to call it a day?"¹ Observational epidemiology has not had a good press in recent years. Conflicting results from epidemiological studies of the risks of daily life, such as coffee, hair dye, or hormones, are frequently and eagerly reported in the popular press, providing a constant source of anxiety for the public.^{2 3} In many cases deeply held beliefs, given credibility by numerous observational studies over long periods of time, are challenged only when contradicted by randomised trials. In the most recent example, a Cochrane review of randomised trials shows that antioxidant vitamins do not prevent gastrointestinal cancer and may even increase all cause mortality.^{4 5}

Now Pocock et al describe the quality and the litany of problems of 73 epidemiological studies published in January 2001 in general medical and specialist journals (p 883).⁶ Perhaps the most relevant findings relate to how investigators dealt with confounding, multiple comparisons, and subgroup analyses. Confounding, the situation in which an apparent effect of an exposure on risk is explained by its association with other factors, is probably the most important cause of spurious associations in observational epidemiology. For example, a recent meta-analysis of observational studies shows how confounding could have been responsible for the belief that hormone replacement therapy provides protection against cardiovascular disease.⁷ A protective effect of hormone replacement therapy was evident in studies that did not control for socioeconomic status, but not in studies that did (figure).⁷ Higher socioeconomic position is strongly associated with both more frequent use of hormone replacement therapy and lower risk of coronary heart disease. In the large (unconfounded) Women's Health Initiative randomised trial hormone replacement therapy had no beneficial effect on cardiovascular disease.⁸

View larger version (35K): [in this window] [in a new window]   Meta-analysis of cohort studies and case-control studies of hormone replacement therapy and coronary heart disease. There is little evidence for a protective effect when analyses are adjusted for, in contrast to studies not adjusted for, socioeconomic status. Adapted from Humphrey et al, reference 7

 

Worryingly, Pocock et al find that the rationale behind the choice of confounders is usually unclear, and that the extent of adjustment varies greatly. They also confirm that observational studies often consider several exposures, outcomes, and subgroups. This results in multiple statistical tests of hypotheses and a high probability of finding associations that are statistically significant but spurious. In many studies 20% or more of the findings may be erroneous, rather than the expected 5% false positive associations (P < 0.05).⁹

Epidemiology is not alone: and there is hope. As Altman pointed out 10 years ago, much medical research is of poor quality.¹⁰ Efforts so far to remedy what Altman described as the "scandal of poor medical research" have focused on controlled clinical trials. Empirical research has shed light on aspects of methodological quality that are crucial to prevent bias,¹¹ and Consolidated Standards for Reporting Trials (CONSORT) have been developed and endorsed widely.¹²

Clearly, such guidelines are also required for observational epidemiology. A month ago a group of epidemiologists, statisticians, and editors met in Bristol to work on a first draft of STROBE, the Standards for the Reporting of Observational Studies in Epidemiology. STROBE will provide guidance on the reporting of cohort studies, case-control studies, and cross sectional studies; a supporting document will give explanations and examples. Outcomes from the Bristol workshop will be posted on a dedicated website (www.strobe-statement.org), and you and everyone else will be invited to comment and suggest improvements before a revised version is published some time next year.

Can STROBE do for observational epidemiology what CONSORT does for randomised clinical trials? Not exactly. Both guidelines aim to promote comprehensive reporting of important methodological detail and facilitate appraisal of study quality. In the case of a large high quality randomised trial, this means that the results can be assumed to provide an unbiased estimate of the treatment effect in the population studied. This is not so in observational epidemiology. A well conducted case-control or cohort study might still produce misleading results if, for example, important confounders were not known, not measured, or imprecisely measured.

Importantly for observational studies, STROBE will also pay considerable attention to what investigators should write in the discussion section of their paper by suggesting the inclusion of statements on why methodological approaches were chosen, and why results are interpreted the way they are. The skilful interpretation of epidemiological evidence, bearing in mind theoretical considerations, resisting being seduced by possible mechanisms, and suspending beliefs to allow healthy scepticism will remain the great challenge and joy of working in epidemiology.⁸ More transparent and complete reporting of epidemiological studies, coupled with more thoughtful interpretation of their results, will help restore the reputation of a discipline that has contributed importantly to improving the health of the public, and will continue to do so in the future.

(vonelm{at}ispm.unibe.ch), Department of Social and Preventive Medicine, University of Berne, Finkenhubelweg 11, CH-3012 Berne, Switzerland

(egger{at}ispm.unibe.ch), Department of Social and Preventive Medicine, University of Berne, Finkenhubelweg 11, CH-3012 Berne, Switzerland

---

Papers p 883

Competing interests: EvE and ME are members of the STROBE group.

References

1. Davey Smith G, Ebrahim S. Epidemiology—is it time to call it a day? Int J Epidemiol 2001;30: 1-11.[Free Full Text]
2. Taubes G. Epidemiology faces its limits. Science 1995;269: 164-9.[Free Full Text]
3. Bartlett C, Sterne J, Egger M. What is newsworthy? Longitudinal study of the reporting of medical research in two British newspapers. BMJ 2002;325: 81-4.[Abstract/Free Full Text]
4. Bjelakovic G, Nikolova D, Simonetti RG, Gluud C. Antioxidant supplements for prevention of gastrointestinal cancers: a systematic review and meta-analysis. Lancet 2004;364: 1219-28.[CrossRef][ISI][Medline]
5. Lawlor DA, Davey SG, Kundu D, Bruckdorfer KR, Ebrahim S. Those confounded vitamins: what can we learn from the differences between observational versus randomised trial evidence? Lancet 2004;363: 1724-7.[CrossRef][ISI][Medline]
6. Pocock SJ, Collier TJ, Dandreo KJ, de Stavola BL, Goldman MB, Kalish LA, Kasten LE, McCormack VA. Issues in the reporting of epidemiological studies: a survey of recent practice. BMJ 2004;329: 883-7.[Abstract/Free Full Text]
7. Humphrey LL, Chan BK, Sox HC. Postmenopausal hormone replacement therapy and the primary prevention of cardiovascular disease. Ann Intern Med 2002;137: 273-84.[Abstract/Free Full Text]
8. Petitti D. Commentary: hormone replacement therapy and coronary heart disease: four lessons. Int J Epidemiol 2004;33: 461-3.[Free Full Text]
9. Ottenbacher KJ. Quantitative evaluation of multiplicity in epidemiology and public health research. Am J Epidemiol 1998;147: 615-9.[Abstract/Free Full Text]
10. Altman DG. The scandal of poor medical research. BMJ 1994;308: 283-4.[Free Full Text]
11. Jüni P, Altman DG, Egger M. Assessing the quality of controlled clinical trials. BMJ 2001;323: 42-6.[Free Full Text]
12. Moher D, Schulz KF, Altman DG, for the CONSORT Group. The CON SORT statement: revised recommendations for improving the quality of reports of parallel group randomized trials. JAMA 2001;285: 1987-91.[Abstract/Free Full Text]

CiteULike    Complore    Connotea    Del.icio.us    Digg    Reddit    Technorati    What's this?

Relevant Articles

Issues in reporting epidemiological studies: No data are given to support generalisation

David F Blackburn
BMJ 2005 330: 146. [Extract] [Full Text]

Issues in the reporting of epidemiological studies: a survey of recent practice

Stuart J Pocock, Timothy J Collier, Kimberley J Dandreo, Bianca L de Stavola, Marlene B Goldman, Leslie A Kalish, Linda E Kasten, and Valerie A McCormack
BMJ 2004 329: 883. [Abstract] [Full Text] [PDF]

This article has been cited by other articles:

• Knol, M. J., Vandenbroucke, J. P., Scott, P., Egger, M. (2008). What Do Case-Control Studies Estimate? Survey of Methods and Assumptions in Published Case-Control Research. Am J Epidemiol 0: kwn217v1-kwn217 [Abstract] [Full text]  
• Santibanez, M., Bolumar, F., Garcia, A. M (2007). Occupational risk factors in Alzheimer's disease: a review assessing the quality of published epidemiological studies. Occup. Environ. Med. 64: 723-732 [Abstract] [Full text]  
• Egger, M., Altman, D. G, Vandenbroucke, J. P, of the STROBE group, (2007). Commentary: Strengthening the reporting of observational epidemiology the STROBE statement. Int J Epidemiol 0: dym199v1-dym199 [Full text]  
• Scott, P A, Kingsley, G H, Smith, C M, Choy, E H, Scott, D L (2007). Non-steroidal anti-inflammatory drugs and myocardial infarctions: comparative systematic review of evidence from observational studies and randomised controlled trials. Ann Rheum Dis 66: 1296-1304 [Abstract] [Full text]  
• Sanderson, S., Tatt, I. D, Higgins, J. P. (2007). Tools for assessing quality and susceptibility to bias in observational studies in epidemiology: a systematic review and annotated bibliography. Int J Epidemiol 36: 666-676 [Abstract] [Full text]  
• Lee, W., Bindman, J., Ford, T., Glozier, N., Moran, P., Stewart, R., Hotopf, M. (2007). Bias in psychiatric case-control studies: Literature survey. Br. J. Psychiatry 190: 204-209 [Abstract] [Full text]  
• Thompson, A., Antel, J., Carroll, W. (2007). Editorial. Mult Scler 13: 5-6  
• Johnston, K. C., Holloway, R. G. (2006). There is nothing staid about STARD: progress in the reporting of diagnostic accuracy studies.. Neurology 67: 740-741 [Full text]  
• Smidt, N., Rutjes, A.W.S., van der Windt, D. A.W.M., Ostelo, R. W.J.G., Bossuyt, P. M., Reitsma, J. B., Bouter, L. M., de Vet, H. C.W. (2006). The quality of diagnostic accuracy studies since the STARD statement: has it improved?. Neurology 67: 792-797 [Abstract] [Full text]  
• Etminan, M., Gill, S., FitzGerald, M., Samii, A. (2006). Challenges and Opportunities for Pharmacoepidemiology in Drug-Therapy Decision Making. J Clin Pharmacol 46: 6-9 [Abstract] [Full text]  
• Batty, G. D., Shipley, M. J., Langenberg, C., Marmot, M. G., Davey Smith, G. (2006). Adult height in relation to mortality from 14 cancer sites in men in London (UK): evidence from the original Whitehall study. Ann Oncol 17: 157-166 [Abstract] [Full text]  
• von Elm, E., Altman, D. G., Pocock, S., Vandenbroucke, J. P., Egger, M. (2005). RE: "QUALITY OF REPORTING OF OBSERVATIONAL LONGITUDINAL RESEARCH". Am J Epidemiol 162: 1032-1033 [Full text]  
• Barreto, M. L (2005). Efficacy, effectiveness, and the evaluation of public health interventions. J. Epidemiol. Community Health 59: 345-346 [Full text]  
• Blackburn, D. F (2005). Issues in reporting epidemiological studies: No data are given to support generalisation. BMJ 330: 146-146 [Full text]  
• (2004). Hit parade. BMJ 329: 1350-1350 [Full text]  
• (2004). Robin Goodfellow (43-12). Rheumatology (Oxford) 43: 1598-1598 [Full text]  

Rapid Responses:

Read all Rapid Responses

Epidemiology: overlooked paramount fundamental bias.

Sergio Stagnaro
bmj.com, 15 Oct 2004 [Full text]

Real time epidemiology - when?

Dan Rutherford
bmj.com, 17 Oct 2004 [Full text]

It is worth looking deeper

Vasiliy Vlassov
bmj.com, 19 Oct 2004 [Full text]

SCIENCE WITHOUT SENSE

BM Hegde
bmj.com, 21 Oct 2004 [Full text]