BMJ 1994;308:283-284 (29 January)

Editorials

The scandal of poor medical research

We need less research, better research, and research done for the right reasons

What should we think about a doctor who uses the wrong treatment, either wilfully or through ignorance, or who uses the right treatment wrongly (such as by giving the wrong dose of a drug)? Most people would agree that such behaviour was unprofessional, arguably unethical, and certainly unacceptable.

What, then, should we think about researchers who use the wrong techniques (either wilfully or in ignorance), use the right techniques wrongly, misinterpret their results, report their results selectively, cite the literature selectively, and draw unjustified conclusions? We should be appalled. Yet numerous studies of the medical literature, in both general and specialist journals, have shown that all of the above phenomena are common.^{1 2 3 4 5 6 7} This is surely a scandal.

When I tell friends outside medicine that many papers published in medical journals are misleading because of methodological weaknesses they are rightly shocked. Huge sums of money are spent annually on research that is seriously flawed through the use of inappropriate designs, unrepresentative samples, small samples, incorrect methods of analysis, and faulty interpretation. Errors are so varied that a whole book on the topic,⁷ valuable as it is, is not comprehensive; in any case, many of those who make the errors are unlikely to read it.

Why are errors so common? Put simply, much poor research arises because researchers feel compelled for career reasons to carry out research that they are ill equipped to perform, and nobody stops them. Regardless of whether a doctor intends to pursue a career in research, he or she is usually expected to carry out some research with the aim of publishing several papers. The length of a list of publications is a dubious indicator of ability to do good research; its relevance to the ability to be a good doctor is even more obscure. A common argument in favour of every doctor doing some research is that it provides useful experience and may help doctors to interpret the published research of others. Carrying out a sensible study, even on a small scale, is indeed useful, but carrying out an ill designed study in ignorance of scientific principles and getting it published surely teaches several undesirable lessons.

In many countries a research ethics committee has to approve all research involving patients. Although the Royal College of Physicians has recommended that scientific criteria are an important part of the evaluation of research proposals,⁸ few ethics committees in Britain include a statistician. Indeed, many ethics committees explicitly take a view of ethics that excludes scientific issues. Consequently, poor or useless studies pass such review even though they can reasonably be considered to be unethical.⁹

The effects of the pressure to publish may be seen most clearly in the increase in scientific fraud,¹⁰ much of which is relatively minor and is likely to escape detection. There is nothing new about the message of data or of data torture, as it has recently been called¹¹ - Charles Babbage described its different forms as long ago as 1830.¹² The temptation to behave dishonestly is surely far greater now, when all too often the main reason for a piece of research seems to be to lengthen a researcher's curriculum vitae. Bailar suggested that there may be greater danger to the public welfare from statistical dishonesty than from almost any other form of dishonesty.¹³

Evaluation of the scientific quality of research papers often falls to statisticians. Responsible medical journals invest considerable effort in getting papers refereed by statisticians; however, few papers are rejected solely on statistical grounds.¹⁴ Unfortunately, many journals use little or no statistical refereeing - bad papers are easy to publish.

Statistical refereeing is a form of fire fighting. The time spent refereeing medical papers (often for little or no reward) would be much better spent in education and in direct participation in research as a member of the research team. There is, though, a serious shortage of statisticians to teach and, especially, to participate in research.¹⁵ Many people think that all you need to do statistics is a computer and appropriate software. This view is wrong even for analysis, but it certainly ignores the essential consideration of study design, the foundations on which research is built. Doctors need not be experts in statistics, but they should understand the principles of sound methods of research. If they can also analyse their own data, so much the better. Amazingly, it is widely considered acceptable for medical researchers to be ignorant of statistics. Many are not ashamed (and some seem proud) to admit that they don't know anything about statistics.

The poor quality of much medical research is widely acknowledged, yet disturbingly the leaders of the medical profession seem only minimally concerned about the problem and make no apparent efforts to find a solution. Manufacturing industry has come to recognise, albeit gradually, that quality control needs to be built in from the start rather than the failures being discarded, and the same principles should inform medical research. The issue here is not one of statistics as such. Rather it is a more general failure to appreciate the basic principles underlying scientific research, coupled with the "publish or perish" climate.

As the system encourages poor research it is the system that should be changed. We need less research, better research, and research done for the right reasons. Abandoning using the number of publications as a measure of ability would be a start.

D G Altman 

---

1. Altman DG. Statistics in medical journals. Stat Med 1983;1:59-71.
2. Pocock SJ, Hughes MD, Lee RJ. Statistical problems in the reporting of clinical trials. A survey of three medical journals. N Engl J Med 1987;317:426-32. [Abstract]
3. Smith DG, Clemens J, Crede W, Harvey M, Gracely EJ. Impact of multiple comparisons in randomised clinical trials. Am J Med 1987;83:545-50. [Medline]
4. Murray GD. The task of a statistical referee. Br J Surg 1988;75:664-7. [Medline]
5. Gotzsche PC. Methodology and overt and hidden bias in reports of 196 double-blind trials of non-steroidal antiinflammatory drugs in rheumatoid arthritis. Controlled Clin Trials 1989;10:31-59. [Medline]
6. Williams HC, Seed P. Inadequate size of negative clinical trials in dermatology. Br J Dermatol 1993;128:317-26. [Medline]
7. Andersen B. Methodological errors in medical research. An incomplete catalogue. Oxford: Blackwell, 1990.
8. Royal College of Physicians. Guidelines on the practice of ethics committees in medical research. London: RCP, 1984.
9. Altman DG. Statistics and ethics in medical research. Misuse of statistics is unethical. BMJ 1980;281:1182-4.
10. Lock S, Wells F, eds. Fraud and misconduct in scientific research. London: BMJ Publishing Group, 1993.
11. Mills JL. Data torturing. N Engl J Med 1993;329:1196-9. [Free Full Text]
12. Babbage C. Reflections on the decline of science in England. New York: Augustus M Kelley, 1970:174-83. (Cited in Broad W, Wade N. Betrayers of the truth. Oxford: Oxford University Press, 1982:29-30.)
13. Bailar JC. Bailar's laws of data analysis. Clin Pharmacol Ther 1976;20:113-20. [Medline]
14. Bailar JC. Communicating with a scientific audience. In: Bailar JC, Mosteller F, eds. Medical uses of statistics. Waltham, MA:NEJM Books, 1986:325-37.
15. Bland JM, Altman DG, Royston JP. Statisticians in medical schools. J R Coll Physicians London 1990;24:85-6. [Medline]

CiteULike    Complore    Connotea    Del.icio.us    Digg    Reddit    Technorati    What's this?

Relevant Articles

The scandal of poor medical research

R S Bhopal
BMJ 1994 308: 1438-39. [Extract] [Full Text]

Dietary treatment of hyperlipidaemia Diets were poorly evaluated

L E Ramsay, W W Yeo, P R Jackson, G Riccardi, A A Rivellese, and O Vaccaro
BMJ 1994 308: 916-917. [Extract] [Full Text]

The scandal of poor medical research: Sloppy use of literature often to blame

R Jones, J Scouller, F Grainer, M Lachlan, S Evans, N Torrance, R C Tallis, N A Matheson, J A Morris, T E J Ind, H Dudley, A Sykes, G R Masterson, S G Ashcroft, J P W Bell, and H Goodare
BMJ 1994 308: 591. [Extract] [Full Text]

This article has been cited by other articles:

• Bedard, P. L., Krzyzanowska, M. K., Pintilie, M., Tannock, I. F. (2007). Statistical Power of Negative Randomized Controlled Trials Presented at American Society for Clinical Oncology Annual Meetings. JCO 25: 3482-3487 [Abstract] [Full text]  
• Halligan, S., Altman, D. G. (2007). Evidence-based Practice in Radiology: Steps 3 and 4--Appraise and Apply Systematic Reviews and Meta-Analyses. Radiology 243: 13-27 [Abstract] [Full text]  
• Dias, S., McNamee, R., Vail, A. (2006). Evidence of improving quality of reporting of randomized controlled trials in subfertility. Hum Reprod 21: 2617-2627 [Abstract] [Full text]  
• Ioannidis, J. P A (2006). Commentary: Grading the credibility of molecular evidence for complex diseases. Int J Epidemiol 35: 572-578 [Full text]  
• Blance, A., Tu, Y.-K., Gilthorpe, M. S (2005). A multilevel modelling solution to mathematical coupling. Stat Methods Med Res 14: 553-565 [Abstract]  
• Hait, W. N. (2005). Updated Methods for Reporting Clinical Trials. Clin. Cancer Res. 11: 6753-6754 [Full text]  
• Byrnes, G., Gurrin, L., Dowty, J., Hopper, J. L. (2005). Publication Policy or Publication Bias?. Cancer Epidemiol. Biomarkers Prev. 14: 1363-1363 [Full text]  
• Grimes, D. A., Schulz, K. F. (2005). Surrogate End Points in Clinical Research: Hazardous to Your Health. Obstet Gynecol 105: 1114-1118 [Abstract] [Full text]  
• Wagena, E J (2005). The scandal of unfair behaviour of senior faculty. J. Med. Ethics 31: 308-308 [Full text]  
• Soares, H. P., Kumar, A., Daniels, S., Swann, S., Cantor, A., Hozo, I., Clark, M., Serdarevic, F., Gwede, C., Trotti, A., Djulbegovic, B. (2005). Evaluation of New Treatments in Radiation Oncology: Are They Better Than Standard Treatments?. JAMA 293: 970-978 [Abstract] [Full text]  
• von Elm, E., Egger, M. (2004). The scandal of poor epidemiological research. BMJ 329: 868-869 [Full text]  
• Ramsay, C., Brown, E., Hartman, G., Davey, P., on behalf of the joint BSAC/HIS Working Party on O, (2003). Room for improvement: a systematic review of the quality of evaluations of interventions to improve hospital antibiotic prescribing. J Antimicrob Chemother 52: 764-771 [Abstract] [Full text]  
• Ray, J.G. (2002). Judging the judges: the role of journal editors. QJM 95: 769-774 [Full text]  
• Palmer, C R (2002). Ethics, data-dependent designs, and the strategy of clinical trials: time to start learning-as-we-go?. Stat Methods Med Res 11: 381-402 [Abstract]  
• Grimes, D. A. (2002). The "CONSORT" Guidelines for Randomized Controlled Trials in Obstetrics & Gynecology. Obstet Gynecol 100: 631-632 [Full text]  
• Halpern, S. D., Karlawish, J. H. T., Berlin, J. A. (2002). The Continuing Unethical Conduct of Underpowered Clinical Trials. JAMA 288: 358-362 [Abstract] [Full text]  
• Altman, D. G. (2002). Poor-Quality Medical Research: What Can Journals Do?. JAMA 287: 2765-2767 [Abstract] [Full text]  
• Altman, D. G., Goodman, S. N., Schroter, S. (2002). How Statistical Expertise Is Used in Medical Research. JAMA 287: 2817-2820 [Abstract] [Full text]  
• Bhandari, M., Guyatt, G. H., Lochner, H., Sprague, S., Tornetta, P. III (2002). Application of the Consolidated Standards of Reporting Trials (CONSORT) in the Fracture Care Literature. JBJS 84: 485-489 [Full text]  
• Moher, D., Schulz, K. F., Altman, D. G. (2001). The CONSORT Statement: Revised Recommendations for Improving the Quality of Reports of Parallel-Group Randomized Trials. J. Am. Podiatr. Med. Assoc. 91: 437-442 [Abstract] [Full text]  
• Moher, D., Schulz, K. F., Altman, D., for the CONSORT Group, (2001). The CONSORT Statement: Revised Recommendations for Improving the Quality of Reports of Parallel-Group Randomized Trials. JAMA 285: 1987-1991 [Abstract] [Full text]  
• Silverman, W. A. (2000). Bad Science and the Role of Institutional Review Boards. Arch Pediatr Adolesc Med 154: 1183-1184 [Full text]  
• GØTZSCHE, P. C (2000). Do patients with osteoarthritis get the clinical research they need?. Ann Rheum Dis 59: 407-408 [Full text]  
• Murray, G. D (2000). Promoting good research practice. Stat Methods Med Res 9: 17-24 [Abstract]  
• Wyatt, J. (1999). Same information, different decisions: format counts. BMJ 318: 1501-1502 [Full text]  
• Johnson, T. (1998). Clinical trials in psychiatry: background and statistical perspective. Stat Methods Med Res 7: 209-234 [Abstract]  
• Egger, M., Smith, G. D., Schneider, M., Minder, C. (1997). Bias in meta-analysis detected by a simple, graphical test. BMJ 315: 629-634 [Abstract] [Full text]  
• Greenhalgh, T. (1997). How to read a paper : getting your bearings (deciding what the paper is about). BMJ 315: 243-246 [Full text]  
• Nicol-Smith, L. (1996). Causality, menopause, and depression: a critical review of the literature. BMJ 313: 1229-1232 [Abstract] [Full text]  
• Delamothe, T. (1996). Whose data are they anyway?. BMJ 312: 1241-1242 [Full text]  
• Wyatt, J. C, Altman, D. G (1995). Commentary: Prognostic models: clinically useful or quickly forgotten?. BMJ 311: 1539-1541 [Full text]  
• Smith, R. (1995). Their lordships on medical research. BMJ 310: 1552-1552 [Full text]  
• Freemantle, N., Henry, D., Maynard, A., Torrance, G. (1995). Promoting cost effective prescribing. BMJ 310: 955-956 [Full text]  
• Gifford, M. (1995). Young people and drug misuse. BMJ 310: 672a-672 [Full text]  
• Oakley, A., Fullerton, D., Holland, J., Arnold, S., France-Dawson, M., Kelley, P., McGrellis, S. (1995). Sexual health education interventions for young people: a methodological review. BMJ 310: 158-162 [Abstract] [Full text]  
• Smith, R (1994). Towards a knowledge based health service. BMJ 309: 217-218 [Full text]  
• Smith, R (1994). Promoting research into peer review. BMJ 309: 143-144 [Full text]  
• Bhopal, R. S., Tonks, A. (1994). The role of letters in reviewing research. BMJ 308: 1582-1583 [Full text]  
• Smith, G D (1994). Increasing the accessibility of data. BMJ 308: 1519-1520 [Full text]  
• Bhopal, R S (1994). The scandal of poor medical research. BMJ 308: 1438b-39 [Full text]  
• Ramsay, L E, Yeo, W W, Jackson, P R, Riccardi, G, Rivellese, A A, Vaccaro, O (1994). Dietary treatment of hyperlipidaemia Diets were poorly evaluated. BMJ 308: 916-917 [Full text]  
• Kelly, W, Kelly, M, Mahmood, R, Turner, S, Elliott, K, Lewis, I H (1994). Standards in medical research Criticism unjustified and unfair. BMJ 308: 790-791 [Full text]  
• Jones, R, Scouller, J, Grainer, F, Lachlan, M, Evans, S, Torrance, N, Tallis, R C, Matheson, N A, Morris, J A, Ind, T E J, Dudley, H, Sykes, A, Masterson, G R, Ashcroft, S G, Bell, J P W, Goodare, H (1994). The scandal of poor medical research: Sloppy use of literature often to blame. BMJ 308: 591-591 [Full text]