BMJ  2007;334:349-351 (17 February), doi:10.1136/bmj.39070.527986.68

Analysis

When are randomised trials unnecessary? Picking signal from noise

The relation between a treatment and its effect is sometimes so dramatic that bias can be ruled out as an explanation. Paul Glasziouand colleagues suggest how to determine when observations speak for themselves

Paul Glasziou, professor1, Iain Chalmers, editor2, Michael Rawlins, honorary professor3, Peter McCulloch, reader4

1 Centre for Evidence-Based Medicine, Department of Primary Health Care, University of Oxford, Oxford OX3 7LF, 2 James Lind Library, James Lind Initiative, Oxford OX2 7LG , 3 London School of Hygiene and Tropical Medicine, London WC1E 7HT, 4 Nuffield Department of Surgery, John Radcliffe Hospital, Oxford OX3 9DU

Correspondence to: P Glasziou paul.glasziou@dphpc.ox.ac.uk

Abstract

Although randomised trials are widely accepted as the ideal way of obtaining unbiased estimates of treatment effects, some treatments have dramatic effects that are highly unlikely to reflect inadequately controlled biases. We compiled a list of historical examples of such effects and identified the features of convincing inferences about treatment effects from sources other than randomised trials. A unifying principle is the size of the treatment effect (signal) relative to the expected prognosis (noise) of the condition. A treatment effect is inferred most confidently when the signal to noise ratio is large and its timing is rapid compared with the natural course of the condition. For the examples we considered in detail the rate ratio often exceeds 10 and thus is highly unlikely to reflect bias or factors other than a treatment effect. This model may help to reduce controversy about evidence for treatments whose effects are so dramatic that randomised trials are unnecessary.


Add to CiteULike CiteULike   Add to Complore Complore   Add to Connotea Connotea   Add to Del.icio.us Del.icio.us   Add to Digg Digg   Add to Reddit Reddit   Add to StumbleUpon StumbleUpon   Add to Technorati Technorati    What's this?

Relevant Articles

Developing and evaluating complex interventions: the new Medical Research Council guidance
Peter Craig, Paul Dieppe, Sally Macintyre, Susan Michie, Irwin Nazareth, and Mark Petticrew
BMJ 2008 337: a1655. [Full Text]

What is "quality of evidence" and why is it important to clinicians?
Gordon H Guyatt, Andrew D Oxman, Regina Kunz, Gunn E Vist, Yngve Falck-Ytter, Holger J Schünemann for the GRADE Working Group
BMJ 2008 336: 995-998. [Extract] [Full Text] [PDF]

Too high a pedestal
Jeffrey K Aronson
BMJ 2008 336: 735. [Extract] [Full Text] [PDF]

The urge to sprinkle statistics is irresistible
James Penston
BMJ 2007 334: 440. [Extract] [Full Text] [PDF]

Beware the "Texas sharp shooter" in rate ratios of progression
Anna C Goodman
BMJ 2007 334: 440. [Extract] [Full Text] [PDF]

Controversies revisited
Jane Smith
BMJ 2007 334: 0. [Extract] [Full Text]

Money can't buy you satisfaction
Chris Ham
BMJ 2005 330: 597-599. [Extract] [Full Text] [PDF]

Grading quality of evidence and strength of recommendations
BMJ 2004 328: 1490. [Abstract] [Full Text] [PDF]

This article has been cited by other articles:

  • Terasawa, T., Dvorak, T., Ip, S., Raman, G., Lau, J., Trikalinos, T. A. (2009). Systematic Review: Charged-Particle Radiation Therapy for Cancer. ANN INTERN MED 151: 556-565 [Abstract] [Full text]  
  • Vandenbroucke, J. P (2009). Commentary: 'Smoking and lung cancer'--the embryogenesis of modern epidemiology. Int J Epidemiol 38: 1193-1196 [Full text]  
  • Stylianou, C., Kelnar, C. (2009). The introduction of successful treatment of diabetes mellitus with insulin. JRSM 102: 298-303 [Full text]  
  • Howick, J., Glasziou, P., Aronson, J. K (2009). The evolution of evidence hierarchies: what can Bradford Hill's 'guidelines for causation' contribute?. JRSM 102: 186-194 [Full text]  
  • Vandenbroucke, J. P (2009). Commentary: Maziak's essay, seen from another angle. Int J Epidemiol 38: 410-412 [Full text]  
  • Buckley, B, Murphy, A W (2009). Do patients with angina alone have a more benign prognosis than patients with a history of acute myocardial infarction, revascularisation or both? Findings from a community cohort study. Heart 95: 461-467 [Abstract] [Full text]  
  • Treasure, T. (2009). Are randomised trials needed in the era of rapidly evolving technologies?. Eur. J. Cardiothorac. Surg. 35: 474-478 [Abstract] [Full text]  
  • Glasziou, P., Irwig, L., Deeks, J. J. (2008). When Should a New Test Become the Current Reference Standard?. ANN INTERN MED 149: 816-821 [Abstract] [Full text]  
  • Chong, L.-Y., Treasure, T. (2008). Acupuncture to relieve the pain of thoracotomy: commentary on randomized, controlled trial.. J. Thorac. Cardiovasc. Surg. 136: 1470-1471 [Full text]  
  • Craig, P., Dieppe, P., Macintyre, S., Michie, S., Nazareth, I., Petticrew, M. (2008). Developing and evaluating complex interventions: the new Medical Research Council guidance. BMJ 337: a1655-a1655 [Full text]  
  • Stephenson, T. (2008). Improving patient safety in paediatrics and child health. Arch. Dis. Child. 93: 650-653 [Full text]  
  • Guyatt, G. H., Cook, D. J., Jaeschke, R., Pauker, S. G., Schunemann, H. J. (2008). Grades of Recommendation for Antithrombotic Agents: American College of Chest Physicians Evidence-Based Clinical Practice Guidelines (8th Edition). Chest 133: 123S-131S [Abstract] [Full text]  
  • Macbeth, F. R., Williams, M. V. (2008). Proton Therapy Should Be Tested in Randomized Trials. JCO 26: 2590-2591 [Full text]  
  • Guyatt, G. H, Oxman, A. D, Kunz, R., Vist, G. E, Falck-Ytter, Y., Schunemann, H. J, for the GRADE Working Group, (2008). What is "quality of evidence" and why is it important to clinicians?. BMJ 336: 995-998 [Full text]  
  • Aronson, J. K (2008). Too high a pedestal. BMJ 336: 735-735 [Full text]  
  • Loudon, I. (2008). The use of historical controls and concurrent controls to assess the effects of sulphonamides, 1936-1945. JRSM 101: 148-155 [Full text]  
  • Hemila, H. (2008). THE AUTHOR REPLIES. Am J Epidemiol 167: 501-502 [Full text]  
  • Robinson, J. K., Dellavalle, R. P., Bigby, M., Callen, J. P. (2008). Systematic Reviews: Grading Recommendations and Evidence Quality. Arch Dermatol 144: 97-99 [Full text]  
  • Lloyd, G. (2007). More than high standards needed. BMJ 335: 900-900 [Full text]  
  • Jacobson, P. D., Rettig, R. A., Aubry, W. M. (2007). Litigating the Science of Breast Cancer Treatment. Journal of Health Politics, Policy and Law 32: 785-818 [Abstract]  
  • Auerbach, A. D., Landefeld, C. S., Shojania, K. G. (2007). The Tension between Needing to Improve Care and Knowing How to Do It. NEJM 357: 608-613 [Full text]  
  • Meakins, J. L (2007). Evaluating surgery. JRSM 100: 357-359 [Full text]  
  • Boutron, I., Ravaud, P., Nizard, R. (2007). The design and assessment of prospective randomised, controlled trials in orthopaedic surgery. J Bone Joint Surg Br 89-B: 858-863 [Abstract] [Full text]  
  • Penston, J. (2007). The urge to sprinkle statistics is irresistible. BMJ 334: 440-440 [Full text]  
  • Goodman, A. C (2007). Beware the "Texas sharp shooter" in rate ratios of progression. BMJ 334: 440-440 [Full text]  

Rapid Responses:

Read all Rapid Responses

All or none studies
Olivier Steichen
bmj.com, 16 Feb 2007 [Full text]
The irresistible urge to sprinkle statistics
James Penston
bmj.com, 19 Feb 2007 [Full text]
Beware of the Texas sharp shooter in calculating rate ratios of progression
Anna C Goodman
bmj.com, 20 Feb 2007 [Full text]
Evidence from case series
Mounia N Hocine, et al.
bmj.com, 20 Feb 2007 [Full text]
Mother's Kiss
Daniel Polowetzky
bmj.com, 24 Feb 2007 [Full text]
Response from the Authors
Paul P Glasziou
bmj.com, 5 Mar 2007 [Full text]
75 years ago: 24. December 1932. Streptococci kill, antibiotics help to survive!
Friedrich Flachsbart
bmj.com, 26 Dec 2007 [Full text]
Access jobs at BMJ Careers
Whats new online at Student BMJ