Users' guide to detecting misleading claims in clinical research reportsBMJ 2004; 329 doi: https://doi.org/10.1136/bmj.329.7474.1093 (Published 04 November 2004) Cite this as: BMJ 2004;329:1093
Box A: Examples of faulty comparators
a. Misleading choice of dose and administration regimen:w5
- Safer reported 8 trials sponsored by three different drug companies that compared newer second-generation neuroleptic agents to a fixed high dose (20 mg/day; optimal dosing < 12 mg/dayw6) of haloperidol. Patients using the new agents had fewer extrapyramidal side effects.
- Safer also reported a trial comparing paroxetine against amitriptyline, a sedating tricyclic antidepressant, administered amitriptyline twice daily, possibly leading to excessive daytime somnolence. Standard administration would have patients taking their full daily dose of amitriptyline at bedtime.w7
- Johansen and collaborators noted use of an ineffective comparator (nystatin), and use of an inadequate and unusual administration route (oral amphotericin B, poorly absorbed in the gastrointestinal tract) as comparators in randomized trials of the efficacy of antifungals in patients with cancer and neutropenia.w8
b. Misleading comparison against placebo:
Three important trials of angiotensin receptor blockers for patients with diabetic nephropathy used placebo, rather than drugs of demonstrated effectiveness, ACE inhibitors, as the control management strategy.w9-w11 The accompanying editorial suggested that the economic interests of the sponsor dictated that choice of comparator, and that they may have avoided an ACE inhibitor control group because "…sales of angiotensin-receptor blockers would be lower if the two classes of drugs proved equally effective."w12
Box B: Beware composite endpoints – the case of UKPDS
The UK prospective diabetes study (UKPDS) randomized patients with type 2 diabetes to intensive glycemic control vs. conventional control.
The primary endpoint of the trialw13 was time to first "diabetes-related endpoints" (sudden death, death from hyperglycaemia or hypoglycaemia, fatal or non-fatal myocardial infarction, angina, heart failure, stroke, renal failure, amputation [of at least one digit], vitreous haemorrhage, retinal photocoagulation (endpoint added after the trial onset), blindness in one eye, or cataract extraction); "diabetes-related death" (death from myocardial infarction, stroke, peripheral vascular disease, renal disease, hyperglycaemia or hypoglycaemia, and sudden death); or all-cause mortality.
The investigators reported a significant 12% reduction in the relative risk in the combined endpoint (95% CI 1 to 21%). However, the results do not exclude a harmful effect on diabetes-related deaths (RR 90%, 95% CI 73 to 111%) and all-cause mortality (RR 94%, 95% CI 80 to 110%).w13
Most of the apparent effect was a reduction (2.7% of the 3.2% absolute reduction in risk of microvascular complications) in retinal photocoagulation.w13 w14
Reviewers typically summarize the results as showing a reduction in any of 21 diabetes-related endpoints with intensive glycemic control, and only 1 in 35 reviews of the UKPDS results highlighted the dominance on the overall effect of the reduction in the risk of photocoagulation.w15
Box C: Beware small treatment effects – the presentation of different time frames for risk and benefit in the UKPDS
McCormack and Greenhalghw14 pointed out that report 33 of UKPDSw13 expressed the risk of severe hypoglycaemia as percent of participants per year (e.g., 2.3% per year for patients on insulin).
This contrasts with the expression of the benefits as percent of participants over 10 years (e.g., 3.2% absolute reduction in the risk of any diabetes-related endpoints).
The presentation obscures the fact that the absolute increase in frequency of hypoglycaemia with intensive glycemic control is approximately 7 times the absolute reduction in diabetes complications.
1. Wilkes MM, Navickis RJ. Patient survival after human albumin administration. A meta-analysis of randomized, controlled trials. Ann Intern Med 2001;135(3):149-64.
2. Alderson P, Bunn F, Lefebvre C, Li WP, Li L, Roberts I, et al. Human albumin solution for resuscitation and volume expansion in critically ill patients. Cochrane Database Syst Rev 2002(1):CD001208.
5. Safer DJ. Design and reporting modifications in industry-sponsored comparative psychopharmacology trials. J Nerv Ment Dis 2002;190(9):583-92.
6. Geddes J, Freemantle N, Harrison P, Bebbington P. Atypical antipsychotics in the treatment of schizophrenia: systematic overview and meta-regression analysis. BMJ 2000;321(7273):1371-6.
7. Christiansen PE, Behnke K, Black CH, Ohrstrom JK, Bork-Rasmussen H, Nilsson J. Paroxetine and amitriptyline in the treatment of depression in general practice. Acta Psychiatr Scand 1996;93(3):158-63.
8. Johansen HK, Gotzsche PC. Problems in the design and reporting of trials of antifungal agents encountered during meta-analysis. JAMA 1999;282(18):1752-9.
9. Parving H-H, Lehnert H, Brochner-Mortensen J, Gomis R, Andersen S, Arner P, et al. The Effect of Irbesartan on the Development of Diabetic Nephropathy in Patients with Type 2 Diabetes. N Engl J Med 2001;345(12):870-878.
10. Brenner BM, Cooper ME, de Zeeuw D, Keane WF, Mitch WE, Parving H-H, et al. Effects of Losartan on Renal and Cardiovascular Outcomes in Patients with Type 2 Diabetes and Nephropathy. N Engl J Med 2001;345(12):861-869.
11. Lewis EJ, Hunsicker LG, Clarke WR, Berl T, Pohl MA, Lewis JB, et al. Renoprotective effect of the angiotensin-receptor antagonist irbesartan in patients with nephropathy due to type 2 diabetes. N Engl J Med 2001;345(12):851-60.
12. Hostetter TH. Prevention of End-Stage Renal Disease Due to Type 2 Diabetes. N Engl J Med 2001;345(12):910-912.
13. Intensive blood-glucose control with sulphonylureas or insulin compared with conventional treatment and risk of complications in patients with type 2 diabetes (UKPDS 33). UK Prospective Diabetes Study (UKPDS) Group. Lancet 1998;352(9131):837-53.
14. McCormack J, Greenhalgh T. Seeing what you want to see in randomised controlled trials: versions and perversions of UKPDS data. United Kingdom prospective diabetes study. BMJ 2000;320(7251):1720-3.
15. Shaughnessy AF, Slawson DC. What happened to the valid POEMs? A survey of review articles on the treatment of type 2 diabetes. BMJ 2003;327(7409):266.
- This Week In The BMJ Published: 04 November 2004; BMJ 329 doi:10.1136/bmj.329.7474.0-d
- Analysis Published: 22 May 2008; BMJ 336 doi:10.1136/bmj.39504.506319.80
- Analysis Published: 26 April 2007; BMJ 334 doi:10.1136/bmj.39169.447488.94
- LetterUsers' guide to detecting misleading claims in research: Discussion section helps redress the balancePublished: 13 January 2005; BMJ 330 doi:10.1136/bmj.330.7483.145-a
- LetterUsers' guide to detecting misleading claims in research: Misleading claims may be symptom of even more serious flawsPublished: 13 January 2005; BMJ 330 doi:10.1136/bmj.330.7483.145
- Letter Published: 17 February 2005; BMJ 330 doi:10.1136/bmj.330.7488.421-b
- LetterUsers' guide to detecting misleading claims in research: Meta-analyses have led to misleading claims in the pastPublished: 13 January 2005; BMJ 330 doi:10.1136/bmj.330.7483.145-b
- River blindness in East Africa: gains and lossesBMJ January 21, 2020, 368 m155; DOI: https://doi.org/10.1136/bmj.m155
- Association of ideal cardiovascular health at age 50 with incidence of dementia: 25 year follow-up of Whitehall II cohort studyBMJ August 07, 2019, 366 l4414; DOI: https://doi.org/10.1136/bmj.l4414
- Genetic predisposition to increased serum calcium, bone mineral density, and fracture risk in individuals with normal calcium levels: mendelian randomisation studyBMJ August 01, 2019, 366 l4410; DOI: https://doi.org/10.1136/bmj.l4410
- Dietary fats and mortality among patients with type 2 diabetes: analysis in two population based cohort studiesBMJ July 02, 2019, 366 l4009; DOI: https://doi.org/10.1136/bmj.l4009
- Physical activity trajectories and mortality: population based cohort studyBMJ June 26, 2019, 365 l2323; DOI: https://doi.org/10.1136/bmj.l2323
- Sackler Colloquium on Improving the Reproducibility of Scientific Research: Misrepresentation and distortion of research in biomedical literature
- Do clinicians understand the size of treatment effects? A randomized survey across 8 countries
- Multitreatment comparison meta-analysis: promise and peril
- The 11th hour-time for EBM to return to first principles?
- Teaching, learning and assessing evidence-based psychiatry
- Multimorbidity: a challenge for evidence-based medicine
- Evaluating solutions to sponsorship bias
- Incorporating considerations of resources use into grading recommendations
- Waking up from the DREAM of preventing diabetes with drugs
- Hit parade
- Trial protocols: time for more than tinkering
- Users' guide to detecting misleading claims in research: Discussion section helps redress the balance
- Users' guide to detecting misleading claims in research: Misleading claims may be symptom of even more serious flaws
- Users' guide to detecting misleading claims in research: Meta-analyses have led to misleading claims in the past