Evidence of methodological bias in hospital standardised mortality ratios: retrospective database study of English hospitals
BMJ 2009; 338 doi: https://doi.org/10.1136/bmj.b780 (Published 18 March 2009) Cite this as: BMJ 2009;338:b780- Mohammed A Mohammed, senior lecturer1,
- Jonathan J Deeks, professor of health statistics1,
- Alan Girling, senior research fellow1,
- Gavin Rudge, data scientist1,
- Martin Carmalt, consultant physician2,
- Andrew J Stevens, professor of public health and epidemiology1,
- Richard J Lilford, professor of clinical epidemiology1
- 1Unit of Public Health, Epidemiology and Biostatistics, University of Birmingham, Birmingham B15 2TT
- 2Royal Orthopaedic Hospital, Birmingham B31 2AP
- Correspondence to: M A Mohammed M.A.Mohammed{at}Bham.ac.uk
- Accepted 18 November 2008
Abstract
Objective To assess the validity of case mix adjustment methods used to derive standardised mortality ratios for hospitals, by examining the consistency of relations between risk factors and mortality across hospitals.
Design Retrospective analysis of routinely collected hospital data comparing observed deaths with deaths predicted by the Dr Foster Unit case mix method.
Setting Four acute National Health Service hospitals in the West Midlands (England) with case mix adjusted standardised mortality ratios ranging from 88 to 140.
Participants 96 948 (April 2005 to March 2006), 126 695 (April 2006 to March 2007), and 62 639 (April to October 2007) admissions to the four hospitals.
Main outcome measures Presence of large interaction effects between case mix variable and hospital in a logistic regression model indicating non-constant risk relations, and plausible mechanisms that could give rise to these effects.
Results Large significant (P≤0.0001) interaction effects were seen with several case mix adjustment variables. For two of these variables—the Charlson (comorbidity) index and emergency admission—interaction effects could be explained credibly by differences in clinical coding and admission practices across hospitals.
Conclusions The Dr Foster Unit hospital standardised mortality ratio is derived from an internationally adopted/adapted method, which uses at least two variables (the Charlson comorbidity index and emergency admission) that are unsafe for case mix adjustment because their inclusion may actually increase the very bias that case mix adjustment is intended to reduce. Claims that variations in hospital standardised mortality ratios from Dr Foster Unit reflect differences in quality of care are less than credible.
Footnotes
Editor's note: The embargoed copy of this article, sent to the media, wrongly attributed to Dr Foster Intelligence the authorship of the standardised mortality ratio method that is considered here. The article, as published here, now attributes this standardised mortality ratio method to the Dr Foster Unit at Imperial College.
This independent study was commissioned by the NHS West Midlands Strategic Health Authority. We are grateful for the support of all the members of the steering group, chaired by R Shukla. We especially thank the staff of participating hospitals, in particular P Handslip. Special thanks go to Steve Wyatt for his continued assistance with the project. We also thank our reviewers for their helpful suggestions.
Contributors: MAM drafted the manuscript. MAM and GR did the preliminary analyses. JJD designed and did the statistical modelling to test for interactions, with support from AG. RJL and AJS provided guidance and support. MC provided medical advice and did preliminary investigations into the Charlson index. All authors contributed to the final manuscript. MAM is the guarantor.
Funding: The study was part of a study commissioned by the NHS West Midlands Strategic Health Authority. AG is supported by the EPSRC MATCH consortium.
Competing interests: None declared.
Ethical approval: Not needed.
This is an open-access article distributed under the terms of the Creative Commons Attribution Non-commercial License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.