Published 18 March 2009, doi:10.1136/bmj.b780
Cite this as: BMJ 2009;338:b780

Research

Evidence of methodological bias in hospital standardised mortality ratios: retrospective database study of English hospitals

Mohammed A Mohammed, senior lecturer1, Jonathan J Deeks, professor of health statistics1, Alan Girling, senior research fellow1, Gavin Rudge, data scientist1, Martin Carmalt, consultant physician2, Andrew J Stevens, professor of public health and epidemiology1, Richard J Lilford, professor of clinical epidemiology1

1 Unit of Public Health, Epidemiology and Biostatistics, University of Birmingham, Birmingham B15 2TT , 2 Royal Orthopaedic Hospital, Birmingham B31 2AP

Correspondence to: M A Mohammed M.A.Mohammed{at}Bham.ac.uk

Objective To assess the validity of case mix adjustment methods used to derive standardised mortality ratios for hospitals, by examining the consistency of relations between risk factors and mortality across hospitals.

Design Retrospective analysis of routinely collected hospital data comparing observed deaths with deaths predicted by the Dr Foster Unit case mix method.

Setting Four acute National Health Service hospitals in the West Midlands (England) with case mix adjusted standardised mortality ratios ranging from 88 to 140.

Participants 96 948 (April 2005 to March 2006), 126 695 (April 2006 to March 2007), and 62 639 (April to October 2007) admissions to the four hospitals.

Main outcome measures Presence of large interaction effects between case mix variable and hospital in a logistic regression model indicating non-constant risk relations, and plausible mechanisms that could give rise to these effects.

Results Large significant (P≤0.0001) interaction effects were seen with several case mix adjustment variables. For two of these variables—the Charlson (comorbidity) index and emergency admission—interaction effects could be explained credibly by differences in clinical coding and admission practices across hospitals.

Conclusions The Dr Foster Unit hospital standardised mortality ratio is derived from an internationally adopted/adapted method, which uses at least two variables (the Charlson comorbidity index and emergency admission) that are unsafe for case mix adjustment because their inclusion may actually increase the very bias that case mix adjustment is intended to reduce. Claims that variations in hospital standardised mortality ratios from Dr Foster Unit reflect differences in quality of care are less than credible.

© Mohammed et al 2009
This is an open-access article distributed under the terms of the Creative Commons Attribution Non-commercial License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Add to CiteULike CiteULike   Add to Complore Complore   Add to Connotea Connotea   Add to Del.icio.us Del.icio.us   Add to Digg Digg   Add to Reddit Reddit   Add to StumbleUpon StumbleUpon   Add to Technorati Technorati    What's this?

Relevant Articles

Monitoring mortality
Paul Aylin, Alex Bottle, and Brian Jarman
BMJ 2009 338: b1745. [Extract] [Full Text]

Neither constant nor a fallacy
David I Ben-Tovim, Richard J Woodman, Paul Hakendorf, and James Harrison
BMJ 2009 338: b1748. [Extract] [Full Text]

Institutional performance
Phil McShane, Elizabeth Draper, Patricia McKinney, and Roger Parslow
BMJ 2009 338: b1749. [Extract] [Full Text]

Government will still publish hospital standardised mortality ratios despite criticism
Zosia Kmietowicz
BMJ 2009 338: b1641. [Extract] [Full Text]

Measuring quality
Fiona Godlee
BMJ 2009 338: b1356. [Extract] [Full Text]

Measuring the quality of hospital care
John Wright and Kaveh G Shojania
BMJ 2009 338: b569. [Extract] [Full Text]

Use of process measures to monitor the quality of clinical practice
Richard J Lilford, Celia A Brown, and Jon Nicholl
BMJ 2007 335: 648-650. [Extract] [Full Text] [PDF]

Analysis of the distribution of time that patients spend in emergency departments
Thomas E Locker and Suzanne M Mason
BMJ 2005 330: 1188-1189. [Extract] [Full Text] [PDF]

Explaining differences in English hospital death rates using routinely collected data
Brian Jarman, Simon Gault, Bernadette Alves, Amy Hider, Susan Dolan, Adrian Cook, Brian Hurwitz, and Lisa I Iezzoni
BMJ 1999 318: 1515-1520. [Abstract] [Full Text] [PDF]

Rapid Responses:

Read all Rapid Responses

HSMR debate
Paul Aylin, et al.
bmj.com, 19 Mar 2009 [Full text]
Methodological bias in HSMRs
Chris Sherlaw-Johnson, et al.
bmj.com, 25 Mar 2009 [Full text]
HSMR debate addendum
Paul Aylin, et al.
bmj.com, 27 Mar 2009 [Full text]
The use of risk-adjusted SMRs in comparing institutional performance
Phil McShane, et al.
bmj.com, 1 Apr 2009 [Full text]
Authors' reply
Mohammed A Mohammed, et al.
bmj.com, 3 Apr 2009 [Full text]
Standardised mortality- not likely!
Nigel J Saunders
bmj.com, 4 Apr 2009 [Full text]
Substandard Performance and Statistics
Anthony P Morton
bmj.com, 5 Apr 2009 [Full text]
Charslon Index of Co-Morbidity
Pervaiz Iqbal, et al.
bmj.com, 8 Apr 2009 [Full text]
Neither Constant nor a Fallacy
David I Ben-Tovim, et al.
bmj.com, 11 Apr 2009 [Full text]
Death outside of Hospital
Michael C Scott
bmj.com, 23 Apr 2009 [Full text]
Using HSMR for internal improvement
Richard S Steyn, et al.
bmj.com, 13 May 2009 [Full text]
Zero day stay admissions and coding bias
Rodney P Jones
bmj.com, 15 May 2009 [Full text]
Access jobs at BMJ Careers
Whats new online at Student BMJ