Published 18 March 2009, doi:10.1136/bmj.b569
Cite this as: BMJ 2009;338:b569

Editorials

Measuring the quality of hospital care

Should focus on effective learning and improvement rather than judgment

Measuring the quality of hospital care is a thorny business. Health care is complex, and links between clinical practice and patient outcomes are often tenuous and distant. These challenges have not prevented the pursuit of simple indicators that identify the "good" and "bad" hospitals. With the claim of providing such an indicator, the hospital standardised mortality ratio was launched with considerable fanfare—first in the United Kingdom and then in Europe and North America. The ratio identifies hospitals where more patients die than would be expected on the basis of their case mix—the bad hospitals—and hospitals with fewer deaths than expected—the good hospitals.^{1 2}

The attraction of using the hospital standardised mortality ratio is clear. These ratios focus on a clear and important clinical outcome and use routinely available data which are as good at predicting death in some conditions as expensive clinical databases.³ However, their use as indicators of quality and safety of care has been criticised because they may not adequately adjust for case mix or account for chance variation and because mortality may not be a valid indicator of quality.^{4 5 6 7 8} In the linked study (doi:10.1136/bmj.b780), Mohammed and colleagues look at a more technical but nonetheless fundamental concern—whether attempts to adjust for case mix augment rather than minimise bias in measuring hospital death rates.⁹

Measures of risk may not be uniformly related to patient outcomes across all constituencies, a point sometimes called the "constant risk fallacy."¹⁰ For example car ownership might seem like a reasonable indicator of financial status. Yet, using this measure as an economic adjustor may produce misleading results when applied to people living in rural areas, where car ownership reflects the need for transport more than it does income.¹⁰ Similarly, patterns of use of the emergency department may in some areas provide a reasonable measure of illness acuity, but in others it may say more about the availability of alternative settings for receiving urgent care.

Mohammed and colleagues found systematic differences in the associations between hospital mortality rates and the factors used in the case mix adjustment for these ratios, including age, emergency admissions, and comorbidity.⁹ For instance, modest changes in comorbidity correlated with large differences in risk of death between hospitals.

These findings undermine the credibility of standardised mortality ratios and indicate that their role in labelling hospitals as good or bad is unjustified. It should be recognised, however, that the widespread interest in these ratios reflects the historic inattention to quality measurement in health care. Practitioners have generally resisted performance measurement (or done little to facilitate it), and researchers have often focused on criticising proposed measures rather than developing valid alternatives.

Measurements of processes rather than outcomes of clinical care have been proposed as more reliable measures of quality and safety.¹¹ Measuring adherence to recommended processes of care seems to avoid the problems of adjusting outcomes for case mix, but many of the same problems arise in the form of eligibility criteria for these processes. For example, patients in hospital A may receive thrombolytic agents less often than those in hospital B because hospital A cares for more complex patients who have more contraindications to thrombolysis.

Hospital standardised mortality ratios will continue to be used to monitor hospital performance because they use data that are widely available and cheap to collect. The methodological limitations highlighted by Mohammed and colleagues pose less of a problem for using these ratios to monitor and improve quality in individual hospitals because associations between case mix adjustors and risk of death should be relatively constant at individual hospitals over periods of a few years.^{1 12} Using hospital mortality ratios for individual hospitals may indeed encourage efforts to improve quality and safety of care. Studies have reported falls in hospital mortality ratios linked to quality improvement programmes that have used the ratios as outcomes.^{1 12} Measurement itself may encourage improvements in processes of clinical care that ultimately lead to improvements in quality and safety.

Publicly reported quality measures require accuracy and precision to prevent unfair stigmatisation and loss of trust. However, the desire to avoid mislabelling hospitals cannot always trump the pursuit of useful measures of healthcare quality. As with diagnostic tests, we must balance the downsides of false positive results with the risk of missing the disease altogether. Rather than advocating the primacy of one quality metric over another, or reverting to "measurement nihilism" with scepticism and distrust about the use of clinical measurements, we should continue to identify a range of indicators that are appropriate for different clinical contexts. We should then concentrate on how these can support internal learning rather than legitimising unfounded and often counterproductive external judgments.

Cite this as: BMJ 2009;338:b569

¹ Bradford Institute for Health Research, Bradford Royal Infirmary, Bradford BD9 6RJ, ² Centre for Patient Safety, University of Toronto, Toronto, ON, Canada M4N 3M5

Research, doi:10.1136/bmj.b780

---

Competing interests: None declared.

Provenance and peer review: Commissioned; not externally peer reviewed.

References

1. Jarman B, Bottle A, Aylin P, Browne M. Monitoring changes in hospital standardised mortality ratios. BMJ 2005;330:329.[Free Full Text]
2. Jarman B, Gault S, Alves B, Hider A, Dolan S, Cook A, et al. Explaining differences in English hospital death rates using routinely collected data. BMJ 1999;318:1515-20.[Abstract/Free Full Text]
3. Aylin P, Bottle A, Majeed A. Use of administrative data or clinical databases as predictors of risk of death in hospital: comparison of models. BMJ 2007;334:1044.[Abstract/Free Full Text]
4. Thomas JW, Hofer TP. Research evidence on the validity of risk-adjusted mortality rate as a measure of hospital quality of care. Med Care Res Rev 1998;55:371-404.[Abstract/Free Full Text]
5. Iezzoni LI. The risks of risk adjustment. JAMA 1997;278:1600-7.[Abstract/Free Full Text]
6. Shojania KG, Forster AJ. Hospital mortality: when failure is not a good measure of success. CMAJ 2008;179:153-7.[Free Full Text]
7. Guru V, Tu JV, Etchells E, Anderson GM, Naylor CD, Novick RJ, et al. Relationship between preventability of death after coronary artery bypass graft surgery and all-cause risk-adjusted mortality rates. Circulation 2008;117:2969-76.[Abstract/Free Full Text]
8. Penfold RB, Dean S, Flemons W, Moffatt M. Do hospital standardized mortality ratios measure patient safety? HSMRs in the Winnipeg Regional Health Authority. Healthc Pap 2008;8:8-24.[Medline]
9. Mohammed MA, Deeks JJ, Girling A, Rudge G, Carmalt M, Stevens AJ, et al. Evidence of methodological bias in hospital standardised mortality ratios: retrospective database study of English hospitals. BMJ 2009;338:b780.[CrossRef]
10. Nicholl J. Case-mix adjustment in non-randomised observational evaluations: the constant risk fallacy. J Epidemiol Community Health 2007;61:1010-3.[Abstract/Free Full Text]
11. Brown C, Hofer T, Johal A, Thomson R, Nicholl J, Franklin BD, et al. An epistemology of patient safety research: a framework for study design and interpretation. Part 3. End points and measurement. Qual Saf Health Care 2008;17:170-7.[Abstract/Free Full Text]
12. Wright J, Dugdale B, Hammond I, Jarman B, Neary M, Newton D, et al. Learning from death: a hospital mortality reduction programme. J R Soc Med 2006;99:303-8.[Abstract/Free Full Text]

CiteULike    Complore    Connotea    Del.icio.us    Digg    Reddit    StumbleUpon    Technorati    What's this?

Rapid Responses:

Read all Rapid Responses

SMR irrationality

David Brookman
bmj.com, 25 Mar 2009 [Full text]

Multi-Dimensional Quality

Rupert J Suckling
bmj.com, 11 Apr 2009 [Full text]

Online poll

Find out more on doc2doc >>