BMJ 1998;316:1931-1935 ( 27 June )

Papers

Annual league tables of mortality in neonatal intensive care units: longitudinal study

Gareth J Parry, research fellow in health services research, a Craig R Gould, data manager, c Chris J McCabe, lecturer in health economics, b William O Tarnow-Mordi, reader in neonatal medicine and perinatal epidemiology, c on behalf of the International Neonatal Network and the Scottish Neonatal Consultants and Nurses Collaborative Study Group.

a Medical Care Research Unit, School of Health and Related Research, University of Sheffield, Sheffield S3 7XL, b Sheffield Health Economics Group, School of Health and Related Research, c Department of Child Health, Ninewells Hospital and Medical School, Dundee DD1 9SY

Correspondence to: Mr Parry g.parry{at}sheffield.ac.uk

Objective: To assess whether crude league tables of mortality and league tables of risk adjusted mortality accurately reflect the performance of hospitals.
Design: Longitudinal study of mortality occurring in hospital.
Setting: 9 neonatal intensive care units in the United Kingdom.
Subjects: 2671 very low birth weight or preterm infants admitted to neonatal intensive care units between 1988 and 1994. 
Main outcome measures: Crude hospital mortality and hospital mortality adjusted using the clinical risk index for babies (CRIB) score.
Results: Hospitals had wide and overlapping confidence intervals when ranked by mortality in annual league tables; this made it impossible to discriminate between hospitals reliably. In most years there was no significant difference between hospitals, only random variation. The apparent performance of individual hospitals fluctuated substantially from year to year.
Conclusions: Annual league tables are not reliable indicators of performance or best practice; they do not reflect consistent differences between hospitals. Any action prompted by the annual league tables would have been equally likely to have been beneficial, detrimental, or irrelevant. Mortality should be compared between groups of hospitals using specific criteria---such as differences in the volume of patients, staffing policy, training of staff, or aspects of clinical practice---after adjusting for risk. This will produce more reliable estimates with narrower confidence intervals, and more reliable and rapid conclusions.

Key messages

  • League tables are being used increasingly to evaluate hospital performance in the United Kingdom

  • In annual league tables the rankings of nine neonatal intensive care units in different hospitals had wide and overlapping confidence intervals and their rankings fluctuated substantially over six years

  • Annual league tables of hospital mortality were inherently unreliable for comparing hospital performance or for indicating best practices

  • The UK government's commitment to using annual league tables of outcomes such as mortality to monitor services and the spread of best practices should be reconsidered

  • Prospective studies of risk adjusted outcome in hospitals grouped according to specific characteristics would provide better information and be a better use of resources



© BMJ 1998
Add to CiteULike CiteULike   Add to Complore Complore   Add to Connotea Connotea   Add to Del.icio.us Del.icio.us   Add to Digg Digg   Add to Reddit Reddit   Add to StumbleUpon StumbleUpon   Add to Technorati Technorati   Add to Twitter Twitter    What's this?

Relevant Articles

Paediatric cardiac surgical mortality after Bristol: Details of risk adjustment tools were not given
Gareth Parry, Elizabeth S Draper, and Patricia McKinney
BMJ 2005 330: 43. [Extract] [Full Text]

Any variability in outcome comparisons adjusted for case mix must be accounted for
D F Signorini and N U Weir
BMJ 1999 318: 128. [Extract] [Full Text]

Through the lens of Bristol
BMJ 1998 316: 0. [Full Text] [PDF]

League tables are inaccurate in ranking hospital mortality outcomes
BMJ 1998 316: 0. [Full Text]

This article has been cited by other articles:

  • Draper, E S, Zeitlin, J, Fenton, A C, Weber, T, Gerrits, J, Martens, G, Misselwitz, B, Breart, G, on behalf of the MOSAIC research group, (2009). Investigating the variations in survival rates for very preterm infants in 10 European regions: the MOSAIC birth cohort. Arch. Dis. Child. Fetal Neonatal Ed. 94: F158-F163 [Abstract] [Full text]  
  • Ranstam, J., Wagner, P., Robertsson, O., Lidgren, L. (2008). Health-care quality registers: OUTCOME-ORIENTATED RANKING OF HOSPITALS IS UNRELIABLE. J Bone Joint Surg Br 90-B: 1558-1561 [Abstract] [Full text]  
  • Lingsma, H F, Dippel, D W J, Hoeks, S E, Steyerberg, E W, Franke, C L, van Oostenbrugge, R J, de Jong, G, Simoons, M L, Scholte op Reimer, W J M, The Netherlands Stroke Survey investigators, (2008). Variation between hospitals in patient outcome after stroke is only partly explained by differences in quality of care: results from the Netherlands Stroke Survey. J. Neurol. Neurosurg. Psychiatry 79: 888-894 [Abstract] [Full text]  
  • Castilla, J. A., Hernandez, J., Cabello, Y., Lafuente, A., Pajuelo, N., Marqueta, J., Coroleu, B., (Assisted Reproductive Technology Register of the, (2008). Defining poor and optimum performance in an IVF programme. Hum Reprod 23: 85-90 [Abstract] [Full text]  
  • Acolet, D (2008). Quality of neonatal care and outcome. Arch. Dis. Child. Fetal Neonatal Ed. 93: F69-F73 [Abstract] [Full text]  
  • Lilford, R. J, Brown, C. A, Nicholl, J. (2007). Use of process measures to monitor the quality of clinical practice. BMJ 335: 648-650 [Full text]  
  • Anderson, J., Hackman, M., Burnich, J., Gurgiolo, T. R. (2007). Determining Hospital Performance Based on Rank Ordering: Is It Appropriate?. American Journal of Medical Quality 22: 177-185 [Abstract]  
  • Lemmers, O., Kremer, J. A.M., Borm, G. F. (2007). Incorporating natural variation into IVF clinic league tables. Hum Reprod 22: 1359-1362 [Abstract] [Full text]  
  • Profit, J., Zupancic, J. A. F., Gould, J. B., Petersen, L. A. (2007). Implementing Pay-for-Performance in the Neonatal Intensive Care Unit. Pediatrics 119: 975-982 [Abstract] [Full text]  
  • Profit, J, Zupancic, J A F, McCormick, M C, Richardson, D K, Escobar, G J, Tucker, J, Tarnow-Mordi, W, Parry, G (2006). Moderately premature infants at Kaiser Permanente Medical Care Program in California are discharged home earlier than their peers in Massachusetts and the United Kingdom. Arch. Dis. Child. Fetal Neonatal Ed. 91: F245-F250 [Abstract] [Full text]  
  • Abdel-Latif, M. E., Bajuk, B., Oei, J., Lui, K., for the New South Wales and the Australian Capital, (2006). Mortality and Morbidities Among Very Premature Infants Admitted After Hours in an Australian Neonatal Intensive Care Unit Network. Pediatrics 117: 1632-1639 [Abstract] [Full text]  
  • Darlow, B A, Hutchinson, J L, Simpson, J M, Henderson-Smart, D J, Donoghue, D A, Evans, N J, on behalf of the Australian and New Zealand Neonat, (2005). Variation in rates of severe retinopathy of prematurity among neonatal intensive care units in the Australian and New Zealand Neonatal Network. Br J Ophthalmol 89: 1592-1596 [Abstract] [Full text]  
  • Jenkins, J, Alderdice, F, McCall, E, on behalf of the Neonatal Intensive Care Outcomes, (2005). Improvement in neonatal intensive care in Northern Ireland through sharing of audit data. Qual Saf Health Care 14: 202-206 [Abstract] [Full text]  
  • Parry, G., Draper, E. S, McKinney, P. (2005). Paediatric cardiac surgical mortality after Bristol: Details of risk adjustment tools were not given. BMJ 330: 43-43 [Full text]  
  • Rogowski, J. A., Horbar, J. D., Staiger, D. O., Kenny, M., Carpenter, J., Geppert, J. (2004). Indirect vs Direct Hospital Quality Indicators for Very Low-Birth-Weight Infants. JAMA 291: 202-209 [Abstract] [Full text]  
  • Simpson, J M, Evans, N, Gibberd, R W, Heuchan, A M, Henderson-Smart, D J (2003). Analysing differences in clinical outcomes between hospitals. Qual Saf Health Care 12: 257-262 [Abstract] [Full text]  
  • HOWLEY, P. P., GIBBERD, R. (2003). Using hierarchical models to analyse clinical indicators: a comparison of the gamma-Poisson and beta-binomial models. Int J Qual Health Care 15: 319-329 [Abstract] [Full text]  
  • Beardsall, K, White, D K, Pinto, E M, Kelsall, A W R (2003). Pericardial effusion and cardiac tamponade as complications of neonatal long lines: are they really a problem?. Arch. Dis. Child. Fetal Neonatal Ed. 88: F292-F295 [Abstract] [Full text]  
  • Tekkis, P. P, McCulloch, P., Steger, A. C, Benjamin, I. S, Poloniecki, J. D (2003). Mortality control charts for comparing performance of surgical units: validation study using hospital mortality data. BMJ 326: 786-788 [Abstract] [Full text]  
  • Lindstrom, M, Moghaddassi, M, Merlo, J (2003). Social capital and leisure time physical activity: a population based multilevel analysis in Malmo, Sweden. J. Epidemiol. Community Health 57: 23-28 [Abstract] [Full text]  
  • Miles, H, Litton, E, Curran, A, Goldsworthy, L, Sharples, P, Henderson, A J (2002). The PATRIARCH Study. Using outcome measures for league tables: Can a North American prediction of admission score be used in a United Kingdom children's emergency department?. Emerg. Med. J. 19: 536-538 [Abstract] [Full text]  
  • Field, D, Manktelow, B, Draper, E S (2002). Bench marking and performance management in neonatal care: easier said than done!. Arch. Dis. Child. Fetal Neonatal Ed. 87: F163-164 [Full text]  
  • Basinski, A. (2002). Variations in mortality rates among Canadian NICUs --and anonymous reporting. CMAJ 167: 120-120 [Full text]  
  • Heuchan, A M, Evans, N, Henderson Smart, D J, Simpson, J M (2002). Perinatal risk factors for major intraventricular haemorrhage in the Australian and New Zealand Neonatal Network, 1995-97. Arch. Dis. Child. Fetal Neonatal Ed. 86: F86-F90 [Abstract] [Full text]  
  • Tyson, J., Kennedy, K. (2002). Variations in mortality rates among Canadian neonatal intensive care units: interpretation and implications. CMAJ 166: 191-192 [Full text]  
  • Weir, N. U., Sandercock, P. A. G., Lewis, S. C., Signorini, D. F., Warlow, C. P. (2001). Variations Between Countries in Outcome After Stroke in the International Stroke Trial (IST). Stroke 32: 1370-1377 [Abstract] [Full text]  
  • Weir, N., Dennis, M. S. (2001). Towards a National System for Monitoring the Quality of Hospital-Based Stroke Services. Stroke 32: 1415-1421 [Abstract] [Full text]  
  • Merlo, J, Östergren, P-O, Broms, K, Bjorck-Linné, A, Liedholm, H (2001). Survival after initial hospitalisation for heart failure: a multilevel analysis of patients in Swedish acute care hospitals. J. Epidemiol. Community Health 55: 323-329 [Abstract] [Full text]  
  • Evans, D J, Levene, M I (2001). Evidence of selection bias in preterm survival studies: a systematic review. Arch. Dis. Child. Fetal Neonatal Ed. 84: 79F-84 [Abstract] [Full text]  
  • PARRY, G., BILLINGHAM, K., GIBSON, A. (2000). Infant mortality rate in Sheffield. Arch. Dis. Child. Fetal Neonatal Ed. 82: 257F-257 [Full text]  
  • Drake, A J, Baumer, J H (2000). Improved clinical practice but continuing service deficiencies following a regional audit of childhood diabetes mellitus. Arch. Dis. Child. 82: 302-304 [Abstract] [Full text]  
  • Field, D., Draper, E. S (1999). Survival and place of delivery following preterm birth: 1994-96. Arch. Dis. Child. Fetal Neonatal Ed. 80: 111F-114 [Abstract] [Full text]  
  • Signorini, D F, Weir, N U (1999). Any variability in outcome comparisons adjusted for case mix must be accounted for. BMJ 318: 128a-128 [Full text]  

Rapid Responses:

Read all Rapid Responses

Untitled
D F Signorini
bmj.com, 10 Jul 1998 [Full text]
Access jobs at BMJ Careers
Whats new online at Student BMJ