Annual league tables of mortality in neonatal intensive care units: longitudinal study

Address for correspondence :
D.F.Signorini
Senior Statistician
Department of Clinical Neurosciences
University of Edinburgh
Bramwell-Dott Building
Western General Hospital
Edinburgh EH4 2XU
Tel 0131 - 537 3127
Fax 0131 - 332 5150

Annual league tables of mortality in neonatal intensive care units:
longitudinal study

Parry et.al. (1) rightly draw attention to the difficulties faced by
those wishing to use comparative outcome data to indicate performance.
They clearly demonstrate the importance of adjusting for differences
in case-mix and allowing for random variation by placing 95%
confidence intervals around estimates of adjusted outcome. We would
point out, however, that in addition to the uncertainty in the
observed mortality, there is also uncertainty in the predicted
mortality. The overall lack of clarity in the rankings of the neonatal
intensive care units might therefore be even greater if this
additional uncertainty were acknowledged, reinforcing the reservations
expressed about decision-making with these kinds of data.

It should always be remembered that predictive models are only
approximations to reality. They must be estimated from previous data,
and thus are themselves prone to noise and random fluctuation. Both
the size of the original dataset and the predictive ability of the
variables used determine the precision of the predicted outcome. In
practice this uncertainty is reflected in the covariance matrix of the
estimated model parameters, and Hosmer and Lemeshow (2) show how this
can be used to calculate the uncertainty associated with the expected
mortality.

The potential influence of this variability can be illustrated with an
example from stroke medicine. We calculated the expected 30 day
fatality in a cohort of 436 stroke patients admitted to a Scottish
hospital using an externally validated logistic regression model
derived from 530 patients from the Oxfordshire Community Stroke
Project. The ratio of observed and predicted mortality (O/E) was used
to standardise the outcome for case-mix (a method independent of unit
size), giving a value of 0.95. Two different 95% confidence intervals
for this ratio were calculated. The first used only simple binomial
variation (95% CI 0.79 to 1.11) and second, using binomial variation
plus model uncertainty, (95% CI 0.75 to 1.16) was 28% larger. This
considerable increase in uncertainty might be found in other
circumstances such as the study described by Parry et.al. Indeed, the
CRIB model (3) used for case-mix adjustment was derived from a similar
number of cases (812), but without explicit knowledge of the model
covariance it is impossible to confirm this hypothesis.

In the current climate of continual outcome comparison and performance
review, it is vitally important that all sources of variability in
case-mix adjusted outcome comparisons are accounted for, as the
consequences of a `false positive' declaration of `significantly
sub-standard' performance are becoming ever more serious.

D.F. Signorini N.U. Weir
Senior Statistician Wellcome Research Fellow

1. Parry GJ, Gould CR, McCabe CJ, Tarnow-Mordi WO. Annual league
tables of mortality in neonatal intensive care units: longitudinal
study. BMJ 1998 316:1931-1935.
2. Hosmer DW, Lemeshow S. Confidence interval estimates of an index
of quality performance based on logistic regression models. Stat Med
1995; 14: 2161-2172.
3. The International Neonatal Network. The CRIB (clinical risk index
for babies) score: a tool for assessing initial neonatal risk and
comparing performance of neonatal intensive care units. Lancet 1993;
342: 193-198.

**** End of letter ****