Cumulative risk adjusted mortality chart for detecting changes in death rate: observational study of heart surgery

Introduction

If a comparison of hospitals or surgeons is to be useful it must be based on a statistic that is relevant to prospective patients. For heart surgery a key question is: “What are the chances of the patient surviving the operation if it is performed today at this hospital?” We monitored deaths after surgery using case specific estimates of survival, which were then adjusted for recent performance. The method could be applied to any process where failures can be identified.

The cumulative number of deaths after a surgical procedure can be plotted against the total number of operations and the cumulative sum (CUSUM) statistical technique applied to provide an alarm if the mortality exceeds an acceptable level. 1 2 Such a method, according to de Leval et al, would have enabled earlier detection and correction of an unacceptable number of deaths in a series of neonatal arterial switch operations. 1 2 The method is not, however, suitable for prospective monitoring when all types of patient and operation are combined into a single series and there is no national standard for acceptable performance.

Few published series have explored the statistical variation in outcomes, which may explain the difficulty and delay in recognising poorer results. 3 4 Adjustment for type of procedure and the patient's condition (collectively referred to as case mix) is necessary for monitoring deaths associated with general surgery. 5 6 We used a published set of weights for established risk factors in adult heart surgery789 to look for changes in the death rate over time.

Subjects and methods

All heart operations performed at St George's Hospital in London between January 1992 and August 1995 were analysed. Mortality in hospital within 30 days was recorded and checked for completeness against the mortuary records of the hospital. There is no established procedure by which an external auditor could check if a death occurring outside hospital but within 30 days had been missed, and so deaths outside hospital were not included. Table 1 lists the items describing the patient's preoperative condition and the proposed procedure. The items and additive weights used to derive an estimated percentage probability of death were as published,7 with the following exceptions. When the published weights (percentage mortality) are a range of values, we used fixed weights: 5 for congenital heart disease and 30 for preoperative catastrophic state — for example, rupture of the aorta. The category other rare circumstances was used only for tricuspid valve surgery, which was given a weight of 3. Heart transplant operations were not in the series presented by Parsonnet et al,7 and a weight of 15 was used on the basis of survival rates in the United Kingdom.

Case mix for a calendar year was assessed as the percentage mortality predicted on the basis of the factors in table 1. Variations in case mix were analysed using analysis of variance after transformation to the natural logarithm (ln) of the odds of predicted mortality to give more uniform variance. Actual mortality was compared with predicted by the χ²test. Multiple linear logistic regression was used to test for annual effects, and the logit — that is, ln of the odds — of the Parsonnet predictor was included as a linear term. Because the ln of 0 is not possible, a predicted mortality of 0.5% was used instead. To test the combined significance of all four years, the χ² test with three degrees of freedom was applied to the difference between models in twice the log of the likelihood.

The statistical power of varying lengths of series to detect increases in mortality was calculated for integral numbers of predicted deaths and integral numbers of observed deaths by assuming that the number of deaths would follow a Poisson distribution with the mean equal to the predicted number (figure). The power may be overestimated by this approximation. The length of a series is specified in terms of risk exposure the expected number of deaths. The same number of deaths can be expected from a short series of high risk cases as from a longer series of lower risk cases. The calculations were used to decide how many recent operations were needed to compare with earlier performance to try to detect a change in mortality.

Fig 1

Probability of detecting an increase in death rate when the rate increases by a factor of 1.1, 1.5, 2, and 4. For example, for 90% probability of detecting a doubling of death rate from a standard rate of 10% to 20%, 16 expected deaths are required at standard rate that is, 160 operations

• Download figure
• Open in new tab
• Download powerpoint

The difference between the expected number of deaths according to the Parsonnet predictor and the actual number of deaths was plotted against the operation number in chronological sequence.10 Control limits for variations in death rate with time were calculated at a nominal level of significance by comparing recent operations with earlier operations. The control limits indicate when the number of deaths in recent operations is significantly different at P<0.01 from the number of deaths predicted by a combination of the Parsonnet score and the previous rate of overperformance or underperformance relative to the Parsonnet score. The χ² statistic with one degree of freedom was used, but this did not amount to a formal test of significance because the calculations were performed after every operation and no allowance was made for the number of tests.

Results

From January 1992 to August 1995, 3983 heart operations were performed. The youngest patient was 16 years old, and there were no missing data for Parsonnet scoring. Expected predischarge mortality according to the Parsonnet predictor was 376 (9.4%). The case mix as assessed by prospective Parsonnet risk of death in hospital varied with calendar year and ranged from 8.7% to 10.6% (P<0.0001) (table 2). Patient factors rather than procedural factors (table 1) were responsible, and the most significant change was in the age of the patients (table 2).

The actual number of deaths for the whole period was 277 (7.0%), which was 99 (26.3%) fewer than the Parsonnet estimate (P<0.0001). For the 656 (16%) patients with a Parsonnet score of 0 the observed number of deaths was 7 (1.1% (95% confidence interval 0.3% to 1.8%)).

Mortality by calendar year ranged from 5.6% to 7.7%, and the difference between the actual and predicted number of survivors as a proportion of predicted deaths represented an improvement of 16% to 39% compared with the Parsonnet estimate (table 2). Logistic regression estimates for the odds ratio of death in a given year, with 1992 as the reference year, ranged from 1.0 to 1.5, but the annual variations in mortality were not significant (P=0.18). The parameter estimate for the Parsonnet predictor in the regression model was close to 1.0 (0.95 (SE 0.06)), indicating that a factor of proportionality was all that was required to make the relative risk of mortality expressed as ln of the odds consistent with the Parsonnet prediction.

To see whether the death rate changes, we have to decide how far back in time a change might have occurred. This divides the series into recent versus earlier operations and allows a standard statistical test for a change (χ² test) to be applied. The more operations that are included in the recent group the greater the power to detect a small change in the death rate, but this means operations continuing for perhaps an unnecessarily long time after there has been a large increase in death rate.

To choose a suitable number of recent operations, we examined the theoretical statistical power to detect changes in mortality. The risk exposure that is, the expected number of deaths rather than the number of operations determines the power, and we can specify the increase in mortality that we want to detect in multiples of the control rate without having to specify what the control rate is. Figure 1 shows the number of expected deaths required for a given probability of detecting increases in mortality. The plots are not smooth because the number of deaths has to be a whole number. A doubling of mortality can be detected with a risk exposure of 16 deaths, and 90% power, and it was decided to use this number of 16 expected deaths as the specification for a suitable number of recent operations.

Each successive run of cases with an expected mortality of 16 deaths was compared with all earlier operations. After 1338 operations performance was 39.3% better than the Parsonnet prediction. For the next 313 cases the Parsonnet prediction was 26.4 deaths, and therefore 16 deaths were expected at the previous level (39%) of performance relative to Parsonnet. Among these 313 cases, however, 27 patients died and the control limits were crossed at operation 1651 (fig 2). After another 86 operations the series was again within control limits for P=0.05 (not shown). There was only one other crossing of the 0.01 control limits, which occurred at operation number 2189; after another 34 operations the series was again within 0.05 limits.

Fig 2

Cumulative risk adjusted mortality chart with 99% control limits for change in mortality in last 16 expected deaths

• Download figure
• Open in new tab
• Download powerpoint

Discussion

Data on risk factors suitable for monitoring deaths after heart surgery can readily be collected, and chart based methods can give an early indication of a deterioration in success rate. Mortality results in our series were overall 26% better than the Parsonnet estimate, which was derived in New Jersey in 1989.7 There were differences in the scoring system that we used compared with that of Parsonnet et al7 because, for example, heart transplants were not performed at their hospital.

Published comparisons of the performance of different hospitals or surgeons are likely to remain of limited value, even with the best possible use of the available data relating to risk factors. The performance data are sometimes anonymous11 and often out of date by the time of publication.12 A more fundamental limitation is that the comparisons are not based on randomised allocation of patients. Thus any differences in performance may be due to unmeasured factors affecting the case mix. Diverting patients or resources from one service to another on the basis of an unrandomised comparison, even if feasible, may not improve results.