# Retrospective cohort study of false alarm rates associated with a series of heart operations: the case for hospital mortality monitoring groups

BMJ 2004; 328 doi: https://doi.org/10.1136/bmj.37956.520567.44 (Published 12 February 2004) Cite this as: BMJ 2004;328:375## Data supplement

**Methods of determining the false positive rate**Typically the type I error rate is predetermined as a design parameter of a statistical test. In other cases an exact probability method of determining the type I error rate may be available. Otherwise, we used computer simulation.

The number of simulations was such that the width of approximate 95% confidence limits for the false positive rate,

*p*, estimated from*n*simulated series as given byjustified the number of significant digits reported. For example, to report the value 0.17 to two decimal places—for instance, ±0.005—we determined

*n*fromie, we used

*n*³22576.We did not evaluate the false positive rate for more than 371 cases. Average run lengths can, however, be calculated for the hypothetical situation that a series continues indefinitely and no change in risk ever occurs. Except perhaps for the cusum with the v-mask these average run lengths are not easily found. Run lengths for the CRAM chart have been reported for simulated series with no change and with an abrupt change to twice or half the previous failure rate.[1] For runs of deaths, we include some further information in this appendix.

**Control limits in the CRAM Chart**The CRAM chart assumes the availability of individual risk estimates for each patient. The ratio of the actual number of deaths to the sum of these external risk estimates can be calculated and is called the performance ratio. It can be used to adjust the external estimate to give a prospective performance-adjusted estimate of risk. It is also used in the calculation of the control limits for detecting a change in the death rate. The control limits are determined by repeated χ

^{2}tests of the number of observed minus the number of expected deaths in successive overlapping sequences of patients. The number of patients in a sequence is chosen so that the expected number of deaths with adjustment for previous performance is greater than or equal to some predetermined number, which is called the "horizon." The adjustment consists of multiplying the sum of the external risk estimates by the performance ratio just prior to the start of the sequence. The χ^{2 }test with 1 degree of freedom is performed at a nominal level of significance without adjustment for repeated significance testing.The CRAM chart differs from the other control procedures in that it uses an internal standard of change and does not require an external benchmark nor does it test for departure from an external benchmark. It does, however, quantify the current death rate, which allows comparison with a benchmark value. CRAM charts were drawn with a uniform target risk of 15% for all patients and also with a set of risk factors which give target risks of 3, 5, 13, or 20% depending on the patient.[11] We chose a horizon of 16 and a nominal P value of 0.01 as previously used.[4]

**V-mask design used for the cusum chart**The v-mask is specified by an apex angle θ and a lead distance d or equivalently by a height h and a slope k. See diagram inset to figure 6. With the benchmark death rate of 15%, the binomial distribution of the number of deaths in a sample of size 1 has standard deviation σ = Ö0.15*(1 – 0.15) = 0.357. A shift in the death rate of ±5% can be expressed in terms of σ as δ = 0.05/σ . A value of α = 0.05 is a standard setting which derives from the similarity of the cusum to the SPRT. However, in the cusum it does not have the interpretation of type I error.[5] The above settings determined the height h = σ *log(α /2)(±δ) = ±9.4, and slope k = σ *(±δ/2)=±0.025. A statistical package gave the average run lengths for this design of v-mask (SAS/QC software usage and reference, version 6, SAS Institute, NC, USA).

**Frequency of runs of deaths**Because a run of inpatient deaths is an easy test, it has potential use for monitoring and quality control. We provide tables for a range of parameter settings so that a run of deaths can be used as an alarm signal in various settings. For a range of benchmark death rates and for runs of deaths from 2 to 8 we report the 1st and 50th centiles of the distribution of the number of operations before a run of deaths of the given length occurs for the first time. Table A relates to the probability of observing runs of deaths for various constant death rates. The 1st and 50th centiles (median) are shown of the distribution of the number of operations,

, up to the first occurrence of a run of*n*consecutive deaths for various supposed death rates,*r*. If a run of the given length is observed before fewer operations have been performed than the 1st centile value, this is evidence against the supposed rate and in favour of a higher death rate. A number of operations near or above the median is consistent with the supposed death rate or a lower death rate.*p*The first run of

deaths occurs at operation number*r*, for*n*>*n*+ 1, if and only if all three of the following:*r*(i) there is no run of length

in operations 1 to*r*1*n - r -*(ii) there is no death at operation number

-*n**r*(iii) there are deaths at all of operation numbers

+ 1 to n*n - r*The probability density function for n is therefore:

This iterative formula can be evaluated exactly provided

is not too large, and hence*n*can be found such that*n*where

*k*= 0.01 for the 1st centile and*k*= 0.5 for the median.For large

the computation time increases appreciably, and rounding errors become significant, so that in practice the exact formula yields only approximate values. Therefore for values of*n*and*p*that produce series lengths exceeding 100 000, we calculate projected values instead of exact ones. The calculation of these projected values is based on the negative exponential distribution as explained next.*r*For rare runs of deaths, the discrete operation number can be approximated by analogy with continuous time

. Waiting times for independent instantaneous events occurring at a constant rate λ per unit time are given by the negative exponential distribution which has cumulative density function*t**F*(*t*) = 1 - e^{t}ie,*t*=λ /log (1/(1-k)), where*k*=*F*(*t*)*.*The ratio of the 100*k*th centiles*t*_{1}and*t*_{2}for two different event rates λ = p_{1}and λ = p_{2}is therefore p_{2}/p_{1}.For a randomly selected operation there is a constant probability

*p*per operation that the selected operation and the previous^{r}*r*- 1 were all deaths, except that a correction must be made for the first*r*- 1 operations when the probability is zero. In series with long intervals between runs of*r*deaths the correction will be small and the waiting times will closely follow the negative exponential distribution with constant event rate of*p*per unit time.^{r}Where run lengths

*t*_{2}are >100 000 we used an "exact" value of the run length*t*_{1}for values of*p*_{1}and*r*that gave a centile <100 000. We estimated*t*_{2}(either 1st or 50th centile) for the same value of*r*but different value*p*_{2}according to the formulaBecause the calculations can be time consuming, we have included a table to make it easier to select a suitable run length for use as an informal alarm signal. Table A shows, for example, that there is only a 1% probability that a run of five deaths will occur in the first 1121 operations if the death rate is 10%. For a death rate of 20%, there is a 50% probability of a run of three deaths in the first 107 operations.

- Sismanidis C, Bland M, Poloniecki J. Properties
of the cumulative risk-adjusted mortality (CRAM) chart, including the number
of deaths before a doubling of the death rate is detected.
*Med Decis Making*2003;23:242-51.

**Table A**(Posted as supplied by author) Distribution (1st and 50th centiles) of number of operations up to first occurrence of target run of deaths**Death rate****Run lengths****2****3****4****5****6****7****8**1st centile 0.01 103 10154 1.06e+6 1.058e+8 1.117e+10 1.184e+12 1.183e+14 0.05 6 87 1696 33858 715008 15149349 3.027e+8 0.10 2 14 115 1121 11172 118354 1182569 0.15 2 6 27 160 1043 6927 46142 0.20 2 4 11 43 201 988 4915 0.30 2 3 5 10 25 72 226 0.40 2 3 4 5 9 16 32 0.50 2 3 4 5 6 8 12 0.60 2 3 4 5 6 7 8 0.70 2 3 4 5 6 7 8 0.80 2 3 4 5 6 7 8 50th centile 0.01 7001 729500 7.701e+7 7.702e+9 8.155e+11 8.664e+13 9.903e+15 0.05 291 5836 123216 2464512 52189839 1.109e+9 2.535e+10 0.10 76 769 7701 77016 815466 8664448 99025173 0.15 35 241 1610 10738 71591 507110 3863808 0.20 21 107 541 2707 13538 67691 386817 0.30 10 36 122 407 1358 4528 15093 0.40 6 17 44 112 282 705 1763 0.50 4 10 21 43 88 177 355 0.60 3 7 12 21 36 61 103 0.70 3 5 8 12 18 27 39 0.80 2 3 6 8 11 14 19

- Sismanidis C, Bland M, Poloniecki J. Properties
of the cumulative risk-adjusted mortality (CRAM) chart, including the number
of deaths before a doubling of the death rate is detected.
**Methods of determining the false positive rate**Typically the type I error rate is predetermined as a design parameter of a statistical test. In other cases an exact probability method of determining the type I error rate may be available. Otherwise, we used computer simulation.

The number of simulations was such that the width of approximate 95% confidence limits for the false positive rate,

*p*, estimated from*n*simulated series as given byjustified the number of significant digits reported. For example, to report the value 0.17 to two decimal places—for instance, ±0.005—we determined

*n*fromie, we used

*n*³22576.We did not evaluate the false positive rate for more than 371 cases. Average run lengths can, however, be calculated for the hypothetical situation that a series continues indefinitely and no change in risk ever occurs. Except perhaps for the cusum with the v-mask these average run lengths are not easily found. Run lengths for the CRAM chart have been reported for simulated series with no change and with an abrupt change to twice or half the previous failure rate.[1] For runs of deaths, we include some further information in this appendix.

**Control limits in the CRAM Chart**The CRAM chart assumes the availability of individual risk estimates for each patient. The ratio of the actual number of deaths to the sum of these external risk estimates can be calculated and is called the performance ratio. It can be used to adjust the external estimate to give a prospective performance-adjusted estimate of risk. It is also used in the calculation of the control limits for detecting a change in the death rate. The control limits are determined by repeated χ

^{2}tests of the number of observed minus the number of expected deaths in successive overlapping sequences of patients. The number of patients in a sequence is chosen so that the expected number of deaths with adjustment for previous performance is greater than or equal to some predetermined number, which is called the "horizon." The adjustment consists of multiplying the sum of the external risk estimates by the performance ratio just prior to the start of the sequence. The χ^{2 }test with 1 degree of freedom is performed at a nominal level of significance without adjustment for repeated significance testing.The CRAM chart differs from the other control procedures in that it uses an internal standard of change and does not require an external benchmark nor does it test for departure from an external benchmark. It does, however, quantify the current death rate, which allows comparison with a benchmark value. CRAM charts were drawn with a uniform target risk of 15% for all patients and also with a set of risk factors which give target risks of 3, 5, 13, or 20% depending on the patient.[11] We chose a horizon of 16 and a nominal P value of 0.01 as previously used.[4]

**V-mask design used for the cusum chart**The v-mask is specified by an apex angle θ and a lead distance d or equivalently by a height h and a slope k. See diagram inset to figure 6. With the benchmark death rate of 15%, the binomial distribution of the number of deaths in a sample of size 1 has standard deviation σ = Ö0.15*(1 – 0.15) = 0.357. A shift in the death rate of ±5% can be expressed in terms of σ as δ = 0.05/σ . A value of α = 0.05 is a standard setting which derives from the similarity of the cusum to the SPRT. However, in the cusum it does not have the interpretation of type I error.[5] The above settings determined the height h = σ *log(α /2)(±δ) = ±9.4, and slope k = σ *(±δ/2)=±0.025. A statistical package gave the average run lengths for this design of v-mask (SAS/QC software usage and reference, version 6, SAS Institute, NC, USA).

**Frequency of runs of deaths**Because a run of inpatient deaths is an easy test, it has potential use for monitoring and quality control. We provide tables for a range of parameter settings so that a run of deaths can be used as an alarm signal in various settings. For a range of benchmark death rates and for runs of deaths from 2 to 8 we report the 1st and 50th centiles of the distribution of the number of operations before a run of deaths of the given length occurs for the first time. Table A relates to the probability of observing runs of deaths for various constant death rates. The 1st and 50th centiles (median) are shown of the distribution of the number of operations,

, up to the first occurrence of a run of*n*consecutive deaths for various supposed death rates,*r*. If a run of the given length is observed before fewer operations have been performed than the 1st centile value, this is evidence against the supposed rate and in favour of a higher death rate. A number of operations near or above the median is consistent with the supposed death rate or a lower death rate.*p*The first run of

deaths occurs at operation number*r*, for*n*>*n*+ 1, if and only if all three of the following:*r*(i) there is no run of length

in operations 1 to*r*1*n - r -*(ii) there is no death at operation number

-*n**r*(iii) there are deaths at all of operation numbers

+ 1 to n*n - r*The probability density function for n is therefore:

This iterative formula can be evaluated exactly provided

is not too large, and hence*n*can be found such that*n*where

*k*= 0.01 for the 1st centile and*k*= 0.5 for the median.For large

the computation time increases appreciably, and rounding errors become significant, so that in practice the exact formula yields only approximate values. Therefore for values of*n*and*p*that produce series lengths exceeding 100 000, we calculate projected values instead of exact ones. The calculation of these projected values is based on the negative exponential distribution as explained next.*r*For rare runs of deaths, the discrete operation number can be approximated by analogy with continuous time

. Waiting times for independent instantaneous events occurring at a constant rate λ per unit time are given by the negative exponential distribution which has cumulative density function*t**F*(*t*) = 1 - e^{t}ie,*t*=λ /log (1/(1-k)), where*k*=*F*(*t*)*.*The ratio of the 100*k*th centiles*t*_{1}and*t*_{2}for two different event rates λ = p_{1}and λ = p_{2}is therefore p_{2}/p_{1}.For a randomly selected operation there is a constant probability

*p*per operation that the selected operation and the previous^{r}*r*- 1 were all deaths, except that a correction must be made for the first*r*- 1 operations when the probability is zero. In series with long intervals between runs of*r*deaths the correction will be small and the waiting times will closely follow the negative exponential distribution with constant event rate of*p*per unit time.^{r}Where run lengths

*t*_{2}are >100 000 we used an "exact" value of the run length*t*_{1}for values of*p*_{1}and*r*that gave a centile <100 000. We estimated*t*_{2}(either 1st or 50th centile) for the same value of*r*but different value*p*_{2}according to the formulaBecause the calculations can be time consuming, we have included a table to make it easier to select a suitable run length for use as an informal alarm signal. Table A shows, for example, that there is only a 1% probability that a run of five deaths will occur in the first 1121 operations if the death rate is 10%. For a death rate of 20%, there is a 50% probability of a run of three deaths in the first 107 operations.

- Sismanidis C, Bland M, Poloniecki J. Properties of the cumulative risk-adjusted mortality
(CRAM) chart, including the number of deaths before a doubling of the death rate is
detected.
*Med Decis Making*2003;23:242-51.

**Table A**(Posted as supplied by author) Distribution (1st and 50th centiles) of number of operations up to first occurrence of target run of deaths**Death rate****Run lengths****2****3****4****5****6****7****8**1st centile 0.01 103 10154 1.06e+6 1.058e+8 1.117e+10 1.184e+12 1.183e+14 0.05 6 87 1696 33858 715008 15149349 3.027e+8 0.10 2 14 115 1121 11172 118354 1182569 0.15 2 6 27 160 1043 6927 46142 0.20 2 4 11 43 201 988 4915 0.30 2 3 5 10 25 72 226 0.40 2 3 4 5 9 16 32 0.50 2 3 4 5 6 8 12 0.60 2 3 4 5 6 7 8 0.70 2 3 4 5 6 7 8 0.80 2 3 4 5 6 7 8 50th centile 0.01 7001 729500 7.701e+7 7.702e+9 8.155e+11 8.664e+13 9.903e+15 0.05 291 5836 123216 2464512 52189839 1.109e+9 2.535e+10 0.10 76 769 7701 77016 815466 8664448 99025173 0.15 35 241 1610 10738 71591 507110 3863808 0.20 21 107 541 2707 13538 67691 386817 0.30 10 36 122 407 1358 4528 15093 0.40 6 17 44 112 282 705 1763 0.50 4 10 21 43 88 177 355 0.60 3 7 12 21 36 61 103 0.70 3 5 8 12 18 27 39 0.80 2 3 6 8 11 14 19

- Sismanidis C, Bland M, Poloniecki J. Properties of the cumulative risk-adjusted mortality
(CRAM) chart, including the number of deaths before a doubling of the death rate is
detected.

## Related articles

- This Week In The BMJ Published: 12 February 2004; BMJ 328 doi:10.1136/bmj.328.7436.0-b
- Editorial Published: 12 February 2004; BMJ 328 doi:10.1136/bmj.328.7436.361

## See more

- Whooping cough: What’s behind the rise in cases and deaths in England?BMJ May 17, 2024, 385 q1118; DOI: https://doi.org/10.1136/bmj.q1118
- Dengue: Argentinians turn to homemade repellent amid surge in casesBMJ April 17, 2024, 385 q885; DOI: https://doi.org/10.1136/bmj.q885
- Devolved powers for Greater Manchester led to some health improvements, study showsBMJ March 28, 2024, 384 q767; DOI: https://doi.org/10.1136/bmj.q767
- Long waits in child mental health are a “ticking time bomb” regulator warnsBMJ March 22, 2024, 384 q724; DOI: https://doi.org/10.1136/bmj.q724
- Doctors report big rise in patients with illness because of socioeconomic factorsBMJ March 01, 2024, 384 q538; DOI: https://doi.org/10.1136/bmj.q538

## Cited by...

- Statistical Methods to Monitor Risk Factors in a Clinical Database: Example of a National Cardiac Surgery Registry
- How to improve surgical outcomes
- Monitoring the rate of re-exploration for excessive bleeding after cardiac surgery in adults
- Continuous monitoring of the performance of hip prostheses.
- Monitoring surgical and medical outcomes: the Bernoulli cumulative SUM chart. A novel application to assess clinical interventions
- Paediatric cardiac surgical mortality in England after Bristol: descriptive analysis of hospital episode statistics 1991-2002
- Facing up to surgical deaths