Probability of adverse events that have not yet occurred: a statistical reminder
BMJ 1995; 311 doi: https://doi.org/10.1136/bmj.311.7005.619 (Published 02 September 1995) Cite this as: BMJ 1995;311:619- aII Department of Surgery, University of Cologne, Kliniken der Stadt Koln, Ostmerheimer Str 200, D-51109 Koln, Germany
- Correspondence to: Dr Eypasch.
- Accepted 15 June 1995
The probability of adverse and undesirable events during and after operations that have not yet occurred in a finite number of patients (n) can be estimated with Hanley's simple formula, which gives the upper limit of the 95% confidence interval of the probability of such an event: upper limit of 95% confidence interval=maximum risk=3/n (for n>30). Doctors and surgeons should keep this simple rule in mind when complication rates of zero are reported in the literature and when they have not (yet) experienced a disastrous complication in a procedure.
Just as aeroplanes should not crash, common bile ducts should not be cut and iliac vessels not be punctured during laparoscopic procedures. In reality, however, these things do happen.1 With the boom in endoscopic surgery, surgeons are claiming to have zero mortality or even zero morbidity in their series of operations. A little reminder, not only for surgeons, may be necessary. If a certain adverse event or complication does not occur in a series, it does not mean that it will never happen. Experience and Murphy's law teach us that catastrophes do happen, and their probability can in fact be calculated by a simple rule of thumb.
In 1983 Hanley, a Canadian statistician, published the paper If nothing goes wrong is everything alright?2 This paper deserves explanation and needs to be highlighted to surgeons in particular. The paper describes in detail the statistical implications if an event of interest fails to occur in a finite number of operations or subjects. Instead of assuming that a technique is safe because of zero numerators, we should look at confidence intervals between zero and a certain upper limit. Hanley gives a simple rule, which should be known by every practising surgeon, to calculate the upper limit of a 95% confidence interval.
Methods
THE FORMULA
Hanley wrote: “This rule of three states that if none of n patients showed the event about which we are concerned, we can be 95% confident that the chance of this event is at most 3 in n (i.e. 3/n). In other words, the upper 95% confidence limit of a 0/n rate is approximately 3/n.”2 The calculations are based on the following consideration. Given the risk of a certain event, the probability of this event not occurring is (1-risk). The probability of this event not occurring in n independent observations (patients or operations) is then (1-risk)n. The higher the risk, the lower the chance of not finding at least one occurrence of the event. One can therefore determine the maximum risk of an event, with a 5% error, that is compatible with n observations of non-occurrence: (1-maximum risk)n=0.05, equal to 1-maximum risk=(n root 0.05), equal to 1-maximum risk=(0.05)1/n. For n>30 this can be approximated by 1-maximum risk=1-(3/n), equal to maximum risk=3/n.
This formula closely fits the upper limit of the 95% confidence interval.2 Even when n=20 the number based on the rule of three does not differ substantially from the exact value (15% v 14%2).
EXAMPLE
The event that most worries endoscopic surgeons is intraoperative vascular injury that leads to loss of a limb or death. We selected well known international reports of series of laparoscopic appendicectomies and cholecystectomies from the literature.3 4 5 6 7 8 9 10 11 12 13 14 15 None of them reported a major vascular injury with subsequent loss of a limb or death. We applied Hanley's rule of three to the data in the papers to calculate the upper limit of a 95% confidence interval for such an adverse event. The table shows the results of these calculations.
Discussion
Several conclusions can be drawn from the table. It is obvious that a small series of any procedure can say hardly anything about the safety of the technique. Even though a major vascular injury with subsequent loss of a limb or death never occurred, the statistical analysis shows that, depending on the study selected, there was the threat that it might occur in four out of every 1000 procedures or even 12 out of every 100. This makes statements like “laparoscopic appendectomy is the method of choice”3 premature or even irresponsible if they are based on single studies.
The non-occurrence of an adverse event in a surgical series does not mean that it cannot happen. It can, and the true rate of occurrence can be estimated from its 95% confidence interval. It is a good estimate of the worst case that is compatible with the observed data. The smaller the sample, the wider the confidence interval. This means that the upper limit of a confidence interval from a small sample is greater than that from a large sample, but this does not mean that the true probability of an adverse event occurring is larger in a small series.
Doctors and surgeons should keep this simple rule of three in mind when complication rates of zero are reported in the literature and when they have not (yet) experienced a disastrous complication in a procedure.