Education and debate

Probability of adverse events that have not yet occurred: a statistical reminder

^a II Department of Surgery, University of Cologne, Kliniken der Stadt Koln, Ostmerheimer Str 200, D-51109 Koln, Germany

Correspondence to: Dr Eypasch.

The probability of adverse and undesirable events during and after operations that have not yet occurred in a finite number of patients (n) can be estimated with Hanley's simple formula, which gives the upper limit of the 95% confidence interval of the probability of such an event: upper limit of 95% confidence interval=maximum risk=3/n (for n>30). Doctors and surgeons should keep this simple rule in mind when complication rates of zero are reported in the literature and when they have not (yet) experienced a disastrous complication in a procedure.

Just as aeroplanes should not crash, common bile ducts should not be cut and iliac vessels not be punctured during laparoscopic procedures. In reality, however, these things do happen.¹ With the boom in endoscopic surgery, surgeons are claiming to have zero mortality or even zero morbidity in their series of operations. A little reminder, not only for surgeons, may be necessary. If a certain adverse event or complication does not occur in a series, it does not mean that it will never happen. Experience and Murphy's law teach us that catastrophes do happen, and their probability can in fact be calculated by a simple rule of thumb.

In 1983 Hanley, a Canadian statistician, published the paper If nothing goes wrong is everything alright?² This paper deserves explanation and needs to be highlighted to surgeons in particular. The paper describes in detail the statistical implications if an event of interest fails to occur in a finite number of operations or subjects. Instead of assuming that a technique is safe because of zero numerators, we should look at confidence intervals between zero and a certain upper limit. Hanley gives a simple rule, which should be known by every practising surgeon, to calculate the upper limit of a 95% confidence interval.

Methods

THE FORMULA

Hanley wrote: "This rule of three states that if none of n patients showed the event about which we are concerned, we can be 95% confident that the chance of this event is at most 3 in n (i.e. 3/n). In other words, the upper 95% confidence limit of a 0/n rate is approximately 3/n."² The calculations are based on the following consideration. Given the risk of a certain event, the probability of this event not occurring is (1-risk). The probability of this event not occurring in n independent observations (patients or operations) is then (1-risk)ⁿ. The higher the risk, the lower the chance of not finding at least one occurrence of the event. One can therefore determine the maximum risk of an event, with a 5% error, that is compatible with n observations of non-occurrence: (1-maximum risk)ⁿ=0.05, equal to 1-maximum risk=(n root 0.05), equal to 1-maximum risk=(0.05)^1/n. For n>30 this can be approximated by 1-maximum risk=1-(3/n), equal to maximum risk=3/n.

Upper limits of 95% confidence intervals for occurrence of immediate
intraoperative death from vascular injury in series of laparoscopic
appendicectomies and cholecystectomies
-----------------------------------------------------------------------------------
                                                          Upper limit of 95%
                               No of        No of deaths  confidence interval
Study                        procedures     due to injury   (rule of three)
-----------------------------------------------------------------------------------
                           Laparoscopic appendicectomy
Hebebrand et al5              25           0              12/100
Attwood et al3                27           0              11/100
McAnena et al8                29           0              10/100
Frazee et al6                 38           0               8/100
Kum et al4                    57           0               5/100
Tate et al7                   70           0               4/100
Pier et al9                  653           0               4/1000
-----------------------------------------------------------------------------------
Total                             842           0               1/1000
-----------------------------------------------------------------------------------
                           Laparoscopic cholecystectomy
Peters et al10               100           0               3/100
Troidl et al11               400           0               8/1000
Cuschieri et al13           1236           0               2/1000
Southern Surgeons Club15    1518           0               2/1000
Larson et al14              1983           0               1/1000
Collet et al12              2955           0               1/1000
-----------------------------------------------------------------------------------
Total                            8192           0               3/10000

This formula closely fits the upper limit of the 95% confidence interval.² Even when n=20 the number based on the rule of three does not differ substantially from the exact value (15% v 14%²).

EXAMPLE

The event that most worries endoscopic surgeons is intraoperative vascular injury that leads to loss of a limb or death. We selected well known international reports of series of laparoscopic appendicectomies and cholecystectomies from the literature.^{3 4 5 6 7 8 9 10 11 12 13 14 15} None of them reported a major vascular injury with subsequent loss of a limb or death. We applied Hanley's rule of three to the data in the papers to calculate the upper limit of a 95% confidence interval for such an adverse event. The table shows the results of these calculations.

Discussion

Several conclusions can be drawn from the table. It is obvious that a small series of any procedure can say hardly anything about the safety of the technique. Even though a major vascular injury with subsequent loss of a limb or death never occurred, the statistical analysis shows that, depending on the study selected, there was the threat that it might occur in four out of every 1000 procedures or even 12 out of every 100. This makes statements like "laparoscopic appendectomy is the method of choice"³ premature or even irresponsible if they are based on single studies.

The non-occurrence of an adverse event in a surgical series does not mean that it cannot happen. It can, and the true rate of occurrence can be estimated from its 95% confidence interval. It is a good estimate of the worst case that is compatible with the observed data. The smaller the sample, the wider the confidence interval. This means that the upper limit of a confidence interval from a small sample is greater than that from a large sample, but this does not mean that the true probability of an adverse event occurring is larger in a small series.

Doctors and surgeons should keep this simple rule of three in mind when complication rates of zero are reported in the literature and when they have not (yet) experienced a disastrous complication in a procedure.

1. Troidl H, Backer B, Langer B, Winkler-Wilfurth. Fehleranalyse-- Evaluierung und Verhutung von Komplikationen; ihre juristische Implikation. Langenbecks Archive fur Chirurgie Supplement Kongressbericht. Heidelberg: Springer Verlag, 1993:59-72.
2. Hanley JA, Lippman-Hand A. If nothing goes wrong, is everything alright? JAMA 1983;259:1743-5.
3. Attwood SEA, Hill ADK, Murphy PG, Thornton J, Stephens RB. A prospective randomised trial of laparoscopic versus open appendectomy. Surgery 1992;112:497-501. [Medline]
4. Kum CK, Ngoi SS, Goh PMY, Tekant Y, Isaac JR. Randomized controlled trial comparing laparoscopic appendectomy to open appendectomy. Br J Surg 1993;80:1599-600. [Medline]
5. Hebebrand D, Troidl H, Spangenberger W, Neugebauer E, Schwalm T, Gunther MW. Laparoscopic or conventional appendectomy? A prospective randomised trial. Der Chirurg 1994;65:112-20.
6. Frazee RC, Roberts JW, Symmonds RE, Snyder SK, Hendricks JC, Smith RW, et al. A prospective randomised trial comparing open versus laparoscopic appendectomy. Ann Surg 1994;219:725-31. [Medline]
7. Tate JJT, Dawson J, Chung SCS, Lau WY, Li AKC. Laparoscopic versus open appendectomy: prospective randomised trial. Lancet 1993;342:633-7. [Medline]
8. McAnena OJ, Austin O, Hederman WP, Gorey TF, Fitzpatrick J, O'Connell PR. Laparoscopic versus open appendicectomy. Lancet 1991;338:693.
9. Pier A, Gotz F, Bacher C. Laparoscopic appendectomy in 625 cases: from innovation to routine. Surg Laparosc Endosc 1991;1:8-13. [Medline]
10. Peters JH, Ellison EC, Innes JT, Liss JL, Nichols KE, Lomano JM, et al. Safety and efficacy of laparoscopic cholecystectomy. Ann Surg 1991;213: 3-12.
11. Troidl H, Spangenberger W, Langen R, Al-Jaziri A, Eypasch E, Neugebauer E, et al. Laparoscopic cholecystectomy. Technical performance, safety, and patient benefits. Endoscopy 1992;24:252-61. [Medline]
12. Collet D, Edye M, Perissat J. Conversions and complications of laparoscopic cholecystectomy. Results of a survey conducted by the French Society of Endoscopic Surgery and Interventional Radiology. Surg Endosc 1993;7: 334-8.
13. Cuschieri A, Dubois F, Mouiel J, Mouret P, Becker H, Buess G, et al. The European experience with laparoscopic cholecystectomy. Am J Surg 1991; 161:385-7.
14. Larson GM, Vitale GC, Casey J, Evans JS, Gilliam G, Heuser L, et al. Multipractice analysis of laparoscopic cholecystectomy in 1983 patients. Am J Surg 1992;163:221-6. [Medline]
15. The Southern Surgeons Club. A prospective analysis of 1518 laparoscopic cholecystectomies. N Engl J Med 1991;324:1073-8. [Abstract]

CiteULike    Complore    Connotea    Del.icio.us    Digg    Reddit    Technorati    What's this?

This article has been cited by other articles:

• Kriechbaum, K, Michels, S, Prager, F, Georgopoulos, M, Funk, M, Geitzenauer, W, Schmidt-Erfurth, U (2008). Intravitreal Avastin for macular oedema secondary to retinal vein occlusion: a prospective study. Br. J. Ophthalmol. 92: 518-522 [Abstract] [Full text]  
• Pandit, J. J., Satya-Krishna, R., Gration, P. (2007). Superficial or deep cervical plexus block for carotid endarterectomy: a systematic review of complications. Br J Anaesth 99: 159-169 [Abstract] [Full text]  
• Kotze, A., Hinton, W., Crabbe, D. C. G., Carrigan, B. J. (2007). Audit of epidural analgesia in children undergoing thoracotomy for decortication of empyema. Br J Anaesth 98: 662-666 [Abstract] [Full text]  
• Siyambalapitiya, S., Ibbotson, V., Doane, A., Ghigo, E., Campbell, M. J., Ross, R. J. (2007). Combining Growth Hormone Releasing Hormone-Arginine and Synacthen Testing Diminishes the Cortisol Response. J. Clin. Endocrinol. Metab. 92: 853-856 [Abstract] [Full text]  
• Heydtmann, M (2006). The statistics of targets.. Gut 55: 1672-1673 [Full text]  
• Randall, M. S., McKevitt, F. M., Venables, G. S., Cleveland, T. J., Gaines, P. A. (2006). Response to Letter by Kramer et al. Stroke 37: 1360-1360 [Full text]  
• Randall, M. S., McKevitt, F. M., Cleveland, T. J., Gaines, P. A., Venables, G. S. (2006). Is There Any Benefit From Staged Carotid and Coronary Revascularization Using Carotid Stents?: A Single-Center Experience Highlights the Need for a Randomized Controlled Trial. Stroke 37: 435-439 [Abstract] [Full text]  
• Narchi, H (2005). Risk of Wilms' tumour with multicystic kidney disease: a systematic review. Arch. Dis. Child. 90: 147-149 [Abstract] [Full text]  
• Rahman, W., Gregson, R. M. (2004). Diplopia after cataract surgery. Br J Anaesth 92: 153-154 [Full text]  
• Williams, H. C. (2003). Applying Trial Evidence Back to the Patient. Arch Dermatol 139: 1195-1200 [Full text]  
• Galloway, P J, McNeill, E, Paterson, W F, Donaldson, M D C (2002). Safety of the insulin tolerance test. Arch. Dis. Child. 87: 354-356 [Abstract] [Full text]  
• Ravnskov, U., Anton, C. (2002). Statins as the new aspirin. BMJ 324: 789-789 [Full text]  
• White, A., Hayhoe, S., Hart, A., Ernst, E. (2001). Adverse events following acupuncture: prospective survey of 32 000 consultations with doctors and physiotherapists. BMJ 323: 485-486 [Full text]  
• MacPherson, H., Thomas, K., Walters, S., Fitter, M. (2001). The York acupuncture safety study: prospective survey of 34 000 treatments by traditional acupuncturists. BMJ 323: 486-487 [Full text]  
• Silcocks, P, Page, M (2001). What constitutes a histological confirmation of cancer? A survey of terminology interpretation in two English regions. J. Clin. Pathol. 54: 246-248 [Abstract] [Full text]  
• Cashley, M., Paterson, J. K, Stevinson, C., Honan, W., Cooke, B., Ernst, E. (2001). Neurological complications of cervical spine manipulation Authors' reply. JRSM 94: 314-315 [Full text]  
• Derry, S., Loke, Y. K. (2000). Risk of gastrointestinal haemorrhage with long term use of aspirin: meta-analysis. BMJ 321: 1183-1187 [Abstract] [Full text]  
• Tyagi, A. K, Scotcher, S., Kozeis, N., Willshaw, H. E (1998). Can convulsions alone cause retinal haemorrhages in infants?. Br. J. Ophthalmol. 82: 659-660 [Abstract] [Full text]  
• Sandramouli, S, Robinson, R, Tsaloumas, M, Willshaw, H E (1997). Retinal haemorrhages and convulsions. Arch. Dis. Child. 76: 449-451 [Abstract] [Full text]  
• Ferner, R E (1996). Newly licensed drugs. BMJ 313: 1157-1158 [Full text]