Is WHO’s surgical safety checklist being hyped?BMJ 2019; 366 doi: https://doi.org/10.1136/bmj.l4700 (Published 05 August 2019) Cite this as: BMJ 2019;366:l4700
- David R Urbach, professor of surgery and health policy, management, and evaluation1,
- Justin B Dimick, professor and chair of surgery2,
- Alex B Haynes, associate chair for investigation and discovery3,
- Atul A Gawande, professor of surgery4
- 1University of Toronto, Toronto, Ontario, Canada
- 2University of Michigan, Ann Arbor, MI, USA
- 3Department of Surgery and Perioperative Care, Dell Medical School of the University of Texas at Austin, Austin, TX, USA
- 4Harvard Medical School, Boston, MA, USA
- Correspondence to: Justin B Dimick , Alex B Haynes
Yes—David R Urbach, Justin B Dimick
Extraordinary claims, Carl Sagan used to say, require extraordinary evidence. The famed astronomer and science communicator was advocating skepticism around pseudoscientific and paranormal phenomena such as astrology and divination, but it’s reasonable to apply a similarly high standard of scientific evidence to the World Health Organization’s surgical safety checklist.
Checklists, after all, are credited with truly extraordinary power, bordering on the miraculous. A simple, inexpensive intervention can purportedly eliminate half of postoperative deaths.1 The supposed mechanism of this dramatic reduction in mortality is equally fantastic: ensuring that the team members, patient, and procedure are properly identified and confirming that the team has contemplated several processes of care. Such large effects seem implausible, especially considering that most of these processes were already required in modern hospitals and that none has been proved to reduce surgical mortality when applied individually.
Populations and subgroups
What kind of evidence supports the effectiveness of checklists? Principally, it comes from before-and-after studies,12 which have limited value in demonstrating causal effects because of their susceptibility to bias. Caution is always warranted when analyzing longitudinal trends in surgical safety, since operative mortality has been declining over time.3 The one published randomized study of checklists was a stepped wedge cluster randomized controlled trial, carried out in just two hospitals.4 That trial didn’t find a statistical reduction in postoperative mortality with the use of a checklist. It did find a reduction in complication rates with checklist use, but the validity of this finding has been criticized because of a lack of blinding and the exclusion (contrary to the intention-to-treat principle) of patients in the “checklist” arm who didn’t comply with the intervention.
Although studies of entire populations where checklists were introduced didn’t demonstrate an impact on mortality with safety checklists,56 analyses that were restricted to select hospitals or subgroups of patients have found striking effects. For example, introducing safety checklists in the US state of South Carolina was found to have no overall impact on mortality. However, analyses restricted to hospitals that completed a comprehensive safety program—including checklists—found large and statistically significant reductions in operative mortality.
This observation highlights a common attribute of studies that found an effect of safety checklists on mortality: patient selection. In another highly cited example, a before-and-after study in a Dutch hospital found no statistical reduction in operative mortality after implementing a checklist.7 Patients for whom checklists were fully completed, however, showed a more than twofold reduction in odds of dying after surgery. By selecting subgroups that completed the checklist (and comparing them with those that did not), these studies introduce a clear bias: the teams or hospitals that completed the checklist will differ in many important ways, thus influencing surgical outcomes.
A more balanced account
The public success of surgical safety checklists is a triumph of selection bias over rigorous implementation science. This is unfortunate, since checklists have real and meaningful benefits, and their continued use should be strongly encouraged. These benefits, however, relate to improvements in team dynamics, engagement of perioperative nurses, staff satisfaction, and advocacy for better support for surgical care in resource poor settings. While these may not seem as exciting as claims of cutting the risk of operative death in half, they are nevertheless important and don’t require tortured interpretations of the empirical scientific evidence.
A more balanced and realistic account of the benefits of safety checklists would be more credible to a skeptical healthcare community that’s becoming increasingly fatigued by the proliferation of patient safety interventions—an alarming number of which have been adopted without compelling scientific evidence.8
No—Alex B Haynes, Atul A Gawande
The World Health Organization introduced its surgical safety checklist over a decade ago as a performance enhancing tool for teams providing surgical care around the world. A prospective pilot study in eight diverse hospitals around the world associated its introduction with a 46% fall in perioperative mortality after inpatient surgery.9
Since then, numerous studies have documented that introducing the checklist into regular use reduces mortality. A recent review of the evidence showed that this was true in a variety of settings using different study designs and found an apparent dosage relation between adherence and improvements in outcomes.10 Following on from these results, the checklist has been enthusiastically adopted in many healthcare environments around the world. A recent assessment of its use in a pooled analysis of international studies of surgical care found that the checklist was used in 75% of operations assessed.11
The primary debate seems to be over whether the checklist is implementable. The evidence suggests so—but it takes time for population level implementation, and the approach taken is important. David Urbach and colleagues, from the University of Toronto in Canada, reported an ecologic analysis of perioperative mortality in the province of Ontario after a provincially mandated introduction of checklists, enacted shortly after the pilot study’s initial report. After a very rapid analysis (only three months since introduction) they found that the 8.5% decline in mortality observed in all cases—including predominantly outpatient procedures such as ophthalmology and arthroscopy—didn’t reach statistical significance.12
In contrast, a more recent analysis of a carefully supported mandatory implementation of the checklist in Scotland (with analysis focusing on inpatient surgery, where mortality rates are more likely to be affected) found a 39% reduction in mortality during the Scottish Patient Safety Programme, which emphasized use of the checklist as a key element of safe surgical care. No such changes were seen in non-operative admissions, decreasing the likelihood that a more generalized improvement in care was responsible for changes in outcomes.13
A further dive into the checklist’s mechanism can be found in an analysis of the Safe Surgery 2015 program in South Carolina, a statewide voluntary collaborative. A structured implementation pathway was provided for the participants, as well as coaching on implementation, team training, and collaborative peer support.14 Using statewide discharge data, hospitals that completed the program saw a 22% reduction in mortality after inpatient surgery—changes that were not seen in other hospitals in the state.15 Additionally, hospitals that completed the program showed changes in team based surgical practices (for example, improvements of 12% in communication and 3% in coordination), and the greater the improvement in these attributes, the greater the associated perioperative mortality reduction.1617
The checklist was intentionally designed as a tool to strengthen team communication, and these data suggest that it does precisely this. However, evidence from the Canadian study illustrates how important the approach to implementation is for effective use of the checklist, as does analysis of its implementation by NHS England, where early mandate of the checklist without implementation support resulted in very high rates of documentation but only patchy clinically meaningful implementation.121819
Some may argue that the effect of the checklist can’t be separated from the implementation program and associated training. Teams with the capacity for improvement are the most likely to effectively integrate a complex behavioral intervention such as the checklist into their workflow, leading to the criticism that studies of checklist implementation simply identify “improvers.”
We do not disagree. However, even highly motivated teams need tools for implementation, and the evidence suggests that the WHO surgical safety checklist is among the most powerful tools for improving the safety of surgical care introduced in recent years. We must endeavor to learn how to maximize its effective implementation in environments around the world, improve it, and continue to evaluate results.
Competing interests (DU and JD): All authors have read and understood BMJ policy and have no interests to declare.
Competing interests (AH and AG): All authors have read and understood BMJ policy and declare the following interests: AG receives royalties from publishers worldwide for his writing on improving healthcare, including through checklists. He is also chief executive of Haven, a venture formed by Amazon, Berkshire Hathaway, and JPMorgan Chase to enable better healthcare for their employees.
Provenance and peer review: Commissioned; not externally peer reviewed.