Is WHO’s surgical safety checklist being hyped?

Populations and subgroups

What kind of evidence supports the effectiveness of checklists? Principally, it comes from before-and-after studies,12 which have limited value in demonstrating causal effects because of their susceptibility to bias. Caution is always warranted when analyzing longitudinal trends in surgical safety, since operative mortality has been declining over time.3 The one published randomized study of checklists was a stepped wedge cluster randomized controlled trial, carried out in just two hospitals.4 That trial didn’t find a statistical reduction in postoperative mortality with the use of a checklist. It did find a reduction in complication rates with checklist use, but the validity of this finding has been criticized because of a lack of blinding and the exclusion (contrary to the intention-to-treat principle) of patients in the “checklist” arm who didn’t comply with the intervention.

Although studies of entire populations where checklists were introduced didn’t demonstrate an impact on mortality with safety checklists,56 analyses that were restricted to select hospitals or subgroups of patients have found striking effects. For example, introducing safety checklists in the US state of South Carolina was found to have no overall impact on mortality. However, analyses restricted to hospitals that completed a comprehensive safety program—including checklists—found large and statistically significant reductions in operative mortality.

This observation highlights a common attribute of studies that found an effect of safety checklists on mortality: patient selection. In another highly cited example, a before-and-after study in a Dutch hospital found no statistical reduction in operative mortality after implementing a checklist.7 Patients for whom checklists were fully completed, however, showed a more than twofold reduction in odds of dying after surgery. By selecting subgroups that completed the checklist (and comparing them with those that did not), these studies introduce a clear bias: the teams or hospitals that completed the checklist will differ in many important ways, thus influencing surgical outcomes.

A more balanced account

The public success of surgical safety checklists is a triumph of selection bias over rigorous implementation science. This is unfortunate, since checklists have real and meaningful benefits, and their continued use should be strongly encouraged. These benefits, however, relate to improvements in team dynamics, engagement of perioperative nurses, staff satisfaction, and advocacy for better support for surgical care in resource poor settings. While these may not seem as exciting as claims of cutting the risk of operative death in half, they are nevertheless important and don’t require tortured interpretations of the empirical scientific evidence.

A more balanced and realistic account of the benefits of safety checklists would be more credible to a skeptical healthcare community that’s becoming increasingly fatigued by the proliferation of patient safety interventions—an alarming number of which have been adopted without compelling scientific evidence.8

Implementation approaches

The primary debate seems to be over whether the checklist is implementable. The evidence suggests so—but it takes time for population level implementation, and the approach taken is important. David Urbach and colleagues, from the University of Toronto in Canada, reported an ecologic analysis of perioperative mortality in the province of Ontario after a provincially mandated introduction of checklists, enacted shortly after the pilot study’s initial report. After a very rapid analysis (only three months since introduction) they found that the 8.5% decline in mortality observed in all cases—including predominantly outpatient procedures such as ophthalmology and arthroscopy—didn’t reach statistical significance.12

In contrast, a more recent analysis of a carefully supported mandatory implementation of the checklist in Scotland (with analysis focusing on inpatient surgery, where mortality rates are more likely to be affected) found a 39% reduction in mortality during the Scottish Patient Safety Programme, which emphasized use of the checklist as a key element of safe surgical care. No such changes were seen in non-operative admissions, decreasing the likelihood that a more generalized improvement in care was responsible for changes in outcomes.13

Team communication

A further dive into the checklist’s mechanism can be found in an analysis of the Safe Surgery 2015 program in South Carolina, a statewide voluntary collaborative. A structured implementation pathway was provided for the participants, as well as coaching on implementation, team training, and collaborative peer support.14 Using statewide discharge data, hospitals that completed the program saw a 22% reduction in mortality after inpatient surgery—changes that were not seen in other hospitals in the state.15 Additionally, hospitals that completed the program showed changes in team based surgical practices (for example, improvements of 12% in communication and 3% in coordination), and the greater the improvement in these attributes, the greater the associated perioperative mortality reduction.1617

The checklist was intentionally designed as a tool to strengthen team communication, and these data suggest that it does precisely this. However, evidence from the Canadian study illustrates how important the approach to implementation is for effective use of the checklist, as does analysis of its implementation by NHS England, where early mandate of the checklist without implementation support resulted in very high rates of documentation but only patchy clinically meaningful implementation.121819

Some may argue that the effect of the checklist can’t be separated from the implementation program and associated training. Teams with the capacity for improvement are the most likely to effectively integrate a complex behavioral intervention such as the checklist into their workflow, leading to the criticism that studies of checklist implementation simply identify “improvers.”

We do not disagree. However, even highly motivated teams need tools for implementation, and the evidence suggests that the WHO surgical safety checklist is among the most powerful tools for improving the safety of surgical care introduced in recent years. We must endeavor to learn how to maximize its effective implementation in environments around the world, improve it, and continue to evaluate results.