Publishing cardiac surgery mortality rates: lessons for other specialtiesBMJ 2013; 346 doi: https://doi.org/10.1136/bmj.f1139 (Published 28 February 2013) Cite this as: BMJ 2013;346:f1139
- Ben Bridgewater, professor124,
- Graeme L Hickey, researcher2,
- Graham Cooper, consultant3,
- John Deanfield, professor4,
- James Roxburgh, consultant5
- on behalf of the Society for Cardiothoracic Surgery in Great Britain and Ireland and the National Institute for Clinical Outcomes Research, UCL
- 1Department of Cardiothoracic Surgery, University Hospital of South Manchester, Manchester M23 9LT, UK
- 2Northwest Institute for BioHealth Informatics, Manchester Academic Health Science Centre, University of Manchester, Manchester, UK
- 3Department of Cardiothoracic Surgery, Northern General Hospital, Sheffield, UK
- 4National Institute for Cardiovascular Outcomes Research, Institute of Cardiovascular Science, London, UK
- 5Department of Cardiothoracic Surgery, St Thomas’ Hospital, London, UK
- Correspondence to: B Bridgewater
- Accepted 14 February 2013
The National Health Service Commissioning Board recently announced that mortality rates and other outcomes of surgery will be published for 10 specialties by summer 2013.1 This comes 12 years after the public inquiry into paediatric cardiac surgery at Bristol Royal Infirmary recommended that results by clinical team should be published to drive quality and prevent failures of clinical governance.2 UK cardiac surgeons were working towards this when the Freedom of Information Act was introduced in 2005. This led to a request from the Guardian newspaper for surgeons’ outcomes from all hospitals, which were published in April 2005.3 The Society for Cardiothoracic Surgery in Great Britain and Ireland (SCTS) has published mortality rates for all NHS hospitals and about 80% of surgeons since then.4 5 This programme has been associated with clear improvements in risk adjusted mortality without obvious adverse consequences.6
The 10 specialties named by the Commissioning Board all have national registries supported by public money through the Healthcare Quality Improvement Partnership. Going from a clinical registry to healthcare provider reports sounds simple, but in practice there are several challenges to overcome. Here we report our experiences in adult cardiac surgery to help inform this process.
Collecting and analysing data
The UK National Adult Cardiac Surgery Audit is managed by the National Institute for Cardiovascular Outcomes Research, with analytical input from Manchester University supported by charitable funding. A dedicated member of the Society of Cardiothoracic Surgeons oversees a team of database managers, an audit project manager, and data analysts. The project manager liaises with hospitals and database technicians to ensure that data are uploaded and validated. In collaboration with clinicians, the data analysts clean the data and apply risk adjustment methods to set a performance standard and define acceptable and unacceptable variation from this standard. Key strategic and operational decisions are made by the society’s executive, which includes patient, trainee, and nursing representation, with the decisions ratified by its membership through twice yearly meetings of surgical representatives from each hospital and an annual meeting of all members.
Several decisions had to be made about how best to analyse and interpret results (table⇓).7 Our experiences suggest that professional organisations should engage with this process and are well placed to maximise the benefits of audit and minimise the potential for negative consequences.8 9 Like many other professional groups, the society is funded solely through the financial contribution of its members, and members elect the officers who run it. Imposing a programme of robust data collection and governance has been challenging politically, but we believe that setting standards and monitoring adherence to those standards must be a primary role of any professional group.
In the initial stages many of the meetings with the membership were bruising because a substantial proportion of surgeons either did not believe in the principle of data transparency or doubted that the methods for risk adjustment could deal adequately with differences in case mix between hospitals or surgeons. Other professional societies will need to balance accountability to their members with the responsibilities of leading their specialty and advancing standards.5 To get our programme to its current standing has required constant focus and leadership resilience. Being overt about the overall aims and using patient representatives in discussions have been essential to delivering the agenda. A recent survey of 110 of SCTS members showed that 94% of surgeons agreed that, “It is important for the Society to have a methodology that benchmarks surgeons’ outcomes.”5
The outcome measures chosen must be important to patients, clinically relevant, and measurable. For adult cardiac surgery we use in-hospital mortality, restricted to the hospital where surgery took place. Although death is now rare after cardiac surgery, it is important to patients and it occurs with sufficient frequency to be a useful discriminator between providers. It is relatively easy to define and measure, but for lower risk procedures patients may also be interested in other outcomes such as postoperative length of stay and complications.10 Our programme to measure mortality has driven a culture of data collection among surgeons, and we are now able to measure a variety of postoperative outcomes including new renal intervention, stroke, and re-exploration for bleeding. Linking national audit data with administrative data could potentially lead to collection of all readmission and re-intervention data.
Cardiac surgery has widely recognised subgroups (such as coronary artery surgery and aortic valve replacement surgery), but for simplicity we decided to use all cardiac surgery in the analyses. The overall observed mortality rate in NHS cardiac surgery units for the period under scrutiny is used as the standard for performance; in the April 2008-March 2011 analysis this was 2.7%.
Most outcomes are influenced by procedural case mix and preoperative patient characteristics, as well as provider performance. Failure to adjust for these may discourage surgeons and hospitals from treating high risk patients, but unless risk adjustment methods are based on current data they may provide false reassurance to providers or patients.7 Despite today’s patients requiring more complex procedures and being at greater risk, outcomes in the UK have continued to improve.11 As a result we found that existing risk adjustment models, such as the EuroSCORE model, were no longer adequate.12 We therefore used established statistical methods to develop a simple formula, easily interpretable by individual units, to recalibrate the model for current outcomes.13 Calibration of the model is subsequently assessed using statistical tests. GH provided the statistical input for this process and it was internally peer reviewed within the professional society and the National Institute for Clinical Outcomes Research. We stress that appropriate risk adjustment is probably the most important factor in any audit of outcomes from both a statistical and a political viewpoint.
It is also important to ensure that the risk model is adequate for specific patient subgroups, again to minimise the risk of inciting risk averse behaviour. We have found that our models do not adjust well for emergency patients (those who need surgery before the next scheduled operating list) and salvage patients (who require immediate surgery);14 these patients often have much to gain from successful surgery but their high risk makes them susceptible to the unintended negative consequences of governance.9 We therefore decided to exclude these cases from analyses to compare outcomes.
Missing data are a routine problem in healthcare. Units work with database managers to ensure that data are largely complete, focusing on variables included in historical risk models. When data are missing it is inappropriate to discard records on both statistical and governance grounds. Use of complex methods to impute data is widely advocated but causes the quality of data to decay over time. We therefore decided to impute all missing data with the value that gives the least possible increase in predicted risk—thus if data on mortality are missing, the patient is presumed to have died in hospital. This approach encourages data completeness as hospitals and surgeons automatically respond to apparently high mortality rates in data validation exercises by ensuring that collection of mortality data is complete. Although our approach has known statistical disadvantages, we have found it to be robust in data validation exercises.15 Moreover, the number of missing variables in recent analyses has fallen as surgeons have become more aware of the method used.
Defining acceptable and unacceptable variation
Once you embark on a programme to collect outcomes you have an obligation to scrutinise data to ensure that there are no potential areas of unacceptable performance. We have analysed risk adjusted mortality data against the standard of the national contemporary observed mortality. We have defined divergence on the basis of severity and frequency.5 When you compare outcomes with a standard there will always be some degree of variation because of natural sampling. To define this statistically, our thresholds for classification are based on approximate 95% and 99% two sided confidence intervals, presented in the form of funnel plots, which have the advantage of not implying an inherent league table (figs 1⇓ and 2⇓). We feel it is important to notify “low level” outliers (those who fall in between these two limits), whose results may be due to chance alone, as well as surgeons or hospitals that are outside the 99% limit.5 16 In addition, when trying simultaneously to classify surgeons, an additional one sided 95% confidence interval limit corrected for multiple comparisons is overlaid (fig 2⇓). For surgeons who fall above this limit, we can confidently say that the probability that this is a chance finding is less than 1 in 20.
We have seen larger variability in risk adjusted outcomes than would be expected by chance alone. This “overdispersion” might be attributable to the risk adjustment model or recalibration, lack of adjustment for unmeasured covariates (such as social deprivation), or even the chosen endpoint (for example, in-hospital rather than 90 day mortality). High volume and lack of case mix specific indicators might also contribute because our comparisons are across all operative groups when some surgeons and units have clear subspecialist practice. Overdispersion can lead to inappropriate classification of surgeons, and we have made appropriate adjustments for this.16 17
Notification and publication
The society’s elected president contacts each surgeon and hospital to notify them when mortality rates are higher than expected. Each time we have conducted analyses 5-10% of surgeons and hospitals appear to have high mortality rates, but only around 1% exceed the maximum threshold.7 In some circumstances these findings came as a surprise to the surgeons and units, and in others the problems had already been identified, explored, and resolved. Some surgeons who have been identified as outliers have stopped operating. In other circumstances high mortality rates are thought to have been related to unusual case mix.
We believe it is right to publish these data: results for all units have been provided, but publication at surgeon level has been voluntary, with about 80% of surgeons agreeing to make the data available. We obtain formal consent before publication. Previously, the results have been disseminated on the Care Quality Commission website, but our next publication will provide information in the form of funnel plots and a description of case mix and will be published on our website (www.scts.org). We are offering hospitals with higher mortality rates than expected the opportunity to publish a commentary. As a professional society, we have access to the data and have conducted the analyses, but we do not know the cause of any divergence observed. High risk adjusted mortality rates do not necessarily indicate bad surgery. Divergence may be due to incorrect data, unusual case mix not reflected by risk models, institutional factors, individual issues, or a combination.7 Local investigation is needed to understand the cause of divergence, with or without involvement from other organisations. We believe that our role in publication is to make the data available, not to make judgments on performance.
Implementation of professional revalidation should be about setting standards for practice and then monitoring to show that practice is acceptable. This is what the society’s programme of governance does. Surgeons whose outcomes are within the limits can provide the data that will enable the responsible officer to issue a “positive recommendation” about revalidation (at least with respect to their clinical outcomes).18 Those whose rates are higher than expected will need to show the responsible officer that there are reasons to explain this and that patients are not at risk. If the officer is not convinced, he or she will either introduce a programme to ensure that patient safety is maintained while performance is improved (submit a revalidation deferral request to the GMC) or issue a “notification of non-engagement in revalidation.” Although this will be challenging for both surgeons who are outliers and the management structure of the hospitals in which they work, it is revalidation in action. The judgments are all underpinned by the complex decisions about the methods of risk adjustment and analysis that lead to definition of outlier status in the first place.
The society has tried to ensure that its methodological and other decisions are subjected to external scrutiny and publication where appropriate, and it has consulted and communicated widely with its membership. There is not one correct way of defining or publishing outliers; different decisions would lead to different conclusions. We encourage all professional groups to embrace publication of outcomes, as there is accumulating evidence that it can have substantial benefits for patients and clinicians by improving care and reinforcing trust.19 The process of clinical engagement from those managing the audits must also be robust. Unless there is strong and informed clinical and methodological input there remains a significant risk of unintended negative consequences.
Cite this as: BMJ 2013;346:f1139
This manuscript has been prepared on behalf of all members of the Society for Cardiothoracic Surgeons in Great Britain and Ireland (SCTS). We acknowledge all those involved in the collection and validation of cardiac surgery data and the National Institute for Cardiovascular Outcomes Research for its role in project management and managing the technical infrastructure of the data for the audit programme.
Contributors and sources: BB, GC, and JR are consultant cardiac surgeons and came up with the idea for this article. BB is the SCTS database chairman and oversees the management and analysis of the governance programme. GC is the SCTS honorary secretary and JR is the president. GLH is a biostatistician involved in cleaning and analysing the SCTS database for research and governance purposes. JD is codirector of the National Institute for Clinical Outcomes Research. The manuscript was prepared by BB and GLH with contributions from all authors. BB will act as guarantor.
Funding: GH’s salary is funded by a grant from Heart Research UK (Grant No RG2583).
Competing interests: We have read and understood the BMJ Group policy on declaration of interests and have no relevant interests to declare.
Provenance and peer review: Not commissioned; externally peer reviewed.