Intended for healthcare professionals

  1. Research
  2. Effect of public...
  3. Effect of public reporting of surgeons’ outcomes on patient selection, “gaming,” and mortality in colorectal cancer surgery in England: population based cohort study
CCBYNC Open access
Research

Effect of public reporting of surgeons’ outcomes on patient selection, “gaming,” and mortality in colorectal cancer surgery in England: population based cohort study

BMJ 2018; 361 doi: https://doi.org/10.1136/bmj.k1581 (Published 02 May 2018) Cite this as: BMJ 2018;361:k1581
  1. Abigail E Vallance, clinical research fellow1,
  2. Nicola S Fearnhead, consultant colorectal surgeon2,
  3. Angela Kuryba, data analyst1,
  4. James Hill, professor of surgery3 4,
  5. Charles Maxwell-Armstrong, consultant colorectal surgeon5,
  6. Michael Braun, consultant medical oncologist6,
  7. Jan van der Meulen, professor of clinical epidemiology1 7,
  8. Kate Walker, lecturer of medical statistics1 7
  1. 1Clinical Effectiveness Unit, Royal College of Surgeons of England, London WC2A 3PE, UK
  2. 2Cambridge University Hospitals NHS Foundation Trust, Addenbrookes Hospital, Hills Road, Cambridge CB2 0QQ, UK
  3. 3Manchester Royal Infirmary, Manchester M13 9WL, UK
  4. 4Manchester Academic Health Science Centre, University of Manchester, Manchester M13 9NT, UK
  5. 5National Institute for Health Research, Nottingham Digestive Diseases Biomedical Research Unit, Nottingham University Hospitals NHS Trust and University of Nottingham, Nottingham NG7 2UH, UK
  6. 6The Christie NHS Foundation Trust, Manchester M20 4BX, UK
  7. 7Department of Health Services Research and Policy, London School of Hygiene and Tropical Medicine, London WC1H 9SH, UK
  1. Correspondence to: J van der Meulen jan.vandermeulen{at}lshtm.ac.uk
  • Accepted 16 March 2018

Abstract

Objective To determine the effect of surgeon specific outcome reporting in colorectal cancer surgery on risk averse clinical practice, “gaming” of clinical data, and 90 day postoperative mortality.

Design National cohort study.

Setting English National Health Service hospital trusts.

Population 111 431 patients diagnosed as having colorectal cancer from 1 April 2011 to 31 March 2015 included in the National Bowel Cancer Audit.

Intervention Public reporting of surgeon specific 90 day mortality in elective colorectal cancer surgery in England introduced in June 2013.

Main outcome measures Proportion of patients with colorectal cancer who had an elective major resection, predicted 90 day mortality based on characteristics of patients and tumours, and observed 90 day mortality adjusted for differences in characteristics of patients and tumours, comparing patients who had surgery between April 2011 and June 2013 and between July 2013 and March 2015.

Results The proportion of patients with colorectal cancer undergoing major resection did not change after the introduction of surgeon specific public outcome reporting (39 792/62 854 (63.3%) before versus 30 706/48 577 (63.2%) after; P=0.8). The proportion of these major resections categorised as elective or scheduled also did not change (33 638/39 792 (84.5%) before versus 25 905/30 706 (84.4%) after; P=0.5). The predicted 90 day mortality remained the same (2.7% v 2.7%; P=0.3), but the observed 90 day mortality fell (952/33 638 (2.8%) v 552/25 905 (2.1%)). Change point analysis showed that this reduction was over and above the existing downward trend in mortality before the introduction of public outcome reporting (P=0.03).

Conclusions This study did not find evidence that the introduction of public reporting of surgeon specific 90 day postoperative mortality in elective colorectal cancer surgery has led to risk averse clinical practice behaviour or “gaming” of data. However, its introduction coincided with a significant reduction in 90 day mortality.

Introduction

In 2012 NHS England, the national body that coordinates the commissioning of health services in the English National Health Service (NHS), called for quality measures and mortality data to be made publicly available.1 In response, public reporting of named surgeon specific outcomes was introduced in June 2013 across nine surgical specialties, with the data being provided by pre-existing national clinical audits. Public reporting has now been extended to most surgical and some interventional specialties.

Since the reporting of surgeon specific outcomes was introduced in 2013, case numbers and risk adjusted 90 day mortality rates in patients undergoing an “elective” or “scheduled” major colorectal cancer resection have been reported for individual consultant colorectal surgeons. These rates are publicly available on the websites of the Association of Coloproctology of Great Britain and Ireland (ACPGBI) and NHS Choices.2 The results are based on analyses of data provided by the National Bowel Cancer Audit (NBOCA).3 This national audit has been publishing surgical mortality results at hospital trust level in its annual reports since 2010.4

The publication of surgeon specific outcomes has been controversial.567 Proponents believe that surgeon specific outcome reporting facilitates transparency, allows patients to make informed choices, and may drive quality improvement.8 Critics argue that the potential professional and financial implications of the public reporting of perceived negative outcomes encourages risk averse behaviour, whereby surgeons are less likely to offer surgery to patients at higher risk. Data may also be manipulated to increase patients’ predicted risk or to make patients ineligible for public reporting, which is often referred to as “gaming.” So far, the evidence that reporting of surgeon specific outcome leads to improvements in the quality of patient care is surprisingly weak. The effect of public reporting of surgeon specific outcomes has been studied only in cardiac surgery, and these studies were almost exclusively carried out in the United States [8].

In this study, we analysed NBOCA data to look for evidence of risk averse behaviour, manipulation of data, and change in surgical mortality in the time period immediately before and after the introduction of surgeon specific outcome reporting in colorectal cancer surgery.

This is the first time that the effect of public reporting of surgeon specific outcomes has been presented outside cardiac surgery. Uniquely, we studied its effect on all patients with colorectal cancer diagnosed during the study period, irrespective of whether they underwent elective major surgery. In this way, we overcame a major limitation in all published studies to date that included only patients who had surgery and therefore could not assess the effect of risk averse behaviour and changes in patient selection.

Methods

Data collection and reporting of surgeon specific outcomes

NHS hospital trusts in England have a mandatory requirement to submit data for all patients with a new diagnosis of colorectal cancer via a secure online platform to the NBOCA database. Information is collected that captures surgical urgency (categorised as elective, scheduled, urgent, or emergency according to the National Confidential Enquiry into Patient Outcome and Death (NCEPOD) classification of intervention),9 the American Society of Anesthesiologists (ASA) grade, pathological TNM staging, cancer site within the colon or rectum, and surgical procedure performed.

Data from NBOCA is linked to Hospital Episode Statistics (HES), the administrative database of all admissions to NHS hospitals in England. Information on admission type (elective or emergency), the formation of a stoma, and comorbidities is identified from these linked HES records. The Royal College of Surgeons’ Charlson score is used to identify comorbid conditions in the HES records in the preceding year.10 The date of death is obtained from linked data from the Office for National Statistics (ONS).11

All elective or scheduled major colorectal cancer resections in patients aged 18 years and older at diagnosis, and listed under a consultant surgeon’s General Medical Council (GMC) code, are eligible for surgeon specific outcome reporting. This code is a unique identifier that is available for all clinicians registered with the GMC, the body that maintains the official register of medical practitioners in the UK.12 In the UK, a consultant surgeon is a surgeon who has completed all his or her specialist training and has been placed on the GMC’s Specialist Register.

Reports of the eligible patients are sent to the NHS hospital trusts, which allows them to check the data before analysis. Hospital trusts are notified of observed and adjusted 90 day mortality rates at the level of both the surgeon and the trust before publication. Surgeons’ outcomes are not published individually for surgeons who have carried out fewer than 10 major cancer resections within the reporting period.

Study population and outcome measures

We included in this study patients submitted to the NBOCA dataset and diagnosed as having primary colorectal cancer from 1 April 2011 to 31 March 2015. The quality of NBOCA data improved considerably from 1 April 2011, which was why we chose this start date. Data collection for metastatic disease was changed from Dukes’ stage to M stage in 2013. This change corresponded with a drop in data completeness.

Risk aversion in bowel cancer surgery may manifest as surgeons becoming less willing to offer patients a surgical resection, especially patients deemed to be at higher risk. In addition, surgeons may be more likely to refer patients to hospital trusts deemed to be more specialised. Also, in treating rectal cancer, surgeons may be more inclined to offer patients a surgical procedure associated with lesser morbidity and mortality, such as performing a defunctioning stoma at the time of anterior resection or preferentially performing an abdominoperineal excision of rectum or Hartmann’s procedure over restoring intestinal continuity. Surgeons may also be more disposed to offer minimally invasive procedures, such as endoscopic mucosal resection or transanal endoscopic microsurgery (which are not included in public reporting).

To investigate risk averse behaviour, we therefore studied the overall rate of major resection, the characteristics of patients undergoing major resection and their tumours, the predicted 90 day mortality according to characteristics of patients and tumours, the proportion of patients undergoing surgery at a different hospital trust from the one at which their tumour was diagnosed, and the type of treatment in patients with rectal cancer. We compared all of these before and after the introduction of public reporting of outcomes.

The deliberate manipulation, or gaming, of data may manifest in data submitted to the audit that is more “subjective” in nature, including the classification of surgical urgency and ASA grade. Gaming of surgical urgency data may result in an increase in the number of patients who had elective surgery being reclassified as “urgent” or “emergency,” and therefore recorded as ineligible for surgeon specific outcome reporting. Of the nine variables included in the risk adjustment score for 90 day mortality,13 only ASA grade can be considered subjective. Consequently, a tendency may exist to report higher ASA grades in patients undergoing elective surgery, which would lead to higher predicted surgical 90 day mortality and, in turn, a lower overall adjusted mortality. To assess for evidence of gaming of data, we compared the proportion of patients undergoing major resection who were classified as urgent or emergency (and therefore ineligible for surgeon specific outcome reporting) and the ASA grade recorded in patients undergoing elective major resection, before and after the introduction of surgeon specific outcome reporting.

To determine the effect of surgeon specific outcome reporting on outcomes for patients eligible for inclusion in public outcome reporting, we compared the observed 90 day postoperative mortality without and with adjustment for characteristics of patients and tumours in patients undergoing elective or scheduled major resection. We included patients ineligible for public outcome reporting as comparison groups. For this purpose, we studied the observed and adjusted 90 day postoperative mortality for patients undergoing urgent or emergency major resection and the observed six month survival after diagnosis of patients not undergoing major resection. We chose a longer analysis period of six months after diagnosis for these patients to reflect the fact that most would not undergo surgery and would have received palliative management.

Statistical analysis

The risk adjustment model used by NBOCA was developed and validated previously.13 The risk factors included in this logistic regression model are age, sex, ASA grade, Charlson comorbidity score, mode of admission, site of tumour, and pathological T stage, N stage, and M stage, with an additional interaction between age and M stage. We modelled a random intercept for each hospital trust to reflect the possible clustering of results within trusts. Missing values for the risk factors were imputed with multiple imputation using chained equations creating 10 datasets and using Rubin’s rules to combine the model estimates across the datasets. An adjusted outcome was then produced by indirect standardisation.14 We compared proportions before and after the introduction of public reporting with a χ2 test, median age with the Kruskal-Wallis test, and differences in predicted 90 day mortality with a t test.

We used a change point analysis to study the change over time in adjusted 90 day mortality after elective/scheduled major resection (eligible for surgeon specific outcome reporting) and after urgent/emergency major resection (ineligible for surgeon specific outcome reporting).15 The change point was fixed to the date of the introduction of surgeon specific outcome reporting (28 June 2013). We used a multivariable logistic regression model for 90 day mortality, with a slope for calendar time and an interaction between time pre-introduction versus post-introduction of surgeon specific outcome reporting, in addition to all of the risk adjustment variables. This modelled a change in the slope of mortality at the point that surgeon specific outcome reporting was introduced but no immediate change in mortality.

As a sensitivity analysis, we fitted two further change point models to the same data. The first modelled an immediate shift in mortality at the introduction of surgeon specific outcome reporting by including a term for before and after the introduction of public reporting. The second modelled both an immediate shift and a change in the slope. We used Stata version 14.1 for all analyses.

Patient involvement

No patients were involved in setting the research question or the outcome measures, nor were they involved in developing plans for recruitment, design, or implementation of the study. No patients were asked to advise on interpretation or writing up of results. The results of this research will be publicly disseminated via the National Bowel Cancer Audit website as well as through the charity and patient representatives who act as part of the audit’s Clinical Advisory Group.

Results

Risk averse behaviour

We identified 111 431 patients diagnosed as having colorectal cancer during the study period. Of the 62 854 patients who received a diagnosis before the introduction of public reporting, 39 792 (63.3%) had a major resection, and this was the case for 30 706 (63.2%) of the 48 577 patients who received a diagnosis thereafter (P=0.8) (fig 1).

Fig 1
Fig 1

Proportion of patients undergoing major resection according to date of diagnosis. Dotted line represents date of introduction of surgeon specific public outcome reporting

The characteristics of the 33 638 patients who had an elective or scheduled major resection before the introduction of public reporting and the 25 905 patients who had one afterwards differed little (table 1). The proportion of patients with metastatic disease decreased from 9.6% before to 7.2% after the introduction (P<0.001). The mean predicted 90 day mortality based on characteristics of patients and tumours did not change (2.7% before and 2.7% after the introduction of public reporting; P=0.3) (fig 2). The proportion of patients undergoing surgical resection at a different hospital trust from where they were given a diagnosis slightly reduced; 1981/33 638 (5.9%) patients given a diagnosis before introduction and 1401/25 905 (5.4%) patients given a diagnosis afterwards were treated in a different hospital (P=0.01).

Table 1

Clinico-pathological characteristics of patients undergoing elective or scheduled major resection from 1 April 2011 to 31 March 2015 according to year of surgery. Values are numbers (percentages) unless stated otherwise

View this table:
Fig 2
Fig 2

Predicted 90 day mortality for elective and scheduled major resection according to date of surgery. Dotted line represents date of introduction of surgeon specific public outcome reporting

We found no evidence of a change in the care received by patients with rectal cancer. Of the 18 700 patients diagnosed as having rectal cancer before the introduction of public reporting, 1274 (6.8%) had a local excision compared with 1012 (7.0%) of the 14 457 diagnosed after the introduction (P=0.6). Of the rectal cancer patients undergoing major resection, no increase occurred in the proportion of patients undergoing procedures with no primary anastomosis (Hartmann’s procedure or abdominoperineal excision of rectum) with the introduction of surgeon specific outcome reporting (3410/10 206 (33.4%) before and 2735/7871 (34.7%) after; P=0.06). The proportion of patients with rectal cancer undergoing anterior resection with the addition of a defunctioning stoma was 4748/6430 (73.8%) before introduction compared with 3466/4788 (72.4%) after introduction (P=0.09).

Gaming of data

We found no evidence of gaming of data (fig 3). The proportion of major resections categorised as urgent or emergency—and therefore ineligible for surgeon specific outcome reporting—did not change, as 6154 (15.5%) of 39 792 major resections were categorised as urgent or emergency before and 4801 (15.6%) of 30 706 major resections after the introduction of public reporting (P=0.5). The distribution of ASA grades did not change before and after introduction (P=0.08) (table 1).

Fig 3
Fig 3

Proportion of major resections classified as urgent or emergency (therefore ineligible for surgeon specific outcome reporting) according to date of diagnosis. Dotted line represents date of introduction of surgeon specific public outcome reporting

Mortality in patients eligible for inclusion in surgeon specific outcome reporting

The 90 day mortality in patients undergoing an elective or scheduled major resection fell during the study period from 952/33 638 (2.8%) before the introduction of surgeon specific outcome reporting to 552/25 905 (2.1%) after (fig 4). Therefore, we carried out change point analysis which showed a steeper decline in 90 day mortality after the introduction of public reporting (P=0.03). The change point analysis also found a significant effect of public reporting when it was modelled as an immediate shift in 90 day mortality (P=0.01) and when it was modelled as both an immediate shift and a change in slope (P=0.04).

Fig 4
Fig 4

Observed and adjusted mortality in patients undergoing elective or scheduled major resection according to date of surgery. Dotted line represents date of introduction of surgeon specific public outcome reporting

Mortality in patients not eligible for inclusion in surgeon specific outcome reporting

We saw no change in the slope of 90 day mortality in patients undergoing urgent or emergency surgery with the introduction of surgeon specific outcome reporting (941/6154 (15.3%) before and 749/4801 (15.6%) after; P=0.3) (fig 5). The six month mortality of patients not undergoing major resection remained static over the study period; 7514/23 062 (32.6%) had died six months after diagnosis before the introduction of surgeon specific outcome reporting and 5728/17 871 (32.1%) after, as shown in figure 6.

Fig 5
Fig 5

Observed and adjusted mortality in patients undergoing urgent or emergency major resection according to date of surgery. Dotted line represents date of introduction of surgeon specific public outcome reporting

Fig 6
Fig 6

Observed six month mortality (from diagnosis) in patients not undergoing major resection. Dotted line represents date of introduction of surgeon specific public outcome reporting

Discussion

This study, using data from a high profile, national service evaluation project, found no evidence that the introduction of surgeon specific public reporting of 90 day mortality after elective colorectal surgery has led to risk averse behaviour or gaming of data in the English NHS. The proportion of patients undergoing major resection, the proportion of these patients potentially eligible for outcome reporting, and the predicted risk in those undergoing major resection remained the same. However, the introduction of the public reporting of outcomes coincided with a reduction in the 90 day mortality, exceeding the existing downward trend, in patients who were included in the outcome reporting programme—namely, those patients who had an elective or scheduled major resection. No effect on mortality was seen in patients who were not included—colorectal cancer patients who had an urgent or emergency major resection or those who did not have a major resection—showing that outcomes in these groups were not negatively affected.

Strengths and weaknesses of study

This is the first study to evaluate the effect of the public reporting of surgeons’ outcomes in gastrointestinal surgery. Previous studies assessing the effect of public reporting of surgeon specific outcomes were limited to cardiac surgery.16171819 It is also the first study to compare the effect of surgeon specific outcome reporting on all patients with a specific diagnosis, rather than only those undergoing a specific procedure, so that the effect of risk aversive behaviour and selection of patients could also be studied.

We identified more than 111 000 patients who had colorectal cancer in a national database, almost 60 000 of whom had undergone an elective major resection. This group represents 92% of all colorectal cancer patients admitted to an English NHS hospital.3 This national coverage and the high “case ascertainment” reduces the risk of selection bias.

Clinicians had little opportunity to manipulate the outcome data. We obtained mortality data through record linkage with death records provided by the Office for National Statistics. Other clinical data were entered by audit staff from patients’ records. Clinical teams have a role in this process, but surgeons themselves do not have individual access rights to enter or edit the data themselves. Nevertheless, it is conceivable that a patient’s operative urgency could retrospectively be changed from “scheduled” to “urgent,” which would make the patient ineligible for outcome reporting, or that a patient’s ASA grade could be increased, which would decrease the 90 mortality adjusted for patients’ risk factors. However, we were able to examine whether the data entered for patients diagnosed as having colorectal cancer up until June 2013 had changed after the introduction of outcome reporting, and we found no difference in the distribution of reported ASA grades or the number of patients undergoing elective or scheduled major resection.

In studies using a “before-after” design, changes may occur over time in the quality of the data and in clinical factors that may have an effect on the study results. We tried to overcome this limitation by rigorously checking differences in the accuracy and completeness of the data as part of our assessment of possible gaming of data before and after the introduction of public reporting. Also, we aimed to overcome differences in patients’ characteristics through adjustment for possible differences in the risk profile of patients who had surgery before and after the introduction of public reporting, on the basis of a previously published validated prognostic model.13 Lastly, we used change point modelling to take into account the fact that outcomes of colorectal cancer surgery are gradually improving over time. We found that the introduction of public reporting had a significant effect on surgical mortality, irrespective of whether this effect was modelled as a change in the slope (representing a faster decrease in mortality after the introduction of public reporting), an immediate shift in mortality, or a combination of these effects.

Risk averse behaviour

The use of risk adjustment methods that adequately ensure that providers who treat patients at higher risk are not unfairly penalised is critical in public reporting of surgical outcomes for several reasons. Firstly, if surgeons do not fully accept the accuracy of these methods, they may be more inclined to avoid carrying out major surgical procedures in patients at higher risk.20 A recent survey conducted across 293 English colorectal surgeons asking about their views on surgeon specific outcome reporting found that almost all had concerns about the possibility that it could lead to colleagues avoiding operating on patients at high risk.21 These concerns are not supported by the results of our study. For example, we found no change in the expected mortality risk before and after the introduction of public reporting of mortality outcomes.

Evidence of risk averse behaviour after the introduction of surgeon specific outcome reporting is mixed in cardiac surgery. Patients undergoing cardiac bypass procedures in states in America with public reporting of outcomes were found to have lower illness severity than patients in states that did not publicly release such data.22 In contrast, in the UK, the average predicted mortality of patients undergoing cardiac surgery seemed to increase rather than decrease in the years immediately after the introduction of surgeon specific outcome reporting.17

Although the overall risk of patients undergoing elective colorectal cancer resection was unchanged across the study period, the proportion of patients undergoing primary resection with metastatic disease was slightly reduced. Recent population based studies carried out in the US and in Europe found a similar decrease in the percentage of metastatic colorectal cancer patients undergoing primary tumour resection, which suggests that this decrease reflects an overall change in practice rather than risk averse behaviour.2324

Gaming of data

Manipulation of data has been reported in studies investigating the effect of public reporting of individual surgeons’ outcomes. A study carried out in the 1990s evaluating the effect of surgeon specific outcome reporting in New York State found that about 40% of the reduction in risk adjusted mortality from coronary artery bypass grafting could be explained by the gaming of data, either as a result of an increase in risk factor coding or a change in definitions.25

To reduce the possible effect of deliberately increases in the ASA grade recorded for a patient, the Association of Coloproctology of Great Britain and Ireland has recommended that the ASA grade of each patient should be determined by the anaesthetist rather than by the surgeon and that this grade should be recorded before the resection of the cancer.26 This may in part explain why we did not observe any change in the proportion of patients with higher ASA grades among patients undergoing an elective major resection after the introduction of public surgeon specific reporting.

Mortality

The introduction of surgeon specific outcome reporting coincided with a decrease in mortality after elective major colorectal cancer surgery over and above the pre-existing downward trend in mortality. This study did not show a similar reduction in mortality in patients undergoing emergency colorectal cancer resection after the introduction of surgeon specific public outcome reporting. This suggests that changes to preoperative preparation and planning of perioperative and postoperative care—for example, through the implementation of programmes aiming to enhance recovery after surgery27—may have contributed to the improved mortality rate we have shown. Preoperative cardiopulmonary exercise testing, which can only be implemented in patients who undergo an elective or scheduled procedure, may have led to changes in the delivery of postoperative care, such as the increased use of level 2 and level 3 beds in patients at high risk.2829 Rather than resulting in risk aversion, public reporting of outcomes may have focused attention on an individual patient’s risk and ways of minimising that risk.

Technical aspects of the surgical procedure are unlikely to have improved dramatically over the study period, but it is certainly conceivable that the focus on individual consultant surgeons’ practice may have indirectly led to greater involvement of surgeons in the patients’ actual preoperative and postoperative care, with more proactive management of complications. Surgical trainees have also highlighted that public reporting of outcomes has reduced training opportunities in patients eligible for public reporting.7

Effect of public reporting of surgeon specific outcome data

It has been argued that public reporting of outcomes for hospitals or clinicians can have many uses.30 Firstly, it may facilitate patient choice in selecting a specific hospital or clinician that seems to have better outcomes. Evidence from several countries such as the UK, the US, and the Netherlands shows that younger and more affluent patients are more likely to have their treatment in other hospitals than the one closest to where they live.31 Secondly, outcome data may influence decisions of purchasers or commissioners of health services either with the aim of controlling costs or to obtain the best outcomes within a limited budget. Thirdly, regulators of healthcare services, such as the Care Quality Commission in England, can consider reported outcomes when they monitor the quality and safety of care delivered by individual providers. Finally, hospitals and clinicians can use the outcomes to assess the performance of their organisation or their individual staff members when implementing quality improvement initiatives.

The outcomes of surgeons performing fewer than 10 elective or scheduled major resection cases a year are not reported in the public domain, but the outcomes for these patients are included in this analysis. Although it is feasible that due to surgeon specific outcome reporting the number of surgeons performing the occasional resection has decreased, therefore resulting in improved outcomes for patients, we have not seen this reflected in NBOCA data—the proportion of elective or scheduled cases by surgeons performing fewer than 10 major resection a year has remained constant (5.0% pre-outcome reporting, 5.2% post-outcome reporting).

We found a drop in mortality of patients who had elective colorectal surgery immediately after the introduction of surgeon specific public outcome reporting. This immediacy of the effect suggests a direct response by the hospitals and clinicians themselves rather than a more indirect response prompted by choices that patients make or by actions from purchasers or regulators. Similarly, although other major improvements in care of patients with colorectal cancer have been made during the study period, such as advances in chemotherapy, these changes are more likely to affect overall survival rather than specifically 90 day postoperative mortality in patients having elective and scheduled surgery.

However, surgical outcomes are influenced by many organisational factors, and an individual surgeon’s results may therefore merely be a “surrogate reflection” of the hospital’s overall performance.32 Also, the number of bowel cancer procedures that a bowel cancer surgeon does each year is low, as is the power to detect a surgeon who is performing poorly. Previously, we have argued that a major risk of public reporting of individual surgeons’ outcomes is that it could lead to false complacency.33

What general lessons can be learnt from these results? If we assume that the decrease in mortality can be causally linked to the reporting or individual surgeons’ outcomes, the process of reporting itself rather than a response to particular outcomes may have triggered a change in how patients who are scheduled to have major colorectal cancer surgery are managed, not only by the surgeon but by the entire team responsible for a patient’s care, which could be considered as a manifestation of the Hawthorne effect.34 This team response could have been mediated through an individual surgeon’s “heightened responsibility,” which in turn may have galvanised the entire team involved in managing patients before, during, and after elective bowel cancer surgery.

This interpretation provides a more nuanced position in the debate about whether outcome results should be published for clinicians or for teams or hospitals. A further consideration is that the specific circumstances of elective surgery may also play a role, given the emerging evidence for the effectiveness of enhanced recovery programmes and cardiopulmonary exercise testing, which provide possible routes to improve outcomes of patients undergoing scheduled procedures.35 In this context, an increase in the use of radiotherapy before surgery could also play a role for patients with rectal cancer. Unfortunately, we do not have evidence on increases in the use of radiotherapy over time.36

Conclusions

In an era of ongoing “assessment and accountability,”37 consultant colorectal cancer surgeons who practise in the English NHS have an obligation to engage with individual public outcome reporting. The potential positive and negative consequences of public reporting are widely speculated on but are largely unexplored.30 This national study provides unique evidence that the introduction of public reporting of outcomes for individual colorectal cancer surgeons has not led to a decrease in the number of patients at high risk undergoing a major resection and has coincided with an improvement in 90 day mortality for eligible patients.

Public reporting of outcomes for individual clinicians seems to have triggered an improvement in outcomes after elective procedures that can be achieved only through the involvement of the entire clinical team. We did not find similar improvements in the outcome of emergency procedures. This points to improved preoperative preparation and planning of perioperative and postoperative care, only possible for elective procedures, as a potential possible explanation for the reduction in surgical mortality.

What is already known on this topic

  • Public reporting of outcomes at hospital level leads to increased quality improvement activity

  • The introduction of public reporting of individual surgeons’ outcomes has been controversial but has been studied only in cardiac surgery and almost exclusively in the US

  • Public reporting of outcomes is often criticised for encouraging risk averse behaviour and manipulation of data

What this study adds

  • No evidence exists of risk averse clinical practice or “gaming” of data after the introduction of surgeon specific outcome reporting in elective colorectal cancer surgery in England

  • The introduction of surgeon specific outcome reporting coincided with a significant decrease in mortality, which exceeded the existing downward trend

  • Outcomes in patients having emergency surgery for bowel cancer did not change, suggesting improvement in preoperative preparation and planning of perioperative and postoperative care

Acknowledgments

Hospital Episode Statistics data were made available by NHS Digital.

Footnotes

  • Contributors: AEV conceived and designed the study, analysed the data, and drafted and revised the paper. AK prepared and analysed the data. JH designed the study and drafted and revised the paper. NSF conceived and designed the study, interpreted the results, and drafted and revised the paper. CM-A drafted and revised the paper. MB designed the study and drafted and revised the paper. JvdM designed the study, interpreted the results, and drafted and revised the paper. KW designed the study, analysed the data, and drafted and revised the paper. All authors, external and internal, had full access to all of the data (including statistical reports and tables) in the study and can take responsibility for the integrity of the data and the accuracy of the data analysis. JvdM is the guarantor.

  • Funding: The National Bowel Cancer Audit is commissioned by the Healthcare Quality Improvement Partnership (HQIP) as part of the National Clinical Audit and Patient Outcomes Programme, and funded by NHS England and the Welsh Government (www.hqip.org.uk/national-programmes). Neither HQIP nor the funders had any involvement in the study design; in the collection, analysis, and interpretation of data; in the writing of the report; or in the decision to submit the article for publication. The researchers had full independence from the Healthcare Quality Improvement Partnership.

  • Competing interests: All authors have completed the ICMJE uniform disclosure form at www.icmje.org/coi_disclosure.pdf (available on request from the corresponding author) and declare: no support from any organisation for the submitted work; no financial relationships with any organisations that might have an interest in the submitted work in the previous three years; no other relationships or activities that could appear to have influenced the submitted work.

  • Ethical approval: As the National Bowel Cancer Audit involves analysis of data for service evaluation, it is exempt from the UK National Research Ethics Committee approval. Section 251 approval was obtained from the Ethics and Confidentiality Committee for the collection of the personal health data.

  • Transparency declaration: The lead author (AEV) affirms that the manuscript is an honest, accurate, and transparent account of the study being reported; that no important aspects of the study have been omitted; and that any discrepancies from the study as planned (and, if relevant, registered) have been explained.

  • Data sharing: No additional data available.

This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.

References

  1. ↵
    NHS England. Everyone counts: planning for patients 2013/14. 2012. https://www.england.nhs.uk/everyonecounts/.
  2. ↵
    NHS Choices. Consultants specialising in colorectal cancer surgery in England. 2017. https://www.nhs.uk/service-search/consultants/performanceindicators/1030.
  3. ↵
    NHS Digital. National Bowel Cancer Audit. http://content.digital.nhs.uk/bowel.
  4. ↵
    National Bowel Cancer Audit. Reports. nboca.org.uk.
  5. ↵
    1. Moonesinghe SR
    . Individualised surgical outcomes: please look the other way. Postgrad Med J2013;89:677-8. doi:10.1136/postgradmedj-2013-132442 pmid:24243981
    OpenUrlFREE Full Text
  6. ↵
    1. Khajuria A
    . Public reporting of surgeon outcomes in the United Kingdom: potential caveats. Int J Surg2014;12:369-70. doi:10.1016/j.ijsu.2014.02.001 pmid:24513014
    OpenUrlCrossRefPubMed
  7. ↵
    1. Radford PD,
    2. Derbyshire LF,
    3. Shalhoub J,
    4. Fitzgerald JE,
    5. Council of the Association of Surgeons in Training
    . Publication of surgeon specific outcome data: a review of implementation, controversies and the potential impact on surgical training. Int J Surg2015;13:211-6. doi:10.1016/j.ijsu.2014.11.049 pmid:25498494
    OpenUrlCrossRefPubMed
  8. ↵
    1. Alderson D,
    2. Cromwell D
    . Publication of surgeon-specific outcomes. Br J Surg2014;101:1335-7. doi:10.1002/bjs.9641 pmid:25196475
    OpenUrlCrossRefPubMed
  9. ↵
    National Confidential Enquiry into Patient Outcome and Death. The NCEPOD Classification of Intervention. 2004. http://www.ncepod.org.uk/classification.html.
  10. ↵
    1. Armitage JN,
    2. van der Meulen JH,
    3. Royal College of Surgeons Co-morbidity Consensus Group
    . Identifying co-morbidity in surgical patients using administrative data with the Royal College of Surgeons Charlson Score. Br J Surg2010;97:772-81. doi:10.1002/bjs.6930 pmid:20306528
    OpenUrlCrossRefPubMed
  11. ↵
  12. ↵
    1. NHS Classifications Service
    . OPCS Classifications of Interventions and Procedures Version 4.4.Department of Health, 2007.
  13. ↵
    1. Walker K,
    2. Finan PJ,
    3. van der Meulen JH
    . Model for risk adjustment of postoperative mortality in patients with colorectal cancer. Br J Surg2015;102:269-80. doi:10.1002/bjs.9696 pmid:25524216
    OpenUrlCrossRefPubMed
  14. ↵
    1. Spiegelhalter DJ
    . Funnel plots for comparing institutional performance. Stat Med2005;24:1185-202. doi:10.1002/sim.1970 pmid:15568194
    OpenUrlCrossRefPubMedWeb of Science
  15. ↵
    1. Chen J,
    2. Gupta AK
    . Parametric statistical change point analysis: with applications to genetics, medicine, and finance.Springer Science & Business Media, 2011.
  16. ↵
    1. Hannan EL,
    2. Kilburn H Jr.,
    3. Racz M,
    4. Shields E,
    5. Chassin MR
    . Improving the outcomes of coronary artery bypass surgery in New York State. JAMA1994;271:761-6. doi:10.1001/jama.1994.03510340051033 pmid:8114213
    OpenUrlCrossRefPubMedWeb of Science
  17. ↵
    1. Bridgewater B,
    2. Grayson AD,
    3. Brooks N,
    4. et al.,
    5. North West Quality Improvement Programme in Cardiac Interventions
    . Has the publication of cardiac surgery outcome data been associated with changes in practice in northwest England: an analysis of 25,730 patients undergoing CABG surgery under 30 surgeons over eight years. Heart2007;93:744-8. doi:10.1136/hrt.2006.106393 pmid:17237128
    OpenUrlAbstract/FREE Full Text
  18. ↵
    1. Khan OA,
    2. Iyengar S,
    3. Pontefract DE,
    4. Rogers V,
    5. Ohri SK,
    6. Livesey SA
    . Impact of surgeon-specific data reporting on surgical training. Ann R Coll Surg Engl2007;89:796-8. doi:10.1308/003588407X232080 pmid:17999823
    OpenUrlCrossRefPubMed
  19. ↵
    1. Li Z,
    2. Carlisle DM,
    3. Marcin JP,
    4. et al
    . Impact of public reporting on access to coronary artery bypass surgery: the California Outcomes Reporting Program. Ann Thorac Surg2010;89:1131-8. doi:10.1016/j.athoracsur.2009.12.073 pmid:20338320
    OpenUrlCrossRefPubMed
  20. ↵
    1. Resnic FS,
    2. Welt FGP
    . The public health hazards of risk avoidance associated with public reporting of risk-adjusted outcomes in coronary intervention. J Am Coll Cardiol2009;53:825-30. doi:10.1016/j.jacc.2008.11.034 pmid:19264236
    OpenUrlFREE Full Text
  21. ↵
    1. Penna M,
    2. Moran B,
    3. Crane S,
    4. Hompes R,
    5. Cunningham C
    . Surgeon-specific outcome reporting: is it time to move forward?Colorectal Dis2016;18:1031-2. doi:10.1111/codi.13433 pmid:27342085
    OpenUrlCrossRefPubMed
  22. ↵
    1. Dranove D,
    2. Kessler D,
    3. McClellan M,
    4. Satterthwaite M
    . Is More Information Better? The Effects of “Report Cards” on Health Care Providers. J Polit Econ2003;111:555-88doi:10.1086/374180.
    OpenUrlCrossRefWeb of Science
  23. ↵
    1. Tarantino I,
    2. Warschkow R,
    3. Worni M,
    4. et al
    . Prognostic Relevance of Palliative Primary Tumor Removal in 37,793 Metastatic Colorectal Cancer Patients: A Population-Based, Propensity Score-Adjusted Trend Analysis. Ann Surg2015;262:112-20. doi:10.1097/SLA.0000000000000860 pmid:25373464
    OpenUrlCrossRefPubMed
  24. ↵
    1. van der Geest LG,
    2. Lam-Boer J,
    3. Koopman M,
    4. Verhoef C,
    5. Elferink MA,
    6. de Wilt JH
    . Nationwide trends in incidence, treatment and survival of colorectal cancer patients with synchronous metastases. Clin Exp Metastasis2015;32:457-65. doi:10.1007/s10585-015-9719-0 pmid:25899064
    OpenUrlCrossRefPubMed
  25. ↵
    1. Green J,
    2. Wintfeld N
    . Report cards on cardiac surgeons. Assessing New York State’s approach. N Engl J Med1995;332:1229-32. doi:10.1056/NEJM199505043321812 pmid:7700321
    OpenUrlCrossRefPubMedWeb of Science
  26. ↵
    NHS Digital. National Bowel Cancer Audit (NBOCA) frequently asked questions. 2017. http://content.digital.nhs.uk/media/17324/NBOCAFAQs-2017/pdf/NBOCA_FAQs2017_v1.0.pdf.
  27. ↵
    1. Ljungqvist O,
    2. Scott M,
    3. Fearon KC
    . Enhanced Recovery After Surgery: A Review. JAMA Surg2017;152:292-8. doi:10.1001/jamasurg.2016.4952 pmid:28097305
    OpenUrlCrossRefPubMed
  28. ↵
    1. West MA,
    2. Parry MG,
    3. Lythgoe D,
    4. et al
    . Cardiopulmonary exercise testing for the prediction of morbidity risk after rectal cancer surgery. Br J Surg2014;101:1166-72. doi:10.1002/bjs.9551 pmid:24916313
    OpenUrlCrossRefPubMed
  29. ↵
    1. West MA,
    2. Lythgoe D,
    3. Barben CP,
    4. et al
    . Cardiopulmonary exercise variables are associated with postoperative morbidity after major colonic surgery: a prospective blinded observational study. Br J Anaesth2014;112:665-71. doi:10.1093/bja/aet408 pmid:24322573
    OpenUrlCrossRefPubMedWeb of Science
  30. ↵
    1. Marshall MN,
    2. Shekelle PG,
    3. Leatherman S,
    4. Brook RH
    . The public release of performance data: what do we expect to gain? A review of the evidence. JAMA2000;283:1866-74. doi:10.1001/jama.283.14.1866 pmid:10770149
    OpenUrlCrossRefPubMedWeb of Science
  31. ↵
    1. Aggarwal A,
    2. Lewis D,
    3. Mason M,
    4. Sullivan R,
    5. van der Meulen J
    . Patient Mobility for Elective Secondary Health Care Services in Response to Patient Choice Policies: A Systematic Review. Med Care Res Rev2017;74:379-403. doi:10.1177/1077558716654631 pmid:27357394
    OpenUrlCrossRefPubMed
  32. ↵
    1. Ergina PL,
    2. Cook JA,
    3. Blazeby JM,
    4. et al.,
    5. Balliol Collaboration
    . Challenges in evaluating surgical innovation. Lancet2009;374:1097-104. doi:10.1016/S0140-6736(09)61086-2 pmid:19782875
    OpenUrlCrossRefPubMedWeb of Science
  33. ↵
    1. Wallace D,
    2. Walker K,
    3. Kuryba A,
    4. Finan P,
    5. Scott N,
    6. van der Meulen J
    . Identifying patients at risk of emergency admission for colorectal cancer. Br J Cancer2014;111:577-80. doi:10.1038/bjc.2014.300 pmid:24921910
    OpenUrlCrossRefPubMed
  34. ↵
    1. Braunholtz DA,
    2. Edwards SJ,
    3. Lilford RJ
    . Are randomized clinical trials good for us (in the short term)? Evidence for a “trial effect”. J Clin Epidemiol2001;54:217-24. doi:10.1016/S0895-4356(00)00305-X pmid:11223318
    OpenUrlCrossRefPubMedWeb of Science
  35. ↵
    1. Moran J,
    2. Wilson F,
    3. Guinan E,
    4. McCormick P,
    5. Hussey J,
    6. Moriarty J
    . Role of cardiopulmonary exercise testing as a risk-assessment method in patients undergoing intra-abdominal surgery: a systematic review. Br J Anaesth2016;116:177-91. doi:10.1093/bja/aev454 pmid:26787788
    OpenUrlCrossRefPubMed
  36. ↵
    National Bowel Cancer Audit. Report 2016. 2016. https://www.nboca.org.uk/reports/annual-report-2016/.
  37. ↵
    1. Relman AS
    . Assessment and accountability: the third revolution in medical care. N Engl J Med1988;319:1220-2. doi:10.1056/NEJM198811033191810 pmid:3173460
    OpenUrlCrossRefPubMedWeb of Science
Back to top