Performance data on all surgeons in England will be published within two yearsBMJ 2012; 345 doi: https://doi.org/10.1136/bmj.e8377 (Published 07 December 2012) Cite this as: BMJ 2012;345:e8377
All rapid responses
We felt compelled to write this letter in response to a number of recent reports in the media related to the publication of surgeon-specific outcome data and proposed plans for this data to be available in the public domain. Whilst we applaud any attempt to improve surgical standards in the NHS and recognise that publication of surgeon-specific data has been undertaken with some success in the field of Cardiac Surgery, we have concerns with this approach, particularly in surgical specialities with relative low volumes (100 per annum for a single unit) compared to CABG.
Our recent paper (Griffin et al, Target setting for elective infra-renal AAA surgery: a single mortality figure), accepted for publication in The Surgeon outlines the many complexities involved in setting targets for mortality for surgical procedures. These include the challenges of missing or incorrect data (e.g. from the National Vascular Database), the statistical problems of natural variability (which mean that smaller units will naturally have much larger variability in results than larger units) and the intrinsic difficulties of choosing targets and adjusting for case mix.
Many of these problems are amplified if outcome data is presented as surgeon-specific figures (rather than unit data) and in 2000 Irvine et al1 showed that with an adverse-event base rate of 3% it would take an individual surgeon in their unit between 5 and 47 years to perform enough elective AAA repairs to predict an adverse rate of 6% (i.e. twice the base rate). In our opinion this is too long a time period to allow poor practice to go unchecked. Real-time change cannot be affected if there are surgeons of concern and sampling over longer periods of time also introduces further bias as, for example, changes in graft technology or peri-operative care become standard practice.
Whilst we acknowledge that the publication of unit-based outcomes can hide discrepancies between individual surgeons, the publication of surgeon-specific figures would reduce procedure numbers and would therefore increase natural variability further. Also, it should be hoped that targets focussed on unit activity would encourage improvements in overall patient care including the pre-operative assessment of high risk patients and involvement of a multidisciplinary team to inform the decision to operate. It is these factors (rather than surgical skill and experience alone) that probably influence much of the volume-outcome relationship that has been investigated by Holt et al2 and others.
There has been much work undertaken as to whether the publication of results leads to “defensive medicine” whereby high risk patients are turned down in favour of those with fewer co-morbidities (for example, where AAA-screening patients are operated on in favour of those with incidentally diagnosed AAA with multiple co-morbidities) however many, including Ben Bridgewater, have denied that publication of results influences operating practice. Whilst we would hope that this is true, we feel that any publication of results (either by unit or by individual) should be accompanied by mandatory reporting of “turn-down” figures.
Equally, a robust method of patient risk-stratification needs to accompany any published outcome data. The EuroScore is a well validated and widely used tool which allows calculation of such risk stratification data in cardiac surgery patients. Whilst there are many scoring systems for vascular surgery reported in the literature, as yet there is none that is as well validated or widely used as the EuroScore and hence case-mix assessment remains challenging. Work from our group3 has confirmed that data missingness can have a large effect on risk prediction models and even when appropriate risk assessment is possible, there is anecdotal evidence to suggest that the process can be manipulated.
We would urge that these complex challenges (along with others discussed in our paper) need to be considered and solutions sought prior to any changes being made which would demand publication of data. We have an important responsibility to ensure that data in the public domain is accurate, statistically valid and appropriately stratified and the opportunity to meet that responsibility needs to be taken now.
Miss Kathryn Griffin (BHF Research Fellow in Vascular Surgery)
for, and on behalf of,
Miss KJ Griffin (BHF Research Fellow in Vascular Surgery), Dr SJ Fleming (Senior Research Fellow in Biostatistics), Mr MA Bailey (BHF Research Fellow in Vascular Surgery), Dr C Czoski-Murray (Senior Research Fellow in Applied Health Research),
Dr PD Baxter (Senior Lecturer in Biostatistics), Prof DJA Scott (Professor of Vascular Surgery)
1Irvine CD, Grayson D, Lusby RJ. Clinical governance and the vascular surgeon. Br J Surg 2000; 87:766-770
2 Holt PJ; Poloniecki JD; Gerrard D; Loftus IM; Thompson MM. Meta-analysis and systematic review of the relationship between volume and outcome in abdominal aortic aneurysm surgery. Br J Surg. 2007; 94(4):395-403
3Cattle BA, Baxter PD, Fleming TJ, Gale CP, Mitchell DC, Gilthorpe MS, Scott DJA, Czoski-Murray CJ, McCabe C. Data Quality Improvement, Data Linkage and Multiple Imputation in the UK National Vascular Database. International Journal of Probability and Statistics, 2012(2), doi:10.5539/ijsp.v1n2p137
Competing interests: No competing interests
In principle, I agree with the argument that surgeons should keep a record of operative outcome data to audit and inform their practice. However using this data to compare one surgeon with another is fraught with difficulty, since there will be a variability in outcomes based on population age distribution and geographical factors. Not to mention patient related factors such as comorbidity which undoubtedly influence patient outcomes.
League tables comparing one surgeon with another may influence their practice in an adverse way. Some surgeons may be reluctant to undertake complex or high risk procedures which may benefit the patient, but at increased risk of post-operative morbidity. For instance, a surgeon may opt to perform a Hartmanns for an obstructing recto-sigmoid tumour rather than the more complex on table lavage and primary anastomosis. The surgeons who are more concerned about their ‘mortality rates’ will probably be more cautious in operating on high risk patients such as those with cardiovascular co-morbidity and the elderly. Likewise in the elective setting some surgeons may decide to consider conservative management of surgical disease for fear of worsening their league table status and thus despite having a better league score may not be offering such a comprehensive service. Conversely those surgeons willing to operate on higher risk patients to offer surgical treatment for cancer or benign disease will undoubtedly have more complications and poorer outcomes in part related to patient factors but not necessarily a reflection of the ability of the surgeon concerned. All data presented to the public is open to interpretation and given that a recent survey showed that fifty per cent of the public did not understand the data presented to them and how to use it to inform the healthcare decisions, we have to be sure to present this data in way that is understandable accurate and fair comparing like with like where possible. Being open and transparent is something that all surgeons should embrace, but we need to ensure that data is accurate, complete, and independently verifiable.
The article suggests that if the surgeons knew that their performance data was out in public they would concentrate on their performance data and seek help from their colleagues. If teamwork is to be encouraged then having surgeons compete with each other seems counter intuitive and league tables may in the long run be counter-productive for the aim of improving healthcare standards.
Competing interests: No competing interests