Re: Performance data on all surgeons in England will be published within two years
We felt compelled to write this letter in response to a number of recent reports in the media related to the publication of surgeon-specific outcome data and proposed plans for this data to be available in the public domain. Whilst we applaud any attempt to improve surgical standards in the NHS and recognise that publication of surgeon-specific data has been undertaken with some success in the field of Cardiac Surgery, we have concerns with this approach, particularly in surgical specialities with relative low volumes (100 per annum for a single unit) compared to CABG.
Our recent paper (Griffin et al, Target setting for elective infra-renal AAA surgery: a single mortality figure), accepted for publication in The Surgeon outlines the many complexities involved in setting targets for mortality for surgical procedures. These include the challenges of missing or incorrect data (e.g. from the National Vascular Database), the statistical problems of natural variability (which mean that smaller units will naturally have much larger variability in results than larger units) and the intrinsic difficulties of choosing targets and adjusting for case mix.
Many of these problems are amplified if outcome data is presented as surgeon-specific figures (rather than unit data) and in 2000 Irvine et al1 showed that with an adverse-event base rate of 3% it would take an individual surgeon in their unit between 5 and 47 years to perform enough elective AAA repairs to predict an adverse rate of 6% (i.e. twice the base rate). In our opinion this is too long a time period to allow poor practice to go unchecked. Real-time change cannot be affected if there are surgeons of concern and sampling over longer periods of time also introduces further bias as, for example, changes in graft technology or peri-operative care become standard practice.
Whilst we acknowledge that the publication of unit-based outcomes can hide discrepancies between individual surgeons, the publication of surgeon-specific figures would reduce procedure numbers and would therefore increase natural variability further. Also, it should be hoped that targets focussed on unit activity would encourage improvements in overall patient care including the pre-operative assessment of high risk patients and involvement of a multidisciplinary team to inform the decision to operate. It is these factors (rather than surgical skill and experience alone) that probably influence much of the volume-outcome relationship that has been investigated by Holt et al2 and others.
There has been much work undertaken as to whether the publication of results leads to “defensive medicine” whereby high risk patients are turned down in favour of those with fewer co-morbidities (for example, where AAA-screening patients are operated on in favour of those with incidentally diagnosed AAA with multiple co-morbidities) however many, including Ben Bridgewater, have denied that publication of results influences operating practice. Whilst we would hope that this is true, we feel that any publication of results (either by unit or by individual) should be accompanied by mandatory reporting of “turn-down” figures.
Equally, a robust method of patient risk-stratification needs to accompany any published outcome data. The EuroScore is a well validated and widely used tool which allows calculation of such risk stratification data in cardiac surgery patients. Whilst there are many scoring systems for vascular surgery reported in the literature, as yet there is none that is as well validated or widely used as the EuroScore and hence case-mix assessment remains challenging. Work from our group3 has confirmed that data missingness can have a large effect on risk prediction models and even when appropriate risk assessment is possible, there is anecdotal evidence to suggest that the process can be manipulated.
We would urge that these complex challenges (along with others discussed in our paper) need to be considered and solutions sought prior to any changes being made which would demand publication of data. We have an important responsibility to ensure that data in the public domain is accurate, statistically valid and appropriately stratified and the opportunity to meet that responsibility needs to be taken now.
Miss Kathryn Griffin (BHF Research Fellow in Vascular Surgery)
for, and on behalf of,
Miss KJ Griffin (BHF Research Fellow in Vascular Surgery), Dr SJ Fleming (Senior Research Fellow in Biostatistics), Mr MA Bailey (BHF Research Fellow in Vascular Surgery), Dr C Czoski-Murray (Senior Research Fellow in Applied Health Research),
Dr PD Baxter (Senior Lecturer in Biostatistics), Prof DJA Scott (Professor of Vascular Surgery)
1Irvine CD, Grayson D, Lusby RJ. Clinical governance and the vascular surgeon. Br J Surg 2000; 87:766-770
2 Holt PJ; Poloniecki JD; Gerrard D; Loftus IM; Thompson MM. Meta-analysis and systematic review of the relationship between volume and outcome in abdominal aortic aneurysm surgery. Br J Surg. 2007; 94(4):395-403
3Cattle BA, Baxter PD, Fleming TJ, Gale CP, Mitchell DC, Gilthorpe MS, Scott DJA, Czoski-Murray CJ, McCabe C. Data Quality Improvement, Data Linkage and Multiple Imputation in the UK National Vascular Database. International Journal of Probability and Statistics, 2012(2), doi:10.5539/ijsp.v1n2p137