Rapid Responses to:

NEWS: Susan Mayor NHS publishes survival rates for common operations for individual hospitals BMJ 2008; 337: a803 [Full text]

Rapid Responses: Submit a response to this article

Rapid Responses published:

Caution encouraged in interpreting surgical outcome data

Peter J E Holt, Dr Jan D. Poloniecki, Prof Matt M. Thompson   (19 August 2008)

Caution encouraged in interpreting surgical outcome data 19 August 2008
Peter J E Holt, Clinical Lecturer in Vascular Surgery St George's Vascular Institute, Dr Jan D. Poloniecki, Prof Matt M. Thompson Send response to journal: Re: Caution encouraged in interpreting surgical outcome data	As outlined in the BMJ (1), the medical director of the NHS produced some limited data that aimed to quantify the outcome of ‘key’ surgical procedures in NHS hospitals. Outcome measures available on the NHS Choice website included the number of operations performed, readmission rates and a broad estimate of procedural risk (2). The public availability of these data should have been a landmark in reporting surgical outcomes, but we believe the reported data were undermined by inaccuracy. Furthermore, the numeric data appeared to have been available to the press only, rather than being put clearly into the public domain (3). The available surgical outcomes appeared to be based on SUS data that may not have been analysed after the most recent updates, making the results prone to misinterpretation as not all hospitals submit data concurrently. Additionally, and perhaps more significantly, a simplistic approach to case identification appears to have been utilised. Previous work has shown that clinical coding may utilise a variety of diagnostic and procedural codes for any particular condition, making the acquisition of a complete dataset an involved process (4). Clinical consultation for each procedure would have provided approximate case volumes that could have been checked against the published data. This level of clinical consultation does not appear to have occurred, as the case numbers specified for many hospitals differed by an order of magnitude from that expected by local audit and previous national publications (4,5). For example, our own institution performed over 200 elective AAA repairs per annum, but was reported in the data as performing between 5 and 15 cases. These data have now been removed from the website, but differences of this magnitude made the applicability of the data difficult to interpret. Irrespective of the accuracy of the data, one further concern was the system used to identify potential outliers in terms of surgical outcome. The results were presented as standardised mortality ratios and these demonstrated that a number of hospitals had mortality rates four times that elsewhere for elective AAA repair (3). Despite these differences, no hospitals in England were deemed to require further investigation to determine their safety (all death rates were reported “as expected”), which contradicted previous publications (4,5). It may be that more robust analyses are required if outcomes data are to be made public. Finally, there were no data on the NHS national death rates for other specified treatments such as carotid endarterectomy and coronary angioplasty despite the importance of having relevant figures for angioplasty having been stressed (6) and high death rates for CEA having been reported (7). In conclusion, the publication of surgical outcome data is to be applauded and encouraged, but the data must be rigorously quality assured and assiduously analysed. Publication of incorrect data may harm Patient Choice. Yours sincerely, Mr Peter Holt Dr Jan Poloniecki Prof Matt Thompson 1. Mayor S. NHS publishes survival rates for common operations for individual hospitals. BMJ 2008;337:a803 2. http://www.nhs.uk/Pages/Mortalityrates.aspx NHS Choices publishes surgery survival rates 3. Carvel J. Wide variation in trusts’ death rates for major surgery shown in new figures. The Guardian. 11th July 2008. http://www.guardian.co.uk/society/2008/jul/11/nhs.health 4. Holt PJE, Poloniecki JD, Loftus IM, Michaels JA, Thompson MM. Epidemiological study of the relationship between volume and outcome after abdominal aortic aneurysm surgery in the UK from 2000 to 2005. The British journal of surgery 2007;94(4): 441-448. 5. Holt PJE, Poloniecki JD, Loftus IM, Thompson MM. Demonstrating safety through in-hospital mortality following elective abdominal aortic aneurysm repair in England. The British journal of surgery 2007;95(1): 64-71. 6. Blackstone EH. Monitoring surgical performance. Journal of Thoracic and Cardiovascular Surgery 2004;128(6):807-10 7. Holt PJE, Poloniecki JD, Loftus IM, Thompson MM. The relationship between hospital case volume and outcome from carotid endartectomy in England from 2000 to 2005. Eur J Vasc Endovasc Surg. 2007;34(6):646-54 Competing interests: None declared