Chance of identifying poor surgeons from mortality data is low, say analystsBMJ 2013; 347 doi: http://dx.doi.org/10.1136/bmj.f4377 (Published 05 July 2013) Cite this as: BMJ 2013;347:f4377
Publishing patient death rates for individual surgeons will not identify all poor performers and may create a sense of false reassurance, warns a team from the London School of Hygiene and Tropical Medicine in the Lancet.1
In many specialties surgeons do not do enough operations, and nor is death a common enough outcome, for the data to have the statistical power from which to draw safe conclusions, say Jenny Neuberger and colleagues. Individual surgeons’ mortality rates have already begun appearing, and more will follow in the next few months. The NHS’s medical director, Bruce Keogh, has hailed the release of the data as “a major breakthrough in NHS transparency.”
But the London team said that the chances that the data would identify a poorly performing surgeon were low. For hip fracture surgery, for example, surgeons would need to conduct 75 operations a year for there to be a 70% chance of detecting that their death rates were double the national average. The median number of hip fracture operations is much lower: 31 a year. For bowel cancer resection, the figures are even worse: it would take 132 operations a year to provide a 70% chance of detecting a doubled death rate, but the median number is only nine.
Keogh pioneered the publication of death rates in heart surgery, where the numbers are larger. Applying the same measure to other types of surgery gives only a low chance of identifying poor performers and could be falsely interpreted as evidence of acceptable performance, say the London researchers.
Neuberger said, “The reporting of results for individual surgeons should be based on outcomes that are fairly frequent, and fortunately, from the point of view of patients, mortality is not one of them. For specialties in which most surgeons do not perform sufficient numbers of operations to reliably assess their outcomes, reporting should be at the level of the surgical team or hospital and not the surgeon.”
Other possible ways around the problem include using outcomes that are more common than death and collecting data over a longer period, the team adds. But data collected over five years, for example, would mean that any message would not be timely enough to be valuable to patients.
Data published by the NHS in England on 1 July included almost 1600 orthopaedic surgeons who have conducted more than a million hip and knee operations since 2003, with an average mortality rate of 0.5%. None were classified as having unusually high death rates. The same applied to more than 600 cardiologists who carried out 90 000 angioplasties in 2012, where numbers of deaths, strokes, and heart attacks all fell within the normal range for all practitioners.
The Vascular Society has published mortality data for operations to repair abdominal aortic aneurisms, where there is a wider range of outcomes—some surgeons showing death rates 14 times the national average. But the society said that when the age of the patient and the severity of the condition were taken into account, the differences were not statistically significant and were no more than might be expected from chance.2
Most of the high mortality rates were among surgeons who did relatively few operations. One had a death rate of 14.3%, which seemed high, but the surgeon had done the operation only seven times, with a single death.
Some surgeons—fewer than 30, the Royal College of Surgeons said—have refused to have their mortality data published. Six vascular surgeons who refused were named by NHS England, which said that none had mortality rates outside the accepted range. One of them, Peter McCollum, told Radio 4’s World at One: “The data is fairly meaningless. There are other ways of picking up bad surgeons. The problem is that if you want information this data must be interpreted by an expert, and that is not what has happened on this occasion.
“The best way forward is for vascular surgery, which is a high risk area of surgery, to be done at major units and for there to be peer pressure, as there is at my own unit at Hull. We would prefer our results to be analysed at unit level, with a much larger number of cases albeit spread over a number of surgeons.”
Keogh said, “The authors have misunderstood the purpose of public disclosure of information. It is not a statistical exercise to identify poor performers. Rather it is an exercise to provide information on activity and outcomes, to focus minds on improving results and to assure the public that the NHS offers high quality surgery. But it does raise the question as to whether it is better to have your complex operation performed by a surgeon doing a lot of that operation or only a few.”
Ben Bridgewater, the Healthcare Quality Improvement Partnership’s director of outcomes publication, said, “A debate around the initiative to publish individual clinician outcomes should be welcomed, but it’s crucial to raise issues about this study.
“The analysis shows very small numbers: in bowel resection surgery a median of just nine annually on average per surgeon. This is worthy of consideration in its own right as we know there is an association between volume and outcomes for this type of surgery, and this is the point of transparency.
“Furthermore, it is standard practice in scientific study, when volumes are low, to select outcomes with measures with a high enough incidence to be useful, and it would be interesting to reflect on which composite endpoints are appropriate for individual surgeons reporting for different specialties, rather than dismiss the concept. Patients expect someone to be in charge of their overall care, and for bowel resection that should be a surgeon, and everyone should know who that surgeon is.”
Cite this as: BMJ 2013;347:f4377