Patient coding and the ratings gameBMJ 2010; 340 doi: https://doi.org/10.1136/bmj.c2153 (Published 25 April 2010) Cite this as: BMJ 2010;340:c2153
When a hospital singled out last year as being in “urgent need” of improvement by the regulators was rated at the same time the ninth safest in England by health analysts Dr Foster, questions were bound to be asked.
Mid Staffordshire Foundation NHS Trust, branded “appalling” by the Care Quality Commission and since warned that unless it improves it may lose its licence to operate, set tongues wagging when Dr Foster’s 2009 Good Hospital Guide rated it as among the five “most improved hospitals” in the past three years and in the top ten for quality of care.
Somebody must be wrong. Evidence is accumulating that in the case of Mid Staffordshire and some other poorly performing hospitals, Dr Foster’s performance evaluations have been affected by variations in the way patients are allocated diagnostic codes, flattering the hospital standardised mortality ratios (HSMRs) and making the hospitals seem much better than they are.
Take, for example, the death rate after admission for a broken hip recorded by Mid Staffordshire in the 2009 guide. It reported a rate of 19.87 against a national average of 100. That means that an elderly person who breaks a hip is five times less likely to die if admitted to Mid Staffordshire than to the average English hospital.
Among the top 30 hospitals in the Dr Foster guide, Mid Staffordshire’s death rate after hip fracture is less than half that of any other hospital, and less than a seventh of the figure recorded by, for example, Barts and the London.
Struck by the implausibility of this figure, I asked Mid Staffordshire if it was right. It treated my inquiry as a freedom of information request, but in due course replied, saying: “We can confirm that the data Dr Foster has published matches the data that was submitted via the Secondary Users Service. We have also checked the data the trust holds within its own corporate information systems using the Dr Foster methodology for reporting HSMR and have been able to replicate their figures.”
But did that mean the results were right? If they were, Mid Staffordshire had clearly found a way of treating this common condition that deserved to be more widely shared. The trust replied: “We have not always had such a low SMR [standardised mortality ratio] for fractured neck of femur. Our Clinical Coding department advise that the change is due to substantially improved coding procedures. These have been thoroughly reviewed over the past year or so as a result of the improvements arising from the Healthcare Commission Report.”
Importance of coding
Diagnostic coding is used by hospitals to record the conditions of patients admitted to them for treatment. The codes form the basis for payment, under the payments by results system, and a single patient may be assigned several codes to reflect comorbidities. Research by CHKS, a rival health analytical company, shows that the number of codes has been creeping up over recent years, from an average of just over three per patient in April 2005 to almost four in June 2009.
The average number of diagnostic codes per patient also varies widely from hospital to hospital, from under 3% to more than 5%. Brian Jarman, head of the Dr Foster Unit at Imperial College and whose method is used for Dr Foster’s calculations, says that coding depth is only loosely related to HSMR, so in his view it cannot explain more than small differences.
But if “improved” coding can account for the astonishingly low death rates from broken hips at Mid Staffordshire, it suggests that Professor Jarman’s view may be overoptimistic. In some hospitals, CHKS argues, coding changes are driving the improvement in HSMRs without any change in the underlying hospital performance.
How it works is this. If a hospital uses an increasing number of codes year by year, it implies a rise in severity of illness among the patients it is treating. If the death rate remains constant while the severity index increases, the hospital appears to be doing better at keeping people alive.
In fact, as CHKS points out, crude death rates in English hospitals show virtually no change over the past five years. There were 225 439 deaths in English hospitals in 2004-5 and 222 738 in 2008-9, with similar figures in the intermediate years. Yet Dr Foster shows HSMRs (death rates adjusted for severity) falling by 7% last year, and CHKS, using its own similar measure, shows a 50% fall over five years.
This fall is entirely a product of increasing severity, not a fall in the actual numbers dying. At Walsall Hospital, another listed as among the five “most improved” by Dr Foster, crude deaths rates have actually risen over the past five years, CHKS says. Yet the hospital’s HSMR has fallen.
Rise of palliative care
How has this happened? In the case of Walsall, the increasing proportion of patients classified as requiring palliative care is believed to be an important factor. Across England there has been a large increase in the use of Z51.5, the code used for palliative care.
CHKS figures show the number of deaths coded Z51.5 was under 400 a month in 2004 but had reached more than 1800 a month by June 2009. Patients coded Z51.5 are assumed to have come into hospital to die, so both the CHKS and the Dr Foster methods allow for that. Hospitals should not be blamed for the deaths of patients whose lives cannot be saved.
In 2007-8, CHKS figures show that 5% of deaths in English hospitals were coded Z51.5, but for some hospitals the figure exceeded 20%. By 2008-9, the average had risen to 7.8%, with a few hospitals, including Walsall, coding more than half of deaths Z51.5—in one month it reached 80%. By April-June 2009 the overall average in England had risen to 11.3%, with five hospitals coding more than 30% of their deaths as palliative care cases.
Professor Jarman worked out the implications of these changes for me. He calculated the change in HSMR that would have occurred in 2005, 2006, and 2007 if the rate of coding of palliative care that applied in 2008 had also applied in those years. He found five trusts that would have reduced their HSMRs by five points a year, with the greatest change being just under 20 points. This means that a few heavy users of the Z51.5 code could have reduced their HSMRs from 110 (above average) to 90 (below average) simply by increasing the frequency of use of the palliative care code.
“We believe that a central instruction should be given about palliative care coding, as they have done in Canada, to say that a case is only palliative care if the patient has a terminal illness,” he says.
Walsall Hospital denied that changes in palliative care coding were the only reason why its HSMR had gone down. Mike Browne, the medical director, said that the hospital had been shocked when identified by Dr Foster as a poor performer in 2002, and a lot of money and effort had been invested in improving performance. There had been coding changes, including an increase in palliative care coding, to which he attributed a third of the hospital’s recent improvements in HSMRs, but Dr Browne denied that any changes in palliative care coding had been driven by a desire to lower its HSMR.
The issue is not simple because an elderly patient may be admitted with a treatable condition such as a urinary tract infection but then deteriorate. The hospital might then put the patient on the Liverpool care pathway, a pattern of care that is designed to ease the last few days. In such cases the patients can have a secondary diagnosis and be coded as Z51.5. But the fact that they are dying may be partly due to poor treatment of the condition for which they were first admitted. Thus by reclassifying patients as Z51.5 hospitals can, in theory, hide poor care.
Value of data
The analysis does not prove that any hospital is doing this, though it is possible that some may be. Mid Staffordshire’s belief that its coding has improved and is now a more accurate record of the complexity of patients’ conditions may be correct. However, there is no question that the situation is open to manipulation by trusts.
Dr Foster Intelligence, the company responsible for the Good Hospital Guide, believes that HSMRs remain a good measure of the quality of care if used alongside other measures. CHKS, by contrast, believes that although they may be a useful tool within a hospital, they are unsuitable for comparisons between hospitals while such big coding variations exist.
It points out that four of the five hospitals rated as “most improved” in the 2009 guide (Mid Staffordshire, Medway, Stafford, and Walsall) have crude death rates that have fallen no faster than the national average, and the fifth, George Eliot, has shown a sharper fall but from a very high level. In the case of Medway, CHKS advice to increase end of life coding resulted in 37% of deaths, rather than 8%, being excluded from the risk adjusted mortality.
Dr Foster believes that the criticisms of HSMRs apply to only a few hospitals and that for the rest the technique works well. Its confidence was backed by the statistical appendix to the Francis report on Mid Staffordshire, written by David Shahian of Massachusetts General Hospital, which found criticisms of HSMRs to be mostly misplaced. Dr Shahian recorded that in 2006-7, coding depth at the trust was below the England average but by 2007-8 it had surpassed the average, adding that the level of detail available made it impossible to estimate the effect this had on standardised mortality ratios for different conditions.
Dr Browne said that Walsall had found measuring mortality rates for individual conditions very useful in identifying areas for improvement, but he questioned whether HSMRs were so valuable for comparing hospitals. “When we looked bad, we were never that bad, and when we looked good, we were never that good,” he says. “We did look at palliative care coding and it had jumped, so we tightened up in June—we’re trying to focus on making sure it’s not misused.”
The clash of evidence between the Good Hospital Guide and the Care Quality Commission embarrassed the Department of Health. Was Mid Staffordshire hugely improved, as Dr Foster said, or still in trouble, as the commission believed? And what of Basildon and Thurrock, whose HSMR was the worst in England (131) but which had been awarded a “double excellent” in the Healthcare Commission’s annual health check just a month before?
Basildon and Thurrock issued a detailed statement at the time pointing out that although its HSMR had been rising, the CHKS measure (risk adjusted mortality) had been falling. It attributed the difference to “low comorbidity scores”—it was not coding as fully as other hospitals the range of problems a patient had on admission. Comorbidities recorded later are taken account of in the CHKS system but not by Dr Foster, the statement said. Its conclusion was that it needed to employ more coders.
Conflicts like these generated political heat and made the regulator and the Department of Health look as if they didn’t know what they were doing. Bruce Keogh, NHS medical director, said that he was getting calls from trusts saying, “What the hell is going on?” after the appearance of the guide. This in turn prompted him to set up a review of the use of HSMRs in England, led by Ian Dalton, chief executive of NHS North East.
The aim of the review is to ensure that HSMRs cannot catch the department out again. But the picture was further muddied by the NHS Information Centre, which said disagreements about mortality measures were compromising confidence in them and in their effective use. It commissioned an independent assessment to determine the best method and then sought tenders for producing mortality data for the NHS Choices website.
Eight companies bid for the contract, including Dr Foster and CHKS, but the winner was IMS Health, a US based company. This might have been fine but for the fact that IMS, using a method very similar to Dr Foster’s, was unable to generate the same results.
Professor Jarman was angry that Dr Foster had failed to win the contract, he told the Sunday Express. “I couldn’t believe it when I was told. We’ve done an excellent job in helping to protect the public by exposing failing hospitals. Clearly we have done too good a job. I have been told the department wants greater control, perhaps to manipulate the data or present it in a better light.”
Last month he showed he can be a risky man to trifle with by issuing a list of 25 hospital trusts in England that he said had death rates higher than they ought to be. In 2007-8 there had been 4600 excess deaths at these trusts, he said, and the Care Quality Commission was failing to detect bad practice with an inspection regimen that was “fundamentally flawed.” This was just the kind of news the department could have done without.
Mr Dalton will need all his diplomatic skills to negotiate his way through the bruised egos caused by the row over HSMRs. The review has been asked to report back with advice on how HSMRs should be used—or if they should be used at all—and a separate working party is looking at other patient safety indicators.
Dr Browne believes that, used properly, HSMRs remain a valuable tool. “What saddens me is that if it wasn’t being used for making heroes or villains, it’s one of the best measures we’ve got,” he said.
Cite this as: BMJ 2010;340:c2153
Competing interests: NH is director of Straight Statistics.