The criticisms of the paper by Hogan and colleagues [Ref 1] advanced by Aylin and colleagues [Ref 2, 3] are understandable in view of the time, reputation and personal capital they have invested in HSMRs. But their critique misses the central point of the logic behind the call for HSMRs to be abandoned [Refs 4 and 5].
And that logic is simply that what matters is remediable errors and avoidable deaths. The SHMR does not measure avoidable deaths and it gives hospital managers and clinicians no clue as to what is wrong with their systems and what they can do about it. As a piece of information the HSMR statistic is neither specific nor actionable. And that is why I maintain that it is worthless.
What does a hospital do when a low SHMR is reported? It might lead to smugness and complacency. Potential opportunities for improving quality might go unexplored. What does a hospital do when its SHMR is high? Patients might only lose faith, managers and clinicians can only worry and fret and look for explanations in the data and the coding. Of course they will review their clinical and care practices but this is something they ought to be doing anyway, regardless of the SHMR level.
Death is not in and of itself an adverse outcome[Ref 6]. It’s an adverse outcome when poor or inadequate care processes contribute to the death in greater or lesser measure. It will inevitably remain a matter of retrospective clinical judgement whether there were any acts of omission or commission that caused or contributed to a death after a spell of healthcare in hospital. And as with any such judgements there will be error. But the key point is this: what matters to patients is 'avoidable' deaths. That is the only true measure (insofar as mortality is a measure of quality) of the quality of healthcare that matters to patients, families, and to those who are engaged in quality improvement.
Aylin’s argument fails to acknowledge that the HSMR is not a measure of avoidable deaths. And for that reason it is a useless piece of information.
Is there empirical evidence of the usefulness of the use of SHMRs? Aylin comes up with the standard, but flawed, response that of 14 hospitals that were flagged as high on one or both of 2 aggregate mortality indicators (SHMR and SHMI), 11 were found on a detailed inspection to have concerns about their quality of care [Ref 7]. If this is the best empirical evidence they can cite for the usefulness of SHMRs then I have to say that for a team of epidemiologists it shows a remarkable lack of critical reasoning.
Yes, these 14 hospitals had a high vale for either SHMI or SHMR or both. This happened soon after and in the wake of the Mid-Staffs scandal. The media were abuzz with stories of hundreds of deaths caused by poor care in that ill-fated hospital, that these deaths were all avoidable and caused by acts of omission or commission. None of these ludicrous claims stood up to closer scrutiny [Ref 8]. It was against this background that inspection teams went in to these 14 selected hospitals; they knew these hospitals had a high SHMR / SHMI and they were led to believe that there was a link between high SHMRs and poor care, all they had to do was find the evidence of poor care. It was a self-fulfilling prophesy and right on cue 11 hospitals were found to have poor care.
Two questions were never asked or answered:
1. Were there other more direct indicators of poor quality care in these and indeed other hospitals?
2. What would the inspectors have found if they had also looked at a control group of hospitals with low/average SHMRs and had looked at them without prior knowledge of SHMR (blinding).
Sadly we'll never know the answer, but it is surprising for a team of epidemiologists to claim value for an indicator without subjecting it to the same testing rigour that we expect to apply to clinical interventions. To conclude that SHMRs add value to the identification of poor quality in hospitals based on this evidence is to place too heavy a reliance on an uncontrolled observational study in which the observers were led inevitably to their findings by the deliberate introduction of information bias. Indeed the real surprise is that 3 of 14 hospitals were not found to have failings.
Sir Bruce Keogh’s report [Ref 7] is worth a re-read by anyone interested in the subject. Not only does he make the now well-known ‘clinically meaningless and academically reckless’ comment about any attempt to link the SHMR with avoidable mortality, but almost all of the hospital specific actions that he recommends are things that all hospital trusts ought to be doing anyway – whatever the SHMR. Hospitals should listen to patients, review complaints, ensure adequate staffing levels, deal with serious incidents, encourage a culture of open reporting of problems by staff, act on internal reviews, have a strategy for quality improvement, apply infection control and other policies consistently, work with partners to improve patient flow and reduce overcrowding.. .and so on.
These are all standard actions that managers and Boards of all hospitals should be focusing on. Instead by having managers deal with queries about ‘mortality alerts’ that almost turn out to be clinically irrelevant statistical noise, they are taken away from real quality improvement work. And that is why I maintain that SHMRs are not just worthless, they are harmful too.
References
1. Hogan H, Zipfel R, Neuburger J, Hutchings A, Darzi A, Black N. Avoidability of hospital deaths and association with hospital-wide mortality ratios: retrospective case record review and regression analysis. BMJ 2015;351:h3239
2. Aylin P. Rapid Response. http://www.bmj.com/content/351/bmj.h3239/rr-7
3. Aylin P, Bottle A, Jarman B. Rapid Response http://www.bmj.com/content/351/bmj.h3239/rr
4. Black N Assessing the quality of hospitals. Hospital standardised mortality ratios should be abandoned. BMJ 2010;340:c2066 doi:10.1136/bmj.c2066
5. Rao JN. Time to abandon the use of HSMRs in the NHS. BMJ 2015; 351 doi: http://dx.doi.org/10.1136/bmj.h4308.
6. Being Mortal. Gawande A. 2014. Metroplitan Books.
7. Keogh, B. Review into the quality of care and treatment provided by 14 hospital trusts in England: overview report. 16 July 2013 http://www.nhs.uk/nhsengland/bruce-keogh-review/documents/outcomes/keogh...
Competing interests:
No competing interests
14 August 2015
Jammi N Rao
Public health Physician and Non-Executive Director
Rapid Response:
Response to Aylin and colleagues’ rapid response
The criticisms of the paper by Hogan and colleagues [Ref 1] advanced by Aylin and colleagues [Ref 2, 3] are understandable in view of the time, reputation and personal capital they have invested in HSMRs. But their critique misses the central point of the logic behind the call for HSMRs to be abandoned [Refs 4 and 5].
And that logic is simply that what matters is remediable errors and avoidable deaths. The SHMR does not measure avoidable deaths and it gives hospital managers and clinicians no clue as to what is wrong with their systems and what they can do about it. As a piece of information the HSMR statistic is neither specific nor actionable. And that is why I maintain that it is worthless.
What does a hospital do when a low SHMR is reported? It might lead to smugness and complacency. Potential opportunities for improving quality might go unexplored. What does a hospital do when its SHMR is high? Patients might only lose faith, managers and clinicians can only worry and fret and look for explanations in the data and the coding. Of course they will review their clinical and care practices but this is something they ought to be doing anyway, regardless of the SHMR level.
Death is not in and of itself an adverse outcome[Ref 6]. It’s an adverse outcome when poor or inadequate care processes contribute to the death in greater or lesser measure. It will inevitably remain a matter of retrospective clinical judgement whether there were any acts of omission or commission that caused or contributed to a death after a spell of healthcare in hospital. And as with any such judgements there will be error. But the key point is this: what matters to patients is 'avoidable' deaths. That is the only true measure (insofar as mortality is a measure of quality) of the quality of healthcare that matters to patients, families, and to those who are engaged in quality improvement.
Aylin’s argument fails to acknowledge that the HSMR is not a measure of avoidable deaths. And for that reason it is a useless piece of information.
Is there empirical evidence of the usefulness of the use of SHMRs? Aylin comes up with the standard, but flawed, response that of 14 hospitals that were flagged as high on one or both of 2 aggregate mortality indicators (SHMR and SHMI), 11 were found on a detailed inspection to have concerns about their quality of care [Ref 7]. If this is the best empirical evidence they can cite for the usefulness of SHMRs then I have to say that for a team of epidemiologists it shows a remarkable lack of critical reasoning.
Yes, these 14 hospitals had a high vale for either SHMI or SHMR or both. This happened soon after and in the wake of the Mid-Staffs scandal. The media were abuzz with stories of hundreds of deaths caused by poor care in that ill-fated hospital, that these deaths were all avoidable and caused by acts of omission or commission. None of these ludicrous claims stood up to closer scrutiny [Ref 8]. It was against this background that inspection teams went in to these 14 selected hospitals; they knew these hospitals had a high SHMR / SHMI and they were led to believe that there was a link between high SHMRs and poor care, all they had to do was find the evidence of poor care. It was a self-fulfilling prophesy and right on cue 11 hospitals were found to have poor care.
Two questions were never asked or answered:
1. Were there other more direct indicators of poor quality care in these and indeed other hospitals?
2. What would the inspectors have found if they had also looked at a control group of hospitals with low/average SHMRs and had looked at them without prior knowledge of SHMR (blinding).
Sadly we'll never know the answer, but it is surprising for a team of epidemiologists to claim value for an indicator without subjecting it to the same testing rigour that we expect to apply to clinical interventions. To conclude that SHMRs add value to the identification of poor quality in hospitals based on this evidence is to place too heavy a reliance on an uncontrolled observational study in which the observers were led inevitably to their findings by the deliberate introduction of information bias. Indeed the real surprise is that 3 of 14 hospitals were not found to have failings.
Sir Bruce Keogh’s report [Ref 7] is worth a re-read by anyone interested in the subject. Not only does he make the now well-known ‘clinically meaningless and academically reckless’ comment about any attempt to link the SHMR with avoidable mortality, but almost all of the hospital specific actions that he recommends are things that all hospital trusts ought to be doing anyway – whatever the SHMR. Hospitals should listen to patients, review complaints, ensure adequate staffing levels, deal with serious incidents, encourage a culture of open reporting of problems by staff, act on internal reviews, have a strategy for quality improvement, apply infection control and other policies consistently, work with partners to improve patient flow and reduce overcrowding.. .and so on.
These are all standard actions that managers and Boards of all hospitals should be focusing on. Instead by having managers deal with queries about ‘mortality alerts’ that almost turn out to be clinically irrelevant statistical noise, they are taken away from real quality improvement work. And that is why I maintain that SHMRs are not just worthless, they are harmful too.
References
1. Hogan H, Zipfel R, Neuburger J, Hutchings A, Darzi A, Black N. Avoidability of hospital deaths and association with hospital-wide mortality ratios: retrospective case record review and regression analysis. BMJ 2015;351:h3239
2. Aylin P. Rapid Response. http://www.bmj.com/content/351/bmj.h3239/rr-7
3. Aylin P, Bottle A, Jarman B. Rapid Response http://www.bmj.com/content/351/bmj.h3239/rr
4. Black N Assessing the quality of hospitals. Hospital standardised mortality ratios should be abandoned. BMJ 2010;340:c2066 doi:10.1136/bmj.c2066
5. Rao JN. Time to abandon the use of HSMRs in the NHS. BMJ 2015; 351 doi: http://dx.doi.org/10.1136/bmj.h4308.
6. Being Mortal. Gawande A. 2014. Metroplitan Books.
7. Keogh, B. Review into the quality of care and treatment provided by 14 hospital trusts in England: overview report. 16 July 2013 http://www.nhs.uk/nhsengland/bruce-keogh-review/documents/outcomes/keogh...
Competing interests: No competing interests