Re: Avoidability of hospital deaths and association with hospital-wide mortality ratios: retrospective case record review and regression analysis
As pointed out in an earlier response [1], the key issue with the paper by Hogan et al.[2] is the lack of statistical power to be able to detect a relationship between the proportion of “preventable” deaths in each hospital and published mortality indicators. A measure based on a numerator of 3 or 4 “preventable” deaths per trust, as determined by an arbitrary cut-off as to what constitutes “avoidable” by a single case note reviewer with only weak to moderate inter-rater agreement is no gold standard, [3] and is unlikely to demonstrate a relationship with anything other than due to chance.
Abel and Lyratzopoulos's rapid response [4] formalises the statistical argument by pointing out the error in Hogan's power calculation and demonstrating that it is impossible for the study to find the relationship it set out to discover. This is no mere statistical nicety. As Abel puts it “Thus the authors were trying to detect not just an implausibly large, but an impossibly large effect size”. Either through design or error, the study was never going to find the hypothesised relationship, and therefore cannot be used to comment on the validity of hospital wide measures of mortality (or indeed any other indicators of quality of care). There is no basis for claiming that “the absence of a significant association between hospital-wide SMRs and avoidable death proportions suggests that neither metric is a helpful or informative indicator of the quality of a trust”, and there are certainly no policy implications. The fact that the study’s fundamental flaw was not picked up by referees, raises concerns about the review process for the BMJ (and indeed for the project's funder). In particular the lack of a statistical critique of the premise of the study, and potentially the impartiality of the review process (one referee starts his review by saying “I hope that publication and discussion of this paper will get rid of this lousy statistic”).
In addition, there are other worrying aspects of the paper. The original proposal [5] sets out to review case notes in the 14 so called “Keogh” trusts investigated for high hospital mortality rates (11 of which were deemed to be so poor as to merit being put into special measures by the Secretary of State) and to compare with 10 other trusts in addition to the 10 trusts included in the earlier PRISM study [6]. There is no mention of this selection process in the paper, and the authors remain vague about their selection criteria, despite being pressed by at least one of the referees for more detail. Either the sampling method changed since the original proposal, or Hogan et al have obscured this important methodological detail. If their approach is not able to at least identify some of 11 trusts put in to special measures, it cast some doubt on the validity of the case note review method. A related concern is that the statistical methods appear to have changed since the original proposal. This perhaps is acceptable if amended prior to the study, however post-hoc changes to the analysis may be subject to the biases of the investigators.
One of the principal investigator’s views are in the public domain [7], and indeed this point is raised in another rapid response [8]. The original proposal states that “each case that is considered to be a preventable death (estimated to be 120) will be discussed with the principal investigator”. If this indeed happened, with the potential for measurement bias, it should have been disclosed in the paper.
In the paper and rapid responses, there does seem to be some agreement that measurement of quality of care in hospitals is a complex issue and that no single measure is ever going to assess the delivery of safe and effective care. This complexity requires that different metrics should be used. Sadly, this poorly designed (yet presumably costly) study brings us no closer to resolving questions about the validity of summary hospital mortality levels, and certainly should not be used as the basis for policy. Given the inadequacy of the design and the potential biases in execution, the study authors should reconsider their conclusions.
REFERENCES
[1] Aylin P, Bottle A, Jarman B. Rapid Response http://www.bmj.com/content/351/bmj.h3239/rr
[2] Hogan H, Zipfel R, Neuburger J, Hutchings A, Darzi A, Black N. Avoidability of hospital deaths and association with hospital-wide mortality ratios: retrospective case record review and regression analysis. BMJ 2015;351:h3239
[3] Abel G, Lyratzopoulus G. Ranking hospitals on avoidable death rates derived from retrospective case record review: methodological observations and limitations. BMJ Qual Saf 2015. doi:10.1136/bmjqs-2015-004366
[4] Abel G, Lyratzopoulus G. Rapid Response http://www.bmj.com/content/351/bmj.h3239/rr-6
[5] Hogan H, Darzi A, Black N. Review of 14 Trusts: investigation of avoidable mortality. http://www.nhs.uk/NHSEngland/bruce-keogh-review/Documents/NAG-retrospect...
[6] Hogan H, Healey F, Neale G, et al. Preventable deaths due to problems in care in English acute hospitals: a retrospective case record review study. BMJ Qual Saf2012:21:737-45.
[7] Black N Assessing the quality of hospitals. Hospital standardised mortality ratios should be abandoned. BMJ 2010;340:c2066 doi:10.1136/bmj.c2066
[8] Ben-Tovim D. Rapid Response http://www.bmj.com/content/351/bmj.h3239/rr-0
Post script
In fact the study’s final regression coefficient with an upper 95% confidence limit of a 0.7% change in avoidable deaths per 1SD change in HSMR is not far from including the true estimated standard deviation of 0.8% of avoidable deaths which would imply a perfect correlation of a 1SD change in outcome for a 1SD change in exposure. However, the wide confidence intervals (between -0.2% and 0.7%) betray the study’s obvious lack of power.
Out of the 14 hospital trusts picked for inspection on the basis of high hospital wide mortality rates for the Keogh review, 11 were deemed to be so poor as to merit being put into special measures by the Secretary of State, suggesting a positive predictive value of 79% for hospital wide summary mortality statistics. Of course, this figure takes no account of what would have been found if hospitals without raised mortality were inspected. The latest published CQC results suggest none of the 26 most recently inspected hospitals were rated as inadequate [http://www.cqc.org.uk/content/hospitals] but these inspections are carried out under a different regime, and are not necessarily comparable to earlier inspections.
Competing interests:
The Dr Foster Unit is an academic unit within the Department of Primary Care and Public Health, within the School of Public Health, Imperial College London. The unit receives research grant income from a range of funders, including the National Institute of Health Research and Dr Foster Intelligence, an independent health service research organisation (a wholly owned subsidiary of Telstra). The unit is affiliated with the Patient Safety Translation Research Centre at Imperial which is funded by the National Institute of Health Research. We are also grateful for support from the NIHR Biomedical Research Centre funding scheme.
12 August 2015
Paul Aylin
Professor of Epidemiology and Public Health
Imperial College London
Dr Foster Unit, School of Public Health, 3 Dorset Rise, London EC4Y 8EN
Rapid Response:
Re: Avoidability of hospital deaths and association with hospital-wide mortality ratios: retrospective case record review and regression analysis
As pointed out in an earlier response [1], the key issue with the paper by Hogan et al.[2] is the lack of statistical power to be able to detect a relationship between the proportion of “preventable” deaths in each hospital and published mortality indicators. A measure based on a numerator of 3 or 4 “preventable” deaths per trust, as determined by an arbitrary cut-off as to what constitutes “avoidable” by a single case note reviewer with only weak to moderate inter-rater agreement is no gold standard, [3] and is unlikely to demonstrate a relationship with anything other than due to chance.
Abel and Lyratzopoulos's rapid response [4] formalises the statistical argument by pointing out the error in Hogan's power calculation and demonstrating that it is impossible for the study to find the relationship it set out to discover. This is no mere statistical nicety. As Abel puts it “Thus the authors were trying to detect not just an implausibly large, but an impossibly large effect size”. Either through design or error, the study was never going to find the hypothesised relationship, and therefore cannot be used to comment on the validity of hospital wide measures of mortality (or indeed any other indicators of quality of care). There is no basis for claiming that “the absence of a significant association between hospital-wide SMRs and avoidable death proportions suggests that neither metric is a helpful or informative indicator of the quality of a trust”, and there are certainly no policy implications. The fact that the study’s fundamental flaw was not picked up by referees, raises concerns about the review process for the BMJ (and indeed for the project's funder). In particular the lack of a statistical critique of the premise of the study, and potentially the impartiality of the review process (one referee starts his review by saying “I hope that publication and discussion of this paper will get rid of this lousy statistic”).
In addition, there are other worrying aspects of the paper. The original proposal [5] sets out to review case notes in the 14 so called “Keogh” trusts investigated for high hospital mortality rates (11 of which were deemed to be so poor as to merit being put into special measures by the Secretary of State) and to compare with 10 other trusts in addition to the 10 trusts included in the earlier PRISM study [6]. There is no mention of this selection process in the paper, and the authors remain vague about their selection criteria, despite being pressed by at least one of the referees for more detail. Either the sampling method changed since the original proposal, or Hogan et al have obscured this important methodological detail. If their approach is not able to at least identify some of 11 trusts put in to special measures, it cast some doubt on the validity of the case note review method. A related concern is that the statistical methods appear to have changed since the original proposal. This perhaps is acceptable if amended prior to the study, however post-hoc changes to the analysis may be subject to the biases of the investigators.
One of the principal investigator’s views are in the public domain [7], and indeed this point is raised in another rapid response [8]. The original proposal states that “each case that is considered to be a preventable death (estimated to be 120) will be discussed with the principal investigator”. If this indeed happened, with the potential for measurement bias, it should have been disclosed in the paper.
In the paper and rapid responses, there does seem to be some agreement that measurement of quality of care in hospitals is a complex issue and that no single measure is ever going to assess the delivery of safe and effective care. This complexity requires that different metrics should be used. Sadly, this poorly designed (yet presumably costly) study brings us no closer to resolving questions about the validity of summary hospital mortality levels, and certainly should not be used as the basis for policy. Given the inadequacy of the design and the potential biases in execution, the study authors should reconsider their conclusions.
Paul Aylin
p.aylin@imperial.ac.uk
REFERENCES
[1] Aylin P, Bottle A, Jarman B. Rapid Response http://www.bmj.com/content/351/bmj.h3239/rr
[2] Hogan H, Zipfel R, Neuburger J, Hutchings A, Darzi A, Black N. Avoidability of hospital deaths and association with hospital-wide mortality ratios: retrospective case record review and regression analysis. BMJ 2015;351:h3239
[3] Abel G, Lyratzopoulus G. Ranking hospitals on avoidable death rates derived from retrospective case record review: methodological observations and limitations. BMJ Qual Saf 2015. doi:10.1136/bmjqs-2015-004366
[4] Abel G, Lyratzopoulus G. Rapid Response http://www.bmj.com/content/351/bmj.h3239/rr-6
[5] Hogan H, Darzi A, Black N. Review of 14 Trusts: investigation of avoidable mortality. http://www.nhs.uk/NHSEngland/bruce-keogh-review/Documents/NAG-retrospect...
[6] Hogan H, Healey F, Neale G, et al. Preventable deaths due to problems in care in English acute hospitals: a retrospective case record review study. BMJ Qual Saf2012:21:737-45.
[7] Black N Assessing the quality of hospitals. Hospital standardised mortality ratios should be abandoned. BMJ 2010;340:c2066 doi:10.1136/bmj.c2066
[8] Ben-Tovim D. Rapid Response http://www.bmj.com/content/351/bmj.h3239/rr-0
Post script
In fact the study’s final regression coefficient with an upper 95% confidence limit of a 0.7% change in avoidable deaths per 1SD change in HSMR is not far from including the true estimated standard deviation of 0.8% of avoidable deaths which would imply a perfect correlation of a 1SD change in outcome for a 1SD change in exposure. However, the wide confidence intervals (between -0.2% and 0.7%) betray the study’s obvious lack of power.
Out of the 14 hospital trusts picked for inspection on the basis of high hospital wide mortality rates for the Keogh review, 11 were deemed to be so poor as to merit being put into special measures by the Secretary of State, suggesting a positive predictive value of 79% for hospital wide summary mortality statistics. Of course, this figure takes no account of what would have been found if hospitals without raised mortality were inspected. The latest published CQC results suggest none of the 26 most recently inspected hospitals were rated as inadequate [http://www.cqc.org.uk/content/hospitals] but these inspections are carried out under a different regime, and are not necessarily comparable to earlier inspections.
Competing interests: The Dr Foster Unit is an academic unit within the Department of Primary Care and Public Health, within the School of Public Health, Imperial College London. The unit receives research grant income from a range of funders, including the National Institute of Health Research and Dr Foster Intelligence, an independent health service research organisation (a wholly owned subsidiary of Telstra). The unit is affiliated with the Patient Safety Translation Research Centre at Imperial which is funded by the National Institute of Health Research. We are also grateful for support from the NIHR Biomedical Research Centre funding scheme.