Changes in hospital safety following penalties in the US Hospital Acquired Condition Reduction Program: retrospective cohort studyBMJ 2019; 366 doi: https://doi.org/10.1136/bmj.l4109 (Published 03 July 2019) Cite this as: BMJ 2019;366:l4109
- Roshun Sankaran, medical student1 2,
- Devraj Sukul, cardiology fellow2,
- Ushapoorna Nuliyalu, statistician3 4,
- Baris Gulseren, health policy analyst1,
- Tedi A Engler, project manager1 3,
- Emily Arntson, graduate student1 2,
- Hanna Zlotnick, MPP student5,
- Justin B Dimick, George D Zuidema professor of surgery2 3 4,
- Andrew M Ryan, associate professor1 3 4
- 1University of Michigan School of Public Health, 1415 Washington Heights, Ann Arbor, MI 48109, USA
- 2University of Michigan Medical School, Ann Arbor, MI, USA
- 3Center for Healthcare Outcomes and Policy, Ann Arbor, MI, USA
- 4Institute for Healthcare Policy and Innovation, Ann Arbor, MI, USA
- 5University of Michigan Gerald R Ford School of Public Policy, Ann Arbor, MI, USA
- Correspondence to: A M Ryan @andy_ryan_dydx on Twitter) (or
- Accepted 25 April 2019
Objective To evaluate the association between hospital penalization in the US Hospital Acquired Condition Reduction Program (HACRP) and subsequent changes in clinical outcomes.
Design Regression discontinuity design applied to a retrospective cohort from inpatient Medicare claims.
Setting 3238 acute care hospitals in the United States.
Participants Medicare fee-for-service beneficiaries discharged from acute care hospitals between 23 July 2014 and 30 November 2016 and eligible for at least one targeted hospital acquired condition (n=15 470 334).
Intervention Hospital receipt of a penalty in the first year of the HACRP.
Main outcome measures Episode level count of targeted hospital acquired conditions per 1000 episodes, 30 day readmissions, and 30 day mortality.
Results Of 724 hospitals penalized under the HACRP in fiscal year 2015, 708 were represented in the study. Mean counts of hospital acquired conditions were 2.72 per 1000 episodes for penalized hospitals and 2.06 per 1000 episodes for non-penalized hospitals; 30 day readmissions were 14.4% and 14.0%, respectively, and 30 day mortality was 9.0% for both hospital groups. Penalized hospitals were more likely to be large, teaching institutions, and have a greater share of patients with low socioeconomic status than non-penalized hospitals. HACRP penalties were associated with a non-significant change of −0.16 hospital acquired conditions per 1000 episodes (95% confidence interval −0.53 to 0.20), −0.36 percentage points in 30 day readmission (−1.06 to 0.33), and −0.04 percentage points in 30 day mortality (−0.59 to 0.52). No clear patterns of clinical improvement were observed across hospital characteristics.
Conclusions Penalization was not associated with significant changes in rates of hospital acquired conditions, 30 day readmission, or 30 day mortality, and does not appear to drive meaningful clinical improvements. By disproportionately penalizing hospitals caring for more disadvantaged patients, the HACRP could exacerbate inequities in care.
Reducing mortality and morbidity from hospital acquired conditions is a national priority in the United States.1 Historically, hospitals have faced limited incentives to invest in efforts to reduce these conditions, and could in fact benefit financially from certain conditions.2 The Hospital Acquired Condition Reduction Program (HACRP) was created by the US Patient Protection and Affordable Care Act to provide stronger incentives for hospitals to reduce hospital acquired conditions. Expanding on the previous Centers for Medicare and Medicare Services (CMS) policy of non-payment for these conditions, the HACRP imposes a penalty of 1% of Medicare payments on acute care hospitals whose performances on hospital acquired condition measures falls in the bottom quarter.345 In fiscal year 2015, 724 hospitals receiving payments from the CMS were penalized an estimated US$373m (£294m; €330m) for poor performance under the HACRP.36
However, how the receipt of a financial penalty for poor quality performance in the HACRP affects subsequent hospital behavior is poorly understood. On one hand, hospitals that are penalized might be motivated to improve through rigorous quality improvement initiatives.7 On the other hand, penalties might not alter existing efforts to improve or could reduce hospital resources for quality improvement over the longer term. Moreover, hospital characteristics that predict responsiveness to penalization under the HACRP and similar pay-for-performance programs are poorly understood.8 Recent research suggests that hospitals caring for more disadvantaged patients are more likely to be penalized under the HACRP.9 If these hospitals do not improve in response to penalization, it could lead to a long term reduction in payments, exacerbating inequities. These knowledge gaps are especially salient given the continued expansion of value based payment models in the US.1011
In this study, we used national Medicare data to evaluate the association between a hospital’s penalty status under the HACRP and subsequent clinical performance in the first two years of the program. We also examined the hospital characteristics that were associated with penalization and performance improvement after penalization.
We used 100% Medicare provider analysis and review (MedPAR) inpatient claims data for Medicare fee-for-service beneficiaries who were discharged from acute care hospitals subject to the HACRP during the study period (fig 1). Citizens or legal permanent residents in the US are eligible for Medicare if they are 65 or older, have certain disabilities, or have end stage renal disease. Beneficiaries who are enrolled in a privately administered plan for Medicare health (and therefore not fee-for-service beneficiaries) were not included in our study.12 Hospitals’ scores for hospital acquired conditions and the list of penalized hospitals in fiscal years 2015 and 2016 were obtained from the CMS website.13 Other hospital level covariates were obtained from the CMS impact files from fiscal years 2014 and 2015, which provided information on geographical region, resident-to-bed ratio, number of beds, disproportionate share percentage (measuring socioeconomic disadvantage), case mix index (measuring patient severity), and Census Bureau statistical area designation (urban or rural). We used data from the 2012-14 annual surveys by the American Hospital Association for information on the proportion of Medicare and Medicaid days, nurse-to-bed ratio, and for-profit status. CMS Hospital Compare Data from 2014 and 2015 provided data on continuous levels of hospital performance under the HACRP.
The study population included all Medicare fee-for-service beneficiaries aged 65 and older who were discharged from an acute care hospital between 23 July 2014 and 30 November 2016 and who met denominator criteria for any of the hospital acquired conditions that comprise the Agency for Healthcare Research and Quality’s Patient Safety Indicator-90 composite (AHRQ PSI-90), which was targeted under the HACRP (n=15 470 334, appendix figure 1). We defined our study cohort using methods similar to those of the CMS hospital wide readmission measure. Patients were excluded from the study population if they were not enrolled in Medicare parts A and B, which cover inpatient hospital admission, outpatient care, and physician services, for 30 days after discharge; if Medicare was not the primary payer for the care episode; if the patient was transferred out of the hospital; if the patient was discharged against medical advice; or if the index admission was due to cancer treatment or the treatment of a psychiatric illness. Acute care hospitals were included if they had been assigned a total score for hospital acquired conditions in fiscal year 2015, were not located in Maryland or Puerto Rico, had data available from at least one American Hospital Association annual survey from 2012 to 2014, and had at least one eligible patient discharge during the study period. Maryland hospitals were excluded because the state is not part of Medicare’s inpatient prospective payment system, and therefore not eligible for the HACRP.
Our primary outcome was the number of hospital acquired conditions per patient episode. We evaluated those conditions that contributed to the AHRQ PSI-90 composite score. The composite score contributed 35% of the total hospital acquired condition score assigned under the HACRP in fiscal year 2015. Included conditions were: pressure ulcer (PSI-03), iatrogenic pneumothorax (PSI-06), central line-associated bloodstream infections (CLABSI; PSI-07), postoperative hip fracture (PSI-08), perioperative pulmonary embolism or deep vein thrombosis (PSI-12), postoperative sepsis (PSI-13), postoperative wound dehiscence (PSI-14), and accidental puncture or laceration (PSI-15). Patient safety indicators were calculated by AHRQ quality indicators software (version 6.0.2).14 We evaluated 30 day readmission and mortality as secondary outcomes.
Hospitals with a total score for hospital acquired conditions in the worst performing quarter (score ≥7.025) received a 1% payment reduction from the CMS for all discharges occurring between 1 October 2014 and 30 September 2015. The CMS calculated these total scores using discharge data from 2011 to 2013. Hospitals were notified of their penalty status on 23 July 2014, before the start of the penalization period.1516171819 Therefore, we included all discharges between 23 July 2014 (which marked the beginning of the review and correction period for penalized hospitals) and 30 November 2016 in our analyses (fig 1).
We used a regression discontinuity study design to investigate whether penalization was associated with improvements in the study outcomes. This design leverages the fact that hospitals immediately above and below the financial penalty threshold are unlikely to differ in ways that affect study outcomes, other than exposure to the HACRP financial penalty.2021 These designs can therefore closely mimic randomized controlled trials under the assumption that hospitals cannot manipulate their total hospital acquired condition scores around the penalty threshold.22 Regression discontinuity designs are also resistant to confounding from regression to the mean.23
We fit separate models for each outcome of interest, estimating robust, bias corrected, treatment effects with local linear regression using data driven bandwidth selection.24 Evidence suggests that this approach provides the most reliable treatment effect estimates and confidence intervals with the best coverage in regression discontinuity designs.2425 To examine whether hospital characteristics influenced responsiveness to penalization, we estimated stratified models across subgroups of hospitals. Subgroup analyses were performed by hospital size, disproportionate share index, case mix index,26 proportion of Medicare patient days, proportion of Medicaid patient days, nurse-to-bed ratio, for-profit status, teaching status, urbanicity, and whether the hospital was also penalized under the Hospital Readmissions Reduction Program.
We did several sensitivity analyses to support the validity of our results. Firstly, to examine whether variables other than penalization could have affected study outcomes, we performed falsification tests27 by testing discontinuities for several factors that should not be affected by the penalty threshold. These factors included hospital size (both as continuous and categorical variables), for-profit status, Medicare percentage of discharges, teaching status, and penalization status in fiscal year 2016 (appendix table 1, appendix figures 2-10). We then graphically assessed the presence of any discontinuity in the distribution of total hospital acquired condition scores around the penalization threshold, which could indicate potential manipulation of scores (fig 2).
We also estimated alternate model specifications for each outcome using a fourth order polynomial fit and controlling for patient and hospital characteristics (appendix table 2). Total hospital acquired condition scores are a composite measure of two scores, referred to as Domain 1 (the AHRQ PSI-90 composite) and Domain 2 (the Centers for Disease Control and Prevention’s National Healthcare Safety Network healthcare-associated infection measure). We estimated models using patient discharges only from hospitals that reported both Domain 1 and Domain 2 scores under the HACRP, which differ systematically from hospitals that reported only Domain 1 scores (appendix figure 11).28 Finally, because hospitals could have focused quality improvement efforts on perioperative pulmonary embolism or deep vein thrombosis (PSI-12) and accidental puncture or laceration (PSI-15)—which are weighted most heavily in the AHRQ PSI-90 composite used to determine penalties in the first years of the HACRP—we fit models using the counts of only these two patient safety indicators as the outcome of interest (appendix figure 12).
Standard errors in all models were robust to hospital level clustering. P values were two sided with a threshold for significance of less than 0.05. All statistical analyses were conducted by use of Stata version 15 (StataCorp, College Station, TX, USA). This study was deemed exempt from review by the University of Michigan’s institutional review board.
Patient and public involvement
No patients were involved in setting the research question or the outcome measures, nor were they involved in developing plans for the design or implementation of the study. No patients were asked to advise on the interpretation or writing up of results. There are no specific plans to disseminate the results of the research to study participants or the relevant patient community.
Our analytical sample included 15 470 334 discharges of patients between 23 July 2014 and 30 November 2016. Patients were eligible for at least one hospital acquired condition from 3238 hospitals. Of 724 hospitals penalized by the CMS under the HACRP in fiscal year 2015, 708 were represented in our sample.
Patients discharged from non-penalized hospitals had 2.06 hospital acquired conditions per 1000 patient episodes, while patients discharged from penalized hospitals had 2.72 conditions per 1000 patient episodes (table 1). The most common hospital acquired conditions that eligible patients experienced among all study hospitals were perioperative pulmonary embolism or deep vein thrombosis (0.91 and 1.23 per 1000 discharges for non-penalized and penalized hospitals, respectively), iatrogenic pneumothorax (0.30 and 0.36), postoperative sepsis (0.50 and 0.67), and pressure ulcers (0.15 and 0.21).
Penalized hospitals were more likely to be large, in the highest quarter for disproportionate share patients, and teaching institutions (table 1). Penalized hospitals also had a smaller proportion of inpatient days from Medicare patients, a larger proportion of inpatient days from Medicaid patients, and a higher nurse-to-bed ratio. Although they differed in aggregate, penalized and non-penalized hospitals did not differ meaningfully at the penalty threshold, as shown by falsification tests (appendix figures 2-10). These tests of covariates showed no significant discontinuity at the penalty threshold, although discontinuities in hospital size were close to significant (appendix table 1; appendix figures 2 and 6). A sensitivity analysis controlling for hospital size resulted in similar findings. (appendix table 2).
Figure 3 shows the relation between the CMS hospital acquired condition score and the study outcomes on both sides of the penalization threshold. A change in the intercept at the penalization threshold would indicate an association between penalization and the study outcome. Figure 4 shows the estimated association between financial penalization under the HACRP and each study outcome, along with estimated effects for subgroup analyses.
Regression discontinuity analysis showed that, overall, financial penalization under the HACRP was associated with a non-significant decrease of −0.16 hospital acquired conditions per 1000 patient episodes (95% confidence interval −0.53 to 0.20; fig 3 and fig 4). This finding was robust to alternate model specifications (appendix table 2a). Similar non-significant changes in hospital acquired condition rates were found for every hospital subgroup analyzed except for small hospitals, for which penalization was associated with change of −0.54 conditions per 1000 patient episodes (−1.03 to −0.05; fig 4). Yet these apparent improvements for small hospitals were sensitive to the choice of bandwidth (appendix tables 7-8), suggesting this finding is likely by chance.
The 30 day readmission rate was 14.0% for non-penalized hospitals and 14.4% for penalized hospitals in fiscal year 2015. Penalization under HACRP was associated with a non-significant change in readmissions of −0.36 percentage points (95% confidence interval −1.06 to 0.33; fig 3 and fig 4). This finding was also robust to alternate model specifications (appendix table 2b). Financial penalties were associated with significant reductions in 30 day readmission rates for small hospitals (−1.28 percentage points (−2.45 to −0.11)), and hospitals in the highest quarter for nurse-to-bed ratios (ratio ≥2.26; −2.20 percentage points (−3.68 to −0.71); fig 4).
Non-penalized hospitals had an average 30 day mortality rate of 9.0%, identical to that of penalized hospitals. We saw no significant association between a hospital’s penalty status and its subsequent 30 day mortality rate (−0.04 percentage points (95% confidence interval −0.59 to 0.52)), a finding that was robust to alternate model specifications (fig 3, fig 4, and appendix table 2c). Penalization was significantly associated with mortality among the lowest quarter of DSH hospitals (−1.78 percentage points (−2.74 to −0.82)), highest quarter of Medicare discharges (−1.70 (−2.51 to −0.89)), and for-profit hospitals (−2.13 (−3.71 to −0.055); fig 4). However, penalization was not associated with similar changes in HAC rates among these groups, making their mortality reductions likely to be chance findings.
In this national study of the effect of the HACRP penalty on hospital acquired condition rates and other clinical outcomes using a regression discontinuity design, we report three main findings. Firstly, penalization under the program was more likely to occur for large, academic medical centers and hospitals that care for a higher proportion of disadvantaged patients. Secondly, penalization was not associated with a significant change in the rate of hospital acquired conditions. Thirdly, penalization was not associated with a significant overall change in important clinical outcomes, including 30 day readmission and 30 day mortality. While penalization was associated with a reduction in 30 day readmission and 30 day mortality for some subgroups, this finding is most likely to be by chance because the same subgroups did not show a concurrent reduction in hospital acquired condition rates. Penalization was also not associated with any clear pattern of results across different hospital characteristics. Overall, these findings suggest that financial penalties levied against hospitals performing poorly under the CMS’s HACRP have not meaningfully improved patient safety.
This study evaluates the effect of financial penalization under the HACRP on subsequent rates of hospital acquired conditions and other clinical outcomes. Pay-for-performance programs have been widely implemented throughout the world, including the Quality and Outcomes Framework in the United Kingdom,29 hospital pay-for-performance in France,30 and many others.3132 However, we are not aware of strictly penalty based programs for patient safety programs outside of the US. Our study is consistent with previous research showing that penalties in the HACRP are more likely among major teaching hospitals and those caring for more disadvantaged patients.9 Our findings add to this literature by demonstrating that penalization does not appear to drive performance improvement in the program. Research finding that penalization was associated with improvement in the context of the Hospital Readmission Reduction Program could be biased by mean reversion; because penalties are determined in part on the basis of random noise, so-called improvement in the program could be driven by a return to hospitals’ steady state performance.3334 Regression discontinuity designs using some form of pre-intervention outcome to determine treatment are not subject to this bias.23 This is because, in the absence of treatment, expectations for differences in post-intervention outcomes between treated and untreated units are the same, because both groups are close to the treatment threshold, or can be controlled for through statistical modeling.
This study should be interpreted in the context of several important limitations. Firstly, while the HACRP evaluates hospitals using measures from both the AHRQ PSI-90 and the Centers for Disease Control and Prevention’s (CDC) National Healthcare Safety Network, because CDC data were not available, our study outcomes consisted only of measures included in the AHRQ PSI-90. In response to penalization, hospitals might have selectively targeted CDC measures, knowing that those were more heavily weighted under the HACRP. Nonetheless, we saw a strong correlation between patient safety indicator scores and overall hospital acquired condition scores (appendix table 3). Hospitals also varied with respect to reporting data on the CDC measures. The likelihood of reporting was correlated with hospital size, and the probability of penalization under the HACRP differs by domains reported.28 However, sensitivity analysis restricting our sample to discharges from only those hospitals that reported both Domain 1 and Domain 2 scores also found that penalization was not associated with any significant changes in rates of hospital acquired conditions. In addition, hospitals might have improved significantly on the narrow set of patient safety indicators that were most heavily weighted by the AHRQ PSI-90 composite.35 Again, sensitivity analysis found this potential bias was not the case.
It is also possible that hospitals were unable to make substantial process changes to improve rates of hospital acquired conditions in the timeframe we analyzed. However, this is unlikely for two reasons. The HACRP was announced at the beginning of fiscal year 2014, meaning that all hospitals had a full year to adapt to the program and implement changes before the first penalization period in fiscal year 2015.36 Furthermore, penalized hospitals were made aware of their impending penalization in July 2015, and our study period spanned nearly two and a half years after penalization, giving hospitals ample time to improve.
Factors other than penalization could have affected study outcomes, including participation in other value based payment reforms. Yet these reforms would only confound our results if they occurred precisely at the penalization threshold, which they did not. Another limitation was that our analysis might have been underpowered to detect heterogeneous responses to penalization. Inconsistencies among subgroup analyses of 30 day readmission and mortality support this limitation. Finally, our regression discontinuity approach indicated that penalization under the HACRP did not result in reduced rates of hospital acquired conditions, readmission, or mortality for hospitals close to the penalization threshold, but does not provide insight into the effects of penalization on hospitals that performed far below the threshold. Nonetheless, these findings show that hospitals with program performances close to the threshold, and therefore with the best chance at improving their hospital acquired condition scores in subsequent years, failed to do so in response to penalization.
Our findings that penalization under the HACRP was systematically associated with hospital characteristics but not associated with improved performance have important implications. Our study suggests the HACRP could be decreasing equity while not improving quality by penalizing hospitals caring for more disadvantaged patients.
The CMS should consider redesigning the HACRP to deal with two major design challenges. Firstly, instead of levying all-or-nothing penalties for hospitals performing in the bottom quarter, the CMS should consider graduated penalties for all hospitals with higher than expected rates of hospital conditions. This approach, used in the Hospital Readmission Reduction Program, is more equitable and provides incentives for improvement among a larger range of hospitals. Secondly, to improve equity, the CMS should consider modifying penalty thresholds based on hospitals’ share of disadvantaged patients. This potential change would be similar to recent reform to the Hospital Readmission Reduction Program, which established different penalty thresholds for different types of hospitals.37 Thirdly, the CMS should eliminate the financial disincentive to being scored on the CDC measures, which could be accomplished by creating separate penalty criteria according to whether hospitals are scored on the CDC measures. Fundamentally, future research should assess whether the measures used to evaluate patient safety and the design of the financial incentives in the HACRP are appropriately constructed to improve patient safety.
What is already known on this topic
Historically, hospitals have faced limited financial incentives to reduce hospital acquired conditions
Under the US Hospital Acquired Condition Reduction Program, the Centers for Medicare and Medicaid Services impose financial penalties on hospitals with the lowest safety performance (bottom quarter)
What this study adds
Penalization under the Hospital Acquired Condition Reduction Program was not associated with a significant change in the rate of hospital acquired conditions, 30 day readmission, or 30 day mortality
By penalizing hospitals that care for a higher share of patients with low socioeconomic status, the program could exacerbate inequities in care
Contributors: RS, DS, EA, and AMR were responsible for the study concept and design. RS, UN, JBD, AMR, and BG contributed to the acquisition, analysis, and interpretation of the data. RS drafted the manuscript. All the authors were responsible for the critical revision of the manuscript for important intellectual content. RS, UN, AMR, and BG undertook the statistical analysis. JBD and AMR obtained funding. JBD, AMR, and TAE provided administrative, technical, and material support. The corresponding author attests that all listed authors meet authorship criteria and that no others meeting the criteria have been omitted. AMR is guarantor.
Funding: AR acknowledges funding from the National Institute on Aging (grant R01AG047932) and the Agency for Healthcare Research and Quality (R01 HS026244). JBD acknowledges funding from the National Institute on Aging (grant R01AG039434). The funding agencies had no involvement in the analysis presented herein.
Competing interests: All authors have completed the ICMJE uniform disclosure form at www.icmje.org/coi_disclosure.pdf and declare: support from the National Institute on Aging and the Agency for Healthcare Research and Quality for the submitted work; JBD has a financial interest in ArborMetrix, which has no role in the present analysis; no other financial relationships with any organizations that might have an interest in the submitted work in the previous three years; no other relationships or activities that could appear to have influenced the submitted work.
Ethical approval: This study was deemed exempt from oversight by the University of Michigan’s institutional review board.
Data sharing: The analytical files are not available for sharing. The analytical code will be made publicly available and is available on request.
The senior author affirms that the manuscript is an honest, accurate, and transparent account of the study being reported; that no important aspects of the study have been omitted; and that any discrepancies from the study as planned (and, if relevant, registered) have been explained.
This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.