Published 20 November 2008, doi:10.1136/bmj.a2097
Cite this as: BMJ 2008;337:a2097

Analysis

Fairness of financial penalties to improve control of Clostridium difficile

¹ National Institute for Health Research Oxford Biomedical Research Centre Programme, John Radcliffe Hospital, Oxford OX3 9DU, ² MRC Clinical Trials Unit, London NW1 2DA, ³ MRC Biostatistics Unit, Institute of Public Health, Cambridge CB2 0SR, ⁴ Statistical Laboratory, Centre for Mathematical Sciences, Cambridge CB3 0WB, ⁵ Nuffield Department of Clinical Laboratory Sciences, University of Oxford, Oxford, OX3 9DU

Trusts in England have been given targets to reduce C difficile infection rates. But Sarah Walker and colleagues argue that the way they are calculated makes financial penalties almost impossible to avoid

Financial incentives are increasingly being used to promote delivery of efficient, high quality health care.^{1 2 3} However, it is widely agreed that such incentives have to be carefully constructed, with clear and simple links between behaviours or outcomes and incentives, direct indicators that validly measure the desired behaviours or outcomes, and sufficient stability to give surety to hospitals that effort (time, money) put into achieving targets is warranted.^{1 3} We examine how well these criteria are met by the financial penalties introduced for failing to reduce hospital acquired Clostridium difficile infection as part of the 2008-9 standard National Health Service contract for acute services.

What are the targets and financial penalties?

In response to a new hypervirulent strain of C difficile causing large hospital outbreaks since 2003,^{4 5} the Department of Health recently set a target of a 30% reduction in C difficile infections in patients aged 2 years or older across the entire health economy (within and outside acute NHS trusts) from 2007-8 to 2010-11.⁶ All financial penalties relating to this target are based on comparing the number of cases observed in the current "contract year" to the number of cases observed in the previous "baseline" financial year. If 200 or more cases are observed in the baseline or contract year, the penalty is based on failing to meet the target reduction in the contract year (0.2% of total contract year revenue for every percentage point the target is underachieved, capped at 2% when target underachieved by 10%). However, if fewer than 200 cases are observed in the baseline and contract year, the penalty is based on exceeding the number of baseline cases (0.1% for every percentage point increase or 0.05% if baseline is below 50 cases, capped at 2% when there has been a 20% or 40% increase over baseline respectively).

Targets will be set by primary care trusts, which commission services from hospital trusts. Assuming, for simplicity, that the average target reduction is 10% a year, the financial penalties follow the trajectories in fig 1. Although the penalty is capped at 2% of income, the amounts of money are not trivial. A large trust would have a contract revenue of around £500m (600m; $750m), and thus a 2% penalty would be £10m and even 0.1% would be £500 000, the equivalent of 10 mid-grade nurses.

View larger version (48K): [in this window] [in a new window] [PowerPoint Slide for Teaching]   Fig 1 Financial penalty (percentage of total contract revenue) by observed change in number of C difficile cases between contract and baseline year assuming a target reduction of 10%. For example, a trust with 100 baseline cases would be set a target of 90 (10% reduction) and if it recorded 110 cases (10% increase) would be penalised 1% of its total contract revenue

 
The specified financial penalties have extreme discontinuities. For example, a trust with 199 baseline cases will escape financial penalty if it has 199 cases or fewer in the contract year since in this situation the penalty is based on percentage excess over baseline (0%). If, however, it has 200 cases, the penalty instead becomes based on underachieving the 10% target reduction, and immediately increases to 2% because the target has been missed by more than 10%. Thus a single extra case could cost the trust millions of pounds. The consequences of very small increases in the number of cases, entirely explainable by chance, could be particularly severe for both South Devon Healthcare and York Hospitals which had 198 cases in the first baseline year 2007-8,⁷ and another six trusts with 192-195 baseline cases.

Calculating likely financial penalties

Suppose that a trust strengthens or introduces practices believed to improve C difficile control, such as enhanced cleaning with 10% sodium hypochlorite, use of barrier precautions, thorough handwashing with chlorhexidine or soap and water, and changes in antibiotic policies.^{8 9 10 11} The expected financial penalty will depend on three factors:

• The true reduction in C difficile risk from these measures, where risk can be thought of as the true underlying rate
  
• How closely the observed reduction in cases matches this true underlying risk reduction
  
• The formula for calculating financial penalty.
  

The key point is that C difficile infection is not under a hospital’s complete control. C difficile infection occurs relatively rarely (around 2-3 cases/1000 days among hospital patients aged over 65⁷) but has very many opportunities to happen. Therefore the number of cases observed is subject to considerable natural random variation; if the true risk reduction is 10%, the observed reduction is equally likely to be above or below that figure (fig 2). In this situation, the only way to estimate expected penalties is by simulating cases in a baseline year and various true reductions or increases in C difficile risk across three further years and then calculating the average financial penalties. We simulated the consequences of various genuine reductions in C difficile risk (from a 20% reduction to a 10% increase) across four years of observation, with the expected number of cases in the baseline year ranging from 25 to 1000, and estimated the financial penalties over one and three years, assuming that the target is a 10% annual reduction. We repeated each simulation 50 000 times. Poisson distributions were assumed for count data: these assume independent observations in each year and in practice there is considerable clustering of cases leading to substantial volatility from year to year. For example, between 2004 and 2007, the median coefficient of variation (standard deviation divided by mean) of the annual C difficile counts for people over 65 was 0.21: this "average volatility" was represented by Swindon and Marlborough Trust, which had 246, 230, 349, and 322 cases. This extra-Poisson variability means that our estimates can therefore be assumed to be conservative.

View larger version (49K): [in this window] [in a new window] [PowerPoint Slide for Teaching]   Fig 2 Observed number of cases of C difficile over four years in 20 simulated hospitals with expected 400 baseline cases and an annual 10% true risk reduction

 
If the target is a 10% annual reduction, around half of trusts with 250-1000 baseline cases and a true 10% risk reduction would still pay a financial penalty in the following contract year (fig 3 (top)). Once the 200 baseline case threshold is crossed, trusts with smaller numbers of cases pay larger mean penalties despite the same true 10% risk reduction. This is because the risk reduction can be estimated more precisely with more cases, and also because 1/baseline cases, on which the incremental financial penalty is based, goes down faster than (90%xbaseline cases), on which the width of the sampling distribution in the contract year is based, goes up. To have a small (<5%) chance of not paying any financial penalty under the formulas for a 10% target, trusts will have to exceed the target by 5-15% (fig 4).

View larger version (32K): [in this window] [in a new window] [PowerPoint Slide for Teaching]   Fig 3 Percentage of trusts paying any financial penalty and mean financial penalty incurred over one contract year according to expected C difficile cases in baseline year

 

View larger version (22K): [in this window] [in a new window] [PowerPoint Slide for Teaching]   Fig 4 True reduction in risk of C difficile required for trusts to have <5% chance of paying a financial penalty over one year

 
When there is truly no reduction in C difficile risk, over 90% of trusts with 250-1000 baseline cases will pay a financial penalty in the following contract year, although this will not necessarily be the full 2% (mean 1.4-1.8%, fig 3(bottom)).

Thus the 98 (58%) trusts with more than 250 cases in 2007-8 that believe they can achieve the target risk reductions should still reserve 0.4-0.6% total contract revenue for financial penalties; those that do not think they can achieve this should probably reserve the full 2%.

Problems with stipulated financial penalties

The stipulated financial penalties implicitly assume that rates of C difficile infections can be estimated precisely enough to know that an observed 1-10% underachievement of target is actually a true underachievement. This simply does not hold: observed underachievements may merely be an artefact of variability in the observed rate reduction. This variability is increased by using a single baseline year, leading to the well known statistical problem regression to the mean.^{12 13 14} If the number of baseline cases observed happens to be high by chance, the number of cases in the following year is likely to be lower just because getting two random high observations is less likely. Similarly, if the number of observed baseline cases is low by chance, the number in the next year is likely to be closer to or higher than average (fig 2). For example, Taunton and Somerset NHS Trust’s figures for people aged over 65 between 2004 and 2007 are 331, 399, 409, 173, suggesting that the recent reduction may be at least partly due to chance, and giving them an extremely challenging first baseline year. Although both situations are equally likely, the financial penalties work in only one direction, with only chance low numbers followed by natural highs being penalised harshly.

Unfortunately, the rolling baseline year maximises the cumulative impact of regression to the mean over three years. Even if the true underlying risk is going down according to target, sampling error means it is highly likely that the cases observed will lie above the expected trajectory at least once in three years and thus a financial penalty will be paid at least once. In fact, if the target is a 10% annual reduction, 95% of trusts with 250-1000 baseline cases and a true 10% annual risk reduction would pay a financial penalty at least once over three years (fig 5). Trusts would have to reduce risk by 10-15% more than the target to have a less than 5% chance of paying a penalty over the three years (fig 6). As an example, East Sussex Hospitals NHS Trust had 393, 387, 374, and 303 cases in people aged over 65 in successive years from 2004 to 2007—if such a penalty scheme had been in place they would have paid a penalty in two of the three years owing to not achieving a 10% reduction, despite an observed reduction of 30% between the baseline year of 2004 and 2007.

View larger version (32K): [in this window] [in a new window] [PowerPoint Slide for Teaching]   Fig 5 Percentage of trusts paying any financial penalty and mean financial penalty incurred over three contract years according to expected C difficile cases in baseline year

 

View larger version (21K): [in this window] [in a new window] [PowerPoint Slide for Teaching]   Fig 6 True annual reduction in risk of C difficile required for trusts to have <5% chance of paying a financial penalty over three years

 

What should trusts expect?

Figure 7 can be used to estimate the probability of paying a financial penalty and the mean financial penalty incurred over one and three contract years for trusts with various numbers of baseline cases according to different reductions in risk of C difficile infection. Although our simulations assume a 10% annual target reduction, the results generalise to different targets as they are based on well known statistical properties such as sampling variability and regression to the mean. It is clear that achieving the required target is not predictably linked to avoidance of severe financial penalties.

View larger version (45K): [in this window] [in a new window] [PowerPoint Slide for Teaching]   Fig 7 Percentage of trusts paying financial penalties and mean financial penalty incurred over one and three contract years according to true reduction in C difficile risk

 

A fairer approach?

If investing effort to achieve the required target does not predictably avoid penalty, it may be more efficient simply to accept the penalty.¹ Trusts will have to balance the costs and feasibility of achieving large enough real reductions in C difficile infections to minimise the chance that their observed reduction will not incur a penalty against the costs of merely accepting the penalty whether or not they have truly achieved the required target. We believe the current approach runs a serious risk of demotivating trusts, since they stand a 50% chance of being penalised in any one year even if they achieve target reductions in risk and are almost certain to be penalised at least once in three years, just as a consequence of natural random variation.

Given this natural variation, identifying a financial penalty scheme with high specificity (a high chance of penalising trusts not achieving target) and high sensitivity (high chance of not penalising trusts achieving target) is impossible. We therefore consider that a fairer financial penalty would aim for consistently high sensitivity—that is, it should have a low chance of penalising trusts whose observed cases are compatible with the target risk reduction, allowing for natural random variation. It should not use a single year as a baseline nor a rolling baseline year, as this maximises regression to the mean effects, and it should not differentially penalise trusts with different baseline number of cases but the same relative risk reduction.

One proposal would be to estimate a baseline number of cases for each trust using the mean across the last three years. Although this involves some estimation because relevant data were not collected in all subgroups throughout this period,⁷ it will at least reduce variation in the major subgroup of over 65s in hospital. From this baseline, expected target reductions could be calculated for the next three years, and 90th or 95th percentiles around these designated a breach. The Healthcare Commission recommends and uses this approach.¹⁵ Trusts with observed cases above the breach level could be penalised by various degrees according to their tail probability for the expected sampling distribution. For example, for an average trust with a mean of 300 cases a year during 2005-7, successive 10% annual reductions would be 270, 243, and 219 cases a year. The 90th (95th) percentiles sampled from these expected reductions are 291 (297), 263 (269), and 238 (244) respectively. Penalising observed numbers of cases above these levels would approximate sensitivity of 90% or 95%.

General effects of financial penalties

Even if such a financial penalty achieved the presumed goal of encouraging NHS trusts to make the C difficile target one of their highest priorities, a major question remains as to its actual effect on service provision. End of year penalties mean money has to be put aside and not spent on services during a financial year to ensure that fines can be paid at the end of it. Furthermore, primary care trusts will receive large sums of money from these penalties at the end of each financial year, which will then have to be disposed of in some (as yet undetermined) way. Financial targets, particularly as implemented in the current NHS contract for acute services, may therefore have unexpected and unwanted effects on the smooth running of trusts, other than achieving the primary aims of much needed reductions in outcomes such as C difficile infections.

Cite this as: BMJ 2008;337:a2097

---

Contributors and sources: This article arose from a query to ASW, DWC, TEAP, and JM as to the expected financial resources which should be set aside to meet the stipulated financial penalties at different levels of target achievement. ASW, TEAP, and DWC conceived and designed the simulation study, which ASW set up and analysed. All authors contributed to interpretation of data, drafting the article, and revising it critically for structure and important intellectual content, as well as approving the final version. ASW is the guarantor.

Funding: This project was funded as part of the National Institute for Health Research Oxford Biomedical Research Centre Programme.

Competing interests: None declared.

Provenance and peer review: Not commissioned; externally peer reviewed.

References

1. Duckett SJ. Design of price incentives for adjunct policy goals in formula funding for hospitals and health services. BMC Health Serv Res 2008;8:72.[CrossRef][Medline]
2. Lindenauer PK, Remus D, Roman S, Rothberg MB, Benjamin EM, Ma A, et al. Public reporting and pay for performance in hospital quality improvement. N Engl J Med 2007;356:486-96.[Abstract/Free Full Text]
3. Petersen LA, Woodard LD, Urech T, Daw C, Sookanan S. Does pay-for-performance improve the quality of health care? Ann Intern Med 2006;145:265-72.[Abstract/Free Full Text]
4. McDonald LC, Killgore GE, Thompson A, Owens RC Jr, Kazakova SV, Sambol SP, et al. An epidemic, toxin gene-variant strain of Clostridium difficile. N Engl J Med 2005;353:2433-41.[Abstract/Free Full Text]
5. Pepin J, Valiquette L, Cossette B. Mortality attributable to nosocomial Clostridium difficile-associated disease during an epidemic caused by a hypervirulent strain in Quebec. CMAJ 2005;173:1037-42.[Abstract/Free Full Text]
6. UK Department of Health. National target to reduce Clostridium difficile infections: SHA envelopes—reissue. www.dh.gov.uk (search for: 9397).
7. Health Protection Agency. Clostridium difficile. www.hpa.org.uk/web/HPAweb&HPAwebStandard/HPAweb_C/1216193832974.
8. Fowler S, Webber A, Cooper BS, Phimister A, Price K, Carter Y, et al. Successful use of feedback to improve antibiotic prescribing and reduce Clostridium difficile infection: a controlled interrupted time series. J Antimicrob Chemother 2007;59:990-5.[Abstract/Free Full Text]
9. Gerding DN, Muto CA, Owens RC Jr. Measures to control and prevent Clostridium difficile infection. Clin Infect Dis 2008;46(suppl 1):S43-9.[CrossRef][Web of Science][Medline]
10. Muto CA, Blank MK, Marsh JW, Vergis EN, O’Leary MM, Shutt KA, et al. Control of an outbreak of infection with the hypervirulent Clostridium difficile BI strain in a university hospital using a comprehensive "bundle" approach. Clin Infect Dis 2007;45:1266-73.[CrossRef][Web of Science][Medline]
11. Valiquette L, Cossette B, Garant MP, Diab H, Pepin J. Impact of a reduction in the use of high-risk antibiotics on the course of an epidemic of Clostridium difficile-associated disease caused by the hypervirulent NAP1/027 strain. Clin Infect Dis 2007;45(suppl 2):S112-21.[CrossRef][Web of Science][Medline]
12. Healy MJ, Goldstein H. Regression to the mean. Ann Hum Biol 1978;5:277-80.[CrossRef][Web of Science][Medline]
13. Morton V, Torgerson DJ. Effect of regression to the mean on decision making in health care. BMJ 2003;326:1083-4.[Free Full Text]
14. Morton V, Torgerson DJ. Regression to the mean: treatment effect without the intervention. J Eval Clin Pract 2005;11:59-65.[CrossRef][Web of Science][Medline]
15. Healthcare Commission. Statistical banding for national target indicators, annual health check 2006/2007. http://ratings2007.healthcarecommission.org.uk/Indicators_2007/Downloads/Stats_banding_targets_methods_2_12-0.pdf.

(Accepted 15 September 2008)

CiteULike    Complore    Connotea    Del.icio.us    Digg    Reddit    StumbleUpon    Technorati    What's this?

Relevant Article

Effect of regression to the mean on decision making in health care

Veronica Morton and David J Torgerson
BMJ 2003 326: 1083-1084. [Extract] [Full Text] [PDF]

Online poll

Find out more >>