Exempting dissenting patients from pay for performance schemes: retrospective analysis of exception reporting in the UK Quality and Outcomes FrameworkBMJ 2012; 344 doi: http://dx.doi.org/10.1136/bmj.e2405 (Published 17 April 2012) Cite this as: BMJ 2012;344:e2405
- Tim Doran, clinical research fellow1,
- Evangelos Kontopantelis, research fellow1,
- Catherine Fullwood, research associate2,
- Helen Lester, professor of primary care3,
- Jose M Valderas, clinical lecturer4,
- Stephen Campbell, senior research fellow1
- 1Health Sciences Research Group-Primary Care, University of Manchester, Manchester M13 9PL, UK
- 2Manchester Academic Health Science Centre, Central Manchester University Hospitals NHS Foundation Trust, University of Manchester
- 3School of Health and Population Sciences, University of Birmingham, Birmingham, UK
- 4NIHR School for Primary Care Research, Health Services and Policy Research Group, Department of Primary Care Health Sciences, University of Oxford, Oxford, UK
- Correspondence to: T Doran
- Accepted 6 March 2012
Objective To examine the reasons why practices exempt patients from the UK Quality and Outcomes Framework pay for performance scheme (exception reporting) and to identify the characteristics of general practices associated with informed dissent.
Design Retrospective analysis.
Setting Data for 2008-9 extracted from the clinical computing systems of general practices in England.
Participants 8229 English family practices.
Main outcome measures Rates of exception reporting for 37 clinical quality indicators, associations of patient and general practice factors with exception rates, and financial gain for practices relating to their use of exception reporting.
Results The median rate of exception reporting was 2.7% (interquartile range 1.9-3.9%) overall and 0.44% (0.14-1.1%) for informed dissent, but variation in rates was wide between practices and across indicators. Common reasons for exception reporting were logistical (40.6% of exceptions), clinical contraindication (18.7%), and patient informed dissent (30.1%). Higher rates of informed dissent were associated with: higher numbers of registered patients, higher levels of local area deprivation, and failure of the practice to secure maximum remuneration in the previous year. Exception reporting increased the cost of the scheme by £30 844 500 (€36 877 700; $49 053 200) (£0.58 per patient), with two indicators accounting for a quarter of this additional cost.
Conclusions The provision to exception report enables practices to exempt dissenting patients without being financially penalised. Relatively few patients were excluded for informed dissent, however, suggesting that the incentivised activities were broadly acceptable to patients.
Since early 2000 payers across healthcare systems worldwide have experimented with pay for performance schemes that explicitly link doctors’ remuneration to quality of care, with mixed success.1 2 3 In the United Kingdom, a national scheme for primary care—the Quality and Outcomes Framework—was introduced in 2004, providing financial incentives to family practices for meeting targets on a range of clinical, organisational, and patient experience indicators.4 Most practices have performed well under the scheme,5 6 7 8 but improved patient outcomes have not been consistently evident.9 10 11 Even if they successfully stimulate improved performance, pay for performance schemes have several potential unintended consequences. In particular, given a method of remuneration that financially rewards doctors for performing procedures, prescribing drugs, and controlling biological variables, patients may be coerced or refused care if they are non-compliant.12 13 Understanding these potential risks, the designers of the Quality Outcomes Framework included two mechanisms intended to protect patients from coercive care. Firstly, upper payment thresholds are set below 100%, so practices do not have to achieve the targets for all patients to receive the maximum payment. Secondly, doctors are permitted to use their clinical judgment to remove inappropriate patients from achievement calculations for clinical indicators, a process known as exception reporting. The box gives permitted reasons for exception reporting, including logistical considerations (for example, recent registration of the patient with the practice), clinical reasons (for example, a contraindication to treatment), and patient informed dissent (that is, not agreeing to the investigation or treatment). Patients recently registered with the practice or with a recent diagnosis are automatically excepted by clinical computing systems, whereas practices must actively identify patients who meet other exception reporting criteria. The provision to except dissenting patients is intended to counter any financial conflict of interest for doctors in respecting a patient’s choice to refuse an intervention incentivised under the scheme.
Reasons practice staff may use to exempt patients from quality assessment*
The patient has recently received a diagnosis or recently registered with the practice†
A specified investigative service is unavailable to the practice
Clinical—contraindication or intolerance
The patient has had an allergic or other adverse reaction to a specified drug or has another contraindication to the drug
The patient has not tolerated the drug
The patient is taking the maximal tolerated dose of a drug, but the levels remain suboptimal
The indicator is judged inappropriate for the patient because of particular circumstances, such as terminal illness or extreme frailty‡
The patient has a supervening condition that makes the specified treatment clinically inappropriate
The patient has received at least three invitations for a review during the preceding 12 months but has not attended
The patient refuses to be reviewed‡
The patient does not agree to a specific investigation or treatment§
*Not all reasons are available for every indicator—for example, there is no “contraindication” option for indicators relating to measuring biological variables
†The definition of “recent” is three months in the case of measurement indicators (for example, measurement of blood pressure) and nine months in the case of treatment and outcomes indicators (for example, control of blood pressure within target levels)
‡Patient is excluded for the whole clinical domain (for example, all diabetes indicators)
§Patient is excluded for this activity across all clinical domains (for example, measurement of blood pressure)
The principal drawback of exception reporting is that it allows practices to receive the maximum remuneration without necessarily providing the required care for all eligible patients.14 If exception rules are applied too readily or inappropriately, high achievement scores will mask suboptimal care.4 15 To date, overall exception reporting rates have generally been low (less than 6%),16 with little evidence of widespread fraud.17 However, a few practices have achieved high scores by excluding unusually large numbers of patients.18 19 Concerns over inappropriate use of exception reporting have led to calls for the provision to be amended.20 Without the ability to exception report, practices could be penalised for respecting the wishes of dissenting patients, particularly in localities where rates of dissent are high. Recent qualitative work suggests that doctors perceive exception reporting to be an important, defensible safeguard against inappropriate or over treatment of patients.21 However, to date no research has been carried out into the reasons why practices exception report patients and in particular how high rates of patient dissent are for different population groups. We examined the reasons given by practices for exception reporting patients and identified patient and practice characteristics associated with informed dissent.
In 2008-9 the Quality and Outcomes Framework consisted of 127 quality indicators, of which 65 covered the maintenance of disease registers, organisational activities, and patient experience of care. The remaining 62 indicators covered clinical activities across 15 clinical areas, for which practices are permitted to exception report patients for specified reasons (box). Practices are expected to satisfy certain criteria before exception reporting patients. For exceptions due to informed dissent, practices are required to make personal contact with the patient and to record the patient’s reasons for rejecting the intervention in the notes. Practices are awarded points based on the proportion of appropriate patients (that is, those not exception reported) for whom targets are achieved, between a lower achievement threshold of 40% and an upper threshold that varies by indicator (table 1⇓). The maximum number of points awarded also varies by indicator. In 2008-9 each point earned practices £126, adjusted for the relative prevalence of the disease and the size of the practice population. Practice performance is monitored by primary care trusts, which have the power to audit patient records and to investigate the use and misuse of exception reporting.
We carried out a retrospective study of exception reporting by English general practices in 2008-9, identifying practice and population predictors of exception reporting rates through multilevel multivariate linear regression.
We used data from the Quality Management and Analysis System (QMAS), the national information system supporting the Quality and Outcomes Framework, published by the National Health Service Information Centre. The Quality Management and Analysis System holds publicly available data on the achievement of indicators for 8229 English practices (over 99.9% of all practices). Detailed information on exception reporting (including reason for exception) is not publicly reported and we specifically acquired the relevant dataset for this study. Data on patient and practice characteristics were obtained from the Office for National Statistics website22 and the General Medical Services database. Practices were grouped into fourths on the basis of the local level of area deprivation, using boundary datasets from the UKBORDERS website23 to assign index of deprivation scores to practices at the postcode level.
For each practice and each clinical indicator we calculated the rate of exception reporting as the number of excepted patients divided by the number of eligible patients:
Where Ei is the number of patients exception reported for that indicator and Di is the number of patients meeting the criteria for the indicator and not excepted by the practice. As the distributions of exception reporting rates are skewed, we report medians. We calculated the overall rates for a practice (that is, exception rates across multiple indicators) by summing exceptions for all indicators and dividing by the sum of eligible patients. Patients eligible for multiple indicators are double counted in the overall rates, and these rates therefore represent the proportion of “opportunities” to perform the incentivised activity that resulted in an exception report, rather than the proportion of patients excepted.
We calculated exception reporting rates separately for each of the main reasons: logistical, clinical (contraindication or intolerance), clinical (patient unsuitable), and informed dissent. Under the Quality Management and Analysis System, if the indicator is met the patient is counted in Di and not in Ei, even if the patient was previously exception reported—that is, the patient is not counted as having been exception reported. In addition, the Quality Management and Analysis System only records one exception reporting reason for each patient. For patients who satisfy two or more criteria, the first criterion encountered in the business rules is recorded (see appendix on bmj.com).
We also estimated the average financial gain (Gi) from exception reporting, following a previously published method.16 For each indicator in each practice this is calculated as the difference between the estimated remuneration received and the remuneration that would have been received had the practice not excepted any patients (see appendix on bmj.com).
Because patterns of exception reporting differ according to the type of activity,16 we classified indicators into three categories: measurement (for example, monitoring blood pressure levels), treatment (for example, prescribing β blockers), and outcomes (for example, controlling blood pressure levels). For 25 indicators (see supplementary table on bmj.com), including all treatment indicators, the Quality Management and Analysis System does not differentiate between patient refusal of treatment and clinical contraindications, rendering some exceptions non-ascribable. These indicators were excluded from all analyses relating to reasons for exception reporting. Exception reporting rates were analysed for the remaining 37 indicators (table 1), by category and by reason for exclusion. Sensitivity analyses based on the excluded indicators are given in the appendix on bmj.com.
We used two multilevel mixed effects multiple linear regression models to identify indicator, practice, and population characteristics as predictors of overall and of patient informed dissent exception reporting rates. The structure of the data was three level, with indicators crossed with practices and nested within primary care trusts. To account for this structure and to model variability at each level, we used mixed effects models with the xtmixed command in Stata. Owing to computational limitations, models of indicator variability (both fixed and random effects) failed to converge, hence we used a three level model with indicator observations nested within practices and practices nested within primary care trusts. We assessed collinearity by estimating variance-inflation factors for the independent variables. All factors were in the range 1.0 to 2.1, below the conservative threshold for collinearity. All statistical comparisons were made at an α level of 5%. Stata v11.1 software was used for all analyses.
Rates of exception reporting
In 2008-9 the median exception reporting rate across all 62 clinical indicators was 4.5% (interquartile range 3.4-5.8%). Median rates for individual indicators ranged from 0.0% (for seven indicators) to 24.4% (CHD 10: β blocker therapy for patients with coronary heart disease) (table 1). Rates were generally lower for measurement indicators (median 2.4%) than for treatment and intermediate outcomes indicators (median 10.0% and 5.7%, respectively).
The median exception rate across the 37 indicators for which reasons for exception reporting were ascribable was 2.7% (interquartile range 1.9-3.9%). Figure 1⇓ gives the proportions of patients excepted for each indicator. Logistical exceptions were used most frequently (40.6% of exceptions) and clinical contraindications least frequently (7.6% of exceptions, table 2⇓). Logistical exceptions were particularly common for indicators involving referrals to other agencies or complex assessments for patients with a new diagnosis—for example, diagnosis of atrial fibrillation confirmed by specialist assessment (AF 4), assessment of the severity of depression (DEP 2), and confirmation of stroke by further investigation (STR 13).
Informed dissent accounted for 37.4% of measurement exceptions, 21.9% of intermediate outcomes exceptions, and 30.1% of exceptions overall. The median exception rate for informed dissent across all 37 indicators was 0.44% (interquartile range 0.14-1.1%), but a minority of practices had considerably higher rates: 10% of practices excepted over 2.2% of patients for informed dissent and 1% of practices excepted over 5.7%. Median rates for individual indicators ranged from 0.0% (25 of the 37 indicators) to 1.2% (DM 20: patients with diabetes with HbA1C levels ≤7.5%, table 1).
The estimated median rate of informed dissent across the 25 indicators excluded from the main analysis was between 0.73% (assuming no non-ascribable exceptions were due to informed dissent) and 4.8% (assuming all non-ascribable exceptions were due to informed dissent, see appendix on bmj.com). Based on these assumptions, the overall median rate of informed dissent for all 62 clinical indicators was between 0.53% (interquartile range 0.18-1.3%) and 1.7% (1.2-2.5%).
Factors associated with exception reporting
Table 3⇓ gives the results of the regression analyses for the 37 indicators for which reasons for exception reporting were ascribable. Higher overall exception reporting rates were associated with lower payment thresholds, higher points values (maximum remuneration), and lower numbers of eligible patients. After controlling for other factors, exception reporting rates did not differ significantly between intermediate outcomes and measurement indicators. The generally low rates of exception reporting previously noted for measurement indicators may therefore be attributable to characteristics such as their generally high upper payment thresholds and low points values.
Rates of exception reporting varied with practice characteristics: higher rates were associated with younger age profiles of doctors, failure to secure maximum remuneration on that indicator in the previous year, and a higher number of registered patients. Higher rates were also associated with patient and local area characteristics—for example, higher levels of local area deprivation. Most of these effects were small. The most influential factor was previous performance on the pay for performance scheme: practices that failed to acquire all available points for a specific indicator in the previous year (2007-8) had, on average, exception reporting rates 2.3% higher than those that succeeded. With respect to area deprivation, median exception reporting rates across the 37 indicators varied from 2.6% (interquartile range 1.6-3.7%) for practices in the most affluent fourth to 2.9% (2.0-4.2%) for practices in the most deprived fourth.
Factors associated with higher levels of informed dissent exceptions were broadly comparable with those for overall exceptions, but effect sizes tended to be smaller and were non-significant in some cases. With respect to area deprivation, median exception reporting rates for informed dissent varied from 0.42% (0.15-1.0%) for practices in the most affluent fourth to 0.45% (0.14-1.3%) for practices in the most deprived fourth.
Financial gain associated with exception reporting
By removing patients from the denominator for achievement calculations, exception reporting increased a practice’s reported achievement rate. If a practice’s achievement rate before exception reporting was below the upper payment threshold, then exception reporting would also increase remuneration. Figure 2⇓ shows the remuneration practices received for each of the 62 clinical indicators attributable to achieving the clinical targets and to exception reporting. For most indicators the cost of exception reporting was relatively low, as most practices exceeded the upper payment thresholds even before exception reporting was taken into account.
Indicators ordered by remuneration attributable to exception reporting
Overall, 5.4% of clinical points scored by practices—and therefore remuneration received—was attributable to exception reporting. This equates to about £30 844 500 (€36 877 700; $49 053 200) for all English practices, £3834 for the average practice, and £0.58 per patient. Mean gain ranged from £3586 per practice in the most affluent fourth to £4093 in the most deprived fourth. The cost of exception reporting varied widely by indicator, from £1630 for DM 11 (recording the blood pressure of patients with diabetes) to £4.5m for DEP 2 (assessing the severity of depression). Just two of the 62 indicators, DEP 2 and MH 9 (reviewing physical and social care for people with psychotic illness), accounted for £8.4m—over a quarter of the total cost associated with exception reporting.
For the 37 indicators for which reasons for exception reporting were ascribable, 4.9% of remuneration received was attributable to exception reporting (£19 188 917 for all English practices, £2386 for the average practice, and £0.36 per patient). Out of this total, the gain attributable to informed dissent exceptions was £2 406 500 nationally, £300 per practice and £0.05 per patient. Mean gain ranged from £244 per practice in the most affluent fourth to £351 in the most deprived fourth. The cost of informed dissent exceptions was relatively low because most applied to measurement indicators, which attract less remuneration.
Respecting a patient’s decision to refuse an investigation or treatment, even if the decision is considered wrong or irrational by the attending clinician, is central to medical professionalism24 and a legal requirement in most circumstances. By making incomes partly dependent on patient compliance, pay for performance schemes bring a clinician’s financial self interest into conflict with their duty not to pressure patients to accept medical advice. This can have unintended consequences—for example, under California’s primary care pay for performance scheme, some practices forced the disenrollment of non-compliant patients,13 and in a national survey, US primary care physicians identified performance measures as a driver of more aggressive practice.25 Under the UK Quality and Outcomes Framework, dissenting patients are removed from payment calculations, making the quality targets fairer for practitioners and providing some protection from coercion for patients.21 Our paper suggests that rates of informed dissent in the UK scheme are low, with little variation in rates across the spectrum of deprivation.
Strengths and limitations of the study
This is the first study to examine the reasons used by practices for exception reporting patients under the Quality and Outcomes Framework, and in particular the use of the informed dissent criterion. The study is subject to several limitations. Firstly, practices might not record all expressions of informed dissent for several reasons—for example, if the practice has exceeded the upper payment threshold for the year or if another exception reporting criterion applies for a given patient. In the absence of other permissible reasons for excluding patients, rates of informed dissent would possibly be higher than we report. Secondly, the central Quality Management and Analysis System database only records the first exception reporting reason encountered in the business rule algorithm for each indicator (see appendix on bmj.com). Our figures will therefore underestimate the true extent of informed dissent for indicators where dissent codes appear later in the algorithm. Finally, we were unable to analyse in detail the reasons for exclusion for 25 indicators. Rates of informed dissent may be higher for these indicators (see appendix on bmj.com), but precise figures can only be determined by auditing individual practices.
Patterns of exception reporting
As with previous studies,16 26 we found that overall rates of exception reporting were generally low but varied widely by indicator. Examining the reasons used to exception report patients provides additional insights. For example, the unusually high number of exceptions for certain measurement indicators (such as confirmation of stroke diagnosis) were mainly logistical, pointing to potential problems with access to specialist services and the timescales allocated to these quality indicators. Even greater variation existed between indicators in the costs associated with exception reporting: over £8m (27% of the total cost) was attached to just two indicators (reviewing patients with psychotic illness and assessing the severity of depression). It is notable that the latter indicator proved unworkable for many doctors and was recommended for removal from the scheme by the Quality and Outcomes Framework advisory committee of the National Institute for Health and Clinical Excellence in 2011.27
Lower exception reporting rates were associated with indicators for which upper payment thresholds were high. These thresholds were intended to represent what is maximally attainable by the average practice, and they therefore reflect the anticipated level of difficulty of indicators. Fewer patients were therefore excepted for indicators perceived to be less challenging to achieve—for example, most measurement indicators. Some practice characteristics, such as list size, were also associated with exception reporting rates, despite having no direct bearing on patient eligibility for exception reporting. Fraudulent or inappropriate activity is one possible explanation for these relations, particularly given that practices that fail to achieve maximum remuneration in one year except more patients the following year, a finding noted in previous years of the scheme.19 Although monitoring of exception reporting by primary care trusts may have discouraged gross misuse and driven exception reporting rates down, the frequency and thoroughness of monitoring varies across trusts,28 and low level misuse may have gone undetected. However, it is also possible that propensity to legitimately except patients or thoroughness in documenting exceptions is related to certain practice characteristics. For example, larger practices tend to be better organised29 and may therefore be better at identifying patients who should be excepted. Conversely, practices with larger disease registers for a given list size may have detected more patients with less severe disease, who may be less likely to meet exception reporting criteria. Nevertheless, our findings raise the possibility of gaming, and as financial incentive schemes rely on accurate and honest reporting of performance this issue warrants further and more direct investigation than we have provided in this paper.
Characteristics of patients attending practices were also associated with exception reporting rates. Practices located in more deprived areas tended to except more patients, both overall and for reasons of informed dissent. Given that patients excepted for informed dissent are effectively drawn from a subset of patients not excepted for other reasons, it is possible that the relation between deprivation and informed dissent would be stronger in the absence of other reasons for exception reporting. There has been concern that high levels of exception reporting in practices serving deprived populations may be disguising unmet need.26 Our findings suggest that the differences in exception reporting rates attributable to deprivation, although significant, are small. This could therefore be seen in a more positive light as doctors in more deprived areas, where health literacy is likely to be more limited, seem to be listening to and responding appropriately to patients’ concerns about incentivised procedures. This potential explanation resonates with the views of family doctors,21 but further qualitative work is needed to explore these hypotheses in more detail. In contrast, financial “gains” from exception reporting varied substantially with deprivation. Practices in more deprived areas tended to have lower achievement rates for the clinical indicators and so were more likely to achieve below the upper payment thresholds and therefore to benefit financially from excepting patients.
Conclusions and policy implications
Under the UK Quality and Outcomes Framework variations in exception reporting rates and associated costs are wide, both between practices and across indicators, which require further investigation and careful monitoring. The high exception rates for certain quality indicators also raise questions for policy makers about the appropriateness of these indicators and whether the current exception reporting system is being used appropriately to deal with the potential unintended consequences of financial incentives.
Exception reporting does, however, provide some protection from inappropriate and coercive treatment of patients whose providers are subject to financial incentives. Relatively few patients are excepted for reasons of informed dissent in the United Kingdom, which suggests that activities incentivised in the scheme are broadly acceptable to patients (although this does not imply approval of the incentive scheme itself). It could be argued that dissent is so uncommon that a specific exception reporting provision is not required, as long as upper thresholds are set below 100%. However, removing the provision might subject a substantial minority of patients attending practices performing below the upper payment thresholds to coercion. Thousands of patients expressed their wish not to receive interventions under the framework (for example, over 100 000 patients with asthma actively declined a review). At relatively low cost, the provision to exception report enables patients’ voices to be heard and counters some of the critiques of the scheme as endangering the doctor-patient relationship.30
What is already known on this topic
The provision to exempt dissenting patients from pay for performance schemes removes the financial conflict of interest for doctors in respecting a patient’s choice to refuse an incentivised intervention
Exception reporting rates have generally been low in the United Kingdom, with little evidence of widespread, large scale fraud
Concerns over inappropriate use of exception reporting, however, persist in the absence of data on the exact reasons why practices except patients
What this study adds
In the UK pay for performance scheme the most common reasons for excepting patients were logistical (40.7% of exceptions), contraindication to treatment (18.7%), and patient informed dissent (30.1%)
Higher rates of informed dissent within practices were associated with: higher numbers of registered patients, higher levels of local area deprivation, and failure of the practice to secure maximum remuneration in the previous year
For most of the quality indicators, less than 2% of patients were excepted for reasons of informed dissent, suggesting that the incentivised activities were broadly acceptable to patients
Cite this as: BMJ 2012;344:e2405
Contributors: TD and EK participated in the planning of the study, analysis and interpretation of data, drafting and editing the text, and had full access to all of the data in the study. TD takes responsibility for the integrity of the data and the accuracy of the data analysis. He had final responsibility for the decision to submit for publication. CF, HL, JMV, and SC participated in the planning of the study, analysis and interpretation of data, and editing the final text. SC had full access to all the data in the study. All authors have seen and approved the final version of the manuscript.
Funding: This study received no direct source of funding.
Competing interests: All authors have completed the ICMJE uniform disclosure form at www.icmje.org/coi_disclosure.pdf (available on request from the corresponding author) and declare: no support from any organisation for the submitted work; no financial relationships with any organisations that might have an interest in the submitted work in the previous three years; and no other relationships or activities that could appear to have influenced the submitted work.
Ethical approval: Not required.
Data sharing: No additional data available.
This is an open-access article distributed under the terms of the Creative Commons Attribution Non-commercial License, which permits use, distribution, and reproduction in any medium, provided the original work is properly cited, the use is non commercial and is otherwise in compliance with the license. See: http://creativecommons.org/licenses/by-nc/2.0/ and http://creativecommons.org/licenses/by-nc/2.0/legalcode.