Measuring quality of care with routine data: avoiding confusion between performance indicators and health outcomesBMJ 1999; 319 doi: http://dx.doi.org/10.1136/bmj.319.7202.94 (Published 10 July 1999) Cite this as: BMJ 1999;319:94
- Antonio Giuffrida, research fellowa,
- Hugh Gravelle, professor of economicsa,
- Martin Roland, professor of general practice ()b
- a National Primary Care Research and Development Centre, Centre for Health Economics, University of York, Heslington, York YO10 5DD
- b National Primary Care Research and Development Centre, Williamson Building, University of Manchester, Oxford Road, Manchester M13 9PL
- Correspondence to: M Roland
- Accepted 26 March 1999
Objective: To investigate the impact of factors outside the control of primary care on performance indicators proposed as measures of the quality of primary care.
Design: Multiple regression analysis relating admission rates standardised for age and sex for asthma, diabetes, and epilepsy to socioeconomic population characteristics and to the supply of secondary care resources.
Setting: 90 family health services authorities in England, 1989-90 to 1994-5.
Results: At health authority level socioeconomic characteristics, health status, and secondary care supply factors explained 45% of the variation in admission rates for asthma, 33% for diabetes, and 55% for epilepsy. When health authorities were ranked, only four of the 10 with the highest age-sex standardised admission rates for asthma in 1994-5 remained in the top 10 when allowance was made for socioeconomic characteristics, health status, and secondary care supply factors. There was also substantial year to year variation in the rates.
Conclusion: Health outcomes should relate to crude rates of adverse events in the population. These give the best indication of the size of a health problem. Performance indicators, however, should relate to those aspects of care which can be altered by the staff whose performance is being measured.
The NHS executive has proposed that admission rates for asthma, diabetes, and epilepsy could be used at health authority level as indicators of the quality of primary care
There is considerable year to year variation in the ranking of health authorities by admission rates for these conditions, even when rates are aggregated. This makes it hard to interpret a single year's data: a 3 year average would be more reliable
Morbidity, socioeconomic characteristics, and secondary care supply are important confounding factors that explain between a third and a half of the variation in admission rates across health authority areas
Performance indicators should relate to aspects of care that can be controlled by decision makers. Confounding factors have a clear impact on admission rates and must therefore be taken into account if such rates are to be used as performance indicators
Editorial by Jankowski
There is an increasing emphasis on measurement of performance in the NHS. In setting out its future policy on the NHS in the white paper The New NHS the government emphasised the need for a new performance framework to measure progress towards its objectives.1 A subsequent consultation paper proposed performance indicators for the comparison of the quality of health care provided in health authorities.2
We analysed admission rates for asthma, epilepsy, and diabetes, which are three of the indicators proposed for assessment of performance in primary care. Admission rates for chronic conditions have been used in other countries, principally the United States, as measures of access to primary care.3–10 The conditions chosen are those for which timely and effective primary care could be expected to reduce the risk of admission to hospital by preventing the onset of illness, controlling an acute episode of illness, or better long term management.
Previous research in the United Kingdom suggests that some characteristics of primary care that might be taken to reflect quality of practice are related to admission rates for chronic diseases. For example, lower admission rates for asthma have been found in practices whose prescribing patterns suggested better preventive care,11 and lower admission rates for diabetes were found in practices with better organised diabetic care.12 Griffiths et al found that higher admission rates for asthma in east London were found in small practices, in which, the authors said, it might have been more difficult to develop systems for identifying, reviewing, and educating patients.13
The interpretation of admissions for the chronic conditions used in the proposed indicator is not straightforward because of potentially confounding factors such as the socioeconomic characteristics of the population.7–10 14 Admission rates might also be affected by hospital policies. Durojaiye et al found admission rates for asthma in Nottingham increased sharply between 1975 and 1985, a period of time when there seemed to be improvements in primary care; this was attributed in part to changing admission policies.15 Casanova and Starfield, however, found that admission rates for children in Spain were not correlated with supply side or socioeconomic factors.16 More generally, Baker and Klein found that socioeconomic conditions explained considerable proportions of variation among family health services authorities in primary care outcomes such as cervical cytology rates.17
In the absence of direct measures of incidence and prevalence of disease, crude admission rates (table 1) can be used as a measure of the absolute magnitude of a health problem. Such rates should not, however, be used to monitor the performance of a geographical area when they are affected by factors which are outside its control and vary across areas, so that areas are affected differentially. Performance indicators should be measures of what the relevant decision makers can reasonably be held to account for.
We have used the example of the admission rates for the three chronic diseases proposed as primary care performance indicators in the United Kingdom to illustrate the difference between health outcomes and performance indicators. We examine the extent to which demographic composition, socioeconomic factors, measures of population health, and secondary care characteristics influence admission rates and thus cloud any relations betweenadmission rates and the quality of primary care.
Data on hospital admission rates were obtained from the hospital episodes statistics unit at the Department of Health for each of the financial years 1989-90 to 1994-5. The data covered the 367 English local authorities and were aggregated to the level of family health services authority. The admission rate for an area was derived from the number of admissions (including readmissions) or day case episodes of residents, with asthma (ICD-10 (international classification of diseases, 10th revision) codes J45-J46), diabetes mellitus (ICD-10 codes E10-E14) and epilepsy (ICD-10 codes G40-G41) as the primary diagnosis. Rates of hospital admissions were measured per 10 000 residents. We used data from 1989-90 to 1994-5 because we also wanted to examine the magnitude of year to year variation on a consistent geographical basis before the introduction of new unified health authorities.
We used multiple regression analyses to determine how much of the variation in age and sex standardised admission rates for asthma, diabetes, and epilepsy could be explained by the socioeconomic characteristics of the areas, population health, and provision of secondary care.
Socioeconomic variables were derived from the 1991 census and included housing conditions, social class, unemployment, and, as an indicator of wealth, car ownership. We included two measures of population health from the 1991 census: the proportion of the working age population who reported being permanently sick and who reported limiting long term illness.
The variables used to estimate the supply of secondary care services were the number of hospital medical staff in general medicine per 10 000 population and a variable that reflect the number of beds per head of population weighted for distance from the hospital.18 Because of the large number of potential explanatory variables we selected the final set by stepwise regression, retaining only those variables that showed significant partial correlation with the admission rate. For example, we initially included the standardised mortality ratio as a health measure but found that it had no additional explanatory power when it was added to regressions already containing the other health measures. The variables used in the final regression analysis are shown in the table in the Appendix.
We calculated three predicted admission rates for each condition for each geographical area using the coefficients from the regression. The first was the rate predicted by using only the health variables. The second predicted rate used the health variables and the socioeconomic variables; and the third predicted rate used all the variables in the final set: health, socioeconomic factors, and supply of secondary care. The differences between the actual rate for an area and the predicted rates are measures of the possible effect of quality of primary care on admissions after allowance for possible confounding by health, socioeconomic characteristics, and supply of secondary care.
Table 1 shows the clear variation across the 90 health authority areas in crude admission rates for the three conditions. Although there was stability in average rates between years, there was considerable instability between the ranked position of individual health authorities from year to year. For example, when health authorities were ranked in order of admission rate, the ranking of an individual health authority in 1994-5 compared with 1993-4 changed by 10 or more places in 46 (51%) areas for asthma, 28 (31%) areas for diabetes, and 36 (40%) areas for epilepsy. We found that combining these three rates, as proposed in the NHS executive's consultation document, did not produce rankings that were notably more stable, with 28 (31%) authorities still changing rank order by more than 10 places and 10 (11%) by over 20 places.
The regression analyses show that a high proportion of the variance in age and sex standardised admission rates can be explained by socioeconomic and secondary care factors (table 2). Overall, these variable explained 45% of the variance in admission rates for asthma, 33% for diabetes, and 55% for epilepsy.
Table 3 uses the example of admission rates for asthma to show how the ranking of health authorities is affected when they are ranked by crude rates (column 1), by rates adjusted for age and sex (column 2), by rates adjusted for age, sex, socioeconomic factors, and limiting long term illness reported in the census (column 3), and finally by rates which are also adjusted for factors related to supply of secondary care (column 4). It is conventional to standardise for age and sex, but inspection of the table shows that other potentially confounding factors have at least as great an impact on the rankings. Eight out of the 10 areas with the highest crude admission rates are in the top 10 ranked by age and sex standardised rates, whereas only four out of the top 10 by age and sex standardised rates remain in the top 10 when allowance is made for morbidity, socioeconomic factors, and secondary care factors.
Our results highlight the fundamental difference between performance indicators and health outcomes. A simple count of adverse health events—such as deaths, admissions, or disability—is a measure of the burden of a health problem in a population—that is, the health outcome. However, a performance indicator designed to improve that outcome should relate only to those factors that are under the control of the staff to whom it is being applied.
Admission rates for conditions for which admission could be avoided by good primary care have been widely used in the United States. Their main application has been as a measure of access to primary care rather than of the quality of primary care. Nevertheless, there are data which suggest that good primary care should help to avoid admission for these conditions. What we have shown is that these admissions are substantially influenced by factors outside the control of the primary care team—that is, the characteristics of their population and the supply of secondary care resources. The admission rates should be adjusted for these factors before being used as measures of the quality of primary care. Even so, we do not know whether there are other important factors that we have not been able to include in our analyses. For example, there are no data available that would enable allowance to be made for the prevalence of these conditions in individual health authority areas or the admission policies of individual hospitals.
We used a particular type of performance indicator, but the lesson is more general and would apply to other suggested indicators.21 It is essential to test for confounding of indicators by factors outside the control of the decision makers whose performance is being monitored. Our results point to other potential problems of using admission rates as indicators of quality of care. The rates fluctuate greatly from year to year, showing the statistical instability of any relatively rare event.22 While this difficulty could be reduced by using a 3 year moving average for a health authority, it means that the indicators would be even more difficult to apply to individual practices, where greater year to year variation would be expected to occur because of the smaller population size. Marshall and Spiegelhalter have emphasised the importance of accompanying any performance indicators with measures of sampling variability.23
Any single performance indicator may be a misleading guide to the overall performance of an organisation as it covers only one dimension of that performance, and concentration on one aspect of care may produce perverse incentives to ignore other aspects of performance. If performance indicators are to be used, it is important that they cover the full range of outputs and inputs for the sector in question. While health outcome should be related to crude rates of adverse events in the population, performance indicators should relate only to those aspects of care that can be altered by the staff whose performance is being measured.
Contributors: AG, HG, and MR developed the study and wrote the paper jointly, with AG contributing particularly to the econometrics, HG to the modelling, and MR to the clinical input. The paper is guaranteed by all authors. Discussions with Dr Tom Ricketts at the Sheps Center, University of North Carolina, gave us the original idea for carrying out these analyses.
Funding NPCRDC is funded by the Department of Health. The views expressed in this paper are those of the authors and not necessarily those of the Department of Health.
Competing interests None declared.