Introduction

Top Abstract Introduction Method Results Discussion Appendix References

There is an increasing emphasis on measurement of performance in the NHS. In setting out its future policy on the NHS in the white paper The New NHS the government emphasised the need for a new performance framework to measure progress towards its objectives.¹ A subsequent consultation paper proposed performance indicators for the comparison of the quality of health care provided in health authorities.²

We analysed admission rates for asthma, epilepsy, and diabetes, which are three of the indicators proposed for assessment of performance in primary care. Admission rates for chronic conditions have been used in other countries, principally the United States, as measures of access to primary care.^3-10 The conditions chosen are those for which timely and effective primary care could be expected to reduce the risk of admission to hospital by preventing the onset of illness, controlling an acute episode of illness, or better long term management.

Previous research in the United Kingdom suggests that some characteristics of primary care that might be taken to reflect quality of practice are related to admission rates for chronic diseases. For example, lower admission rates for asthma have been found in practices whose prescribing patterns suggested better preventive care,¹¹ and lower admission rates for diabetes were found in practices with better organised diabetic care.¹² Griffiths et al found that higher admission rates for asthma in east London were found in small practices, in which, the authors said, it might have been more difficult to develop systems for identifying, reviewing, and educating patients.¹³

The interpretation of admissions for the chronic conditions used in the proposed indicator is not straightforward because of potentially confounding factors such as the socioeconomic characteristics of the population. ^{7-10 14} Admission rates might also be affected by hospital policies. Durojaiye et al found admission rates for asthma in Nottingham increased sharply between 1975 and 1985, a period of time when there seemed to be improvements in primary care; this was attributed in part to changing admission policies.¹⁵ Casanova and Starfield, however, found that admission rates for children in Spain were not correlated with supply side or socioeconomic factors.¹⁶ More generally, Baker and Klein found that socioeconomic conditions explained considerable proportions of variation among family health services authorities in primary care outcomes such as cervical cytology rates.¹⁷

In the absence of direct measures of incidence and prevalence of disease, crude admission rates (table 1) can be used as a measure of the absolute magnitude of a health problem. Such rates should not, however, be used to monitor the performance of a geographical area when they are affected by factors which are outside its control and vary across areas, so that areas are affected differentially. Performance indicators should be measures of what the relevant decision makers can reasonably be held to account for.

We have used the example of the admission rates for the three chronic diseases proposed as primary care performance indicators in the United Kingdom to illustrate the difference between health outcomes and performance indicators. We examine the extent to which demographic composition, socioeconomic factors, measures of population health, and secondary care characteristics influence admission rates and thus cloud any relations between admission rates and the quality of primary care.

	Method

Top Abstract Introduction Method Results Discussion Appendix References

Data on hospital admission rates were obtained from the hospital episodes statistics unit at the Department of Health for each of the financial years 1989-90 to 1994-5. The data covered the 367 English local authorities and were aggregated to the level of family health services authority. The admission rate for an area was derived from the number of admissions (including readmissions) or day case episodes of residents, with asthma (ICD-10 (international classification of diseases, 10th revision) codes J45-J46), diabetes mellitus (ICD-10 codes E10-E14) and epilepsy (ICD-10 codes G40-G41) as the primary diagnosis. Rates of hospital admissions were measured per 10 000 residents. We used data from 1989-90 to 1994-5 because we also wanted to examine the magnitude of year to year variation on a consistent geographical basis before the introduction of new unified health authorities.

We used multiple regression analyses to determine how much of the variation in age and sex standardised admission rates for asthma, diabetes, and epilepsy could be explained by the socioeconomic characteristics of the areas, population health, and provision of secondary care.

Socioeconomic variables were derived from the 1991 census and included housing conditions, social class, unemployment, and, as an indicator of wealth, car ownership. We included two measures of population health from the 1991 census: the proportion of the working age population who reported being permanently sick and who reported limiting long term illness.

The variables used to estimate the supply of secondary care services were the number of hospital medical staff in general medicine per 10 000 population and a variable that reflect the number of beds per head of population weighted for distance from the hospital.¹⁸ Because of the large number of potential explanatory variables we selected the final set by stepwise regression, retaining only those variables that showed significant partial correlation with the admission rate. For example, we initially included the standardised mortality ratio as a health measure but found that it had no additional explanatory power when it was added to regressions already containing the other health measures. The variables used in the final regression analysis are shown in the table in the Appendix .

We calculated three predicted admission rates for each condition for each geographical area using the coefficients from the regression. The first was the rate predicted by using only the health variables. The second predicted rate used the health variables and the socioeconomic variables; and the third predicted rate used all the variables in the final set: health, socioeconomic factors, and supply of secondary care. The differences between the actual rate for an area and the predicted rates are measures of the possible effect of quality of primary care on admissions after allowance for possible confounding by health, socioeconomic characteristics, and supply of secondary care.

	Results

Top Abstract Introduction Method Results Discussion Appendix References

Table 1 shows the clear variation across the 90 health authority areas in crude admission rates for the three conditions. Although there was stability in average rates between years, there was considerable instability between the ranked position of individual health authorities from year to year. For example, when health authorities were ranked in order of admission rate, the ranking of an individual health authority in 1994-5 compared with 1993-4 changed by 10 or more places in 46 (51%) areas for asthma, 28 (31%) areas for diabetes, and 36 (40%) areas for epilepsy. We found that combining these three rates, as proposed in the NHS executive's consultation document, did not produce rankings that were notably more stable, with 28 (31%) authorities still changing rank order by more than 10 places and 10 (11%) by over 20 places.

The regression analyses show that a high proportion of the variance in age and sex standardised admission rates can be explained by socioeconomic and secondary care factors (table 2). Overall, these variable explained 45% of the variance in admission rates for asthma, 33% for diabetes, and 55% for epilepsy.

Table 3 uses the example of admission rates for asthma to show how the ranking of health authorities is affected when they are ranked by crude rates (column 1), by rates adjusted for age and sex (column 2), by rates adjusted for age, sex, socioeconomic factors, and limiting long term illness reported in the census (column 3), and finally by rates which are also adjusted for factors related to supply of secondary care (column 4). It is conventional to standardise for age and sex, but inspection of the table shows that other potentially confounding factors have at least as great an impact on the rankings. Eight out of the 10 areas with the highest crude admission rates are in the top 10 ranked by age and sex standardised rates, whereas only four out of the top 10 by age and sex standardised rates remain in the top 10 when allowance is made for morbidity, socioeconomic factors, and secondary care factors.

	Discussion

Top Abstract Introduction Method Results Discussion Appendix References

Our results highlight the fundamental difference between performance indicators and health outcomes. A simple count of adverse health eventssuch as deaths, admissions, or disabilityis a measure of the burden of a health problem in a populationthat is, the health outcome. However, a performance indicator designed to improve that outcome should relate only to those factors that are under the control of the staff to whom it is being applied.

Admission rates for conditions for which admission could be avoided by good primary care have been widely used in the United States. Their main application has been as a measure of access to primary care rather than of the quality of primary care. Nevertheless, there are data which suggest that good primary care should help to avoid admission for these conditions. What we have shown is that these admissions are substantially influenced by factors outside the control of the primary care teamthat is, the characteristics of their population and the supply of secondary care resources. The admission rates should be adjusted for these factors before being used as measures of the quality of primary care. Even so, we do not know whether there are other important factors that we have not been able to include in our analyses. For example, there are no data available that would enable allowance to be made for the prevalence of these conditions in individual health authority areas or the admission policies of individual hospitals.

We used a particular type of performance indicator, but the lesson is more general and would apply to other suggested indicators.²¹ It is essential to test for confounding of indicators by factors outside the control of the decision makers whose performance is being monitored. Our results point to other potential problems of using admission rates as indicators of quality of care. The rates fluctuate greatly from year to year, showing the statistical instability of any relatively rare event.²² While this difficulty could be reduced by using a 3 year moving average for a health authority, it means that the indicators would be even more difficult to apply to individual practices, where greater year to year variation would be expected to occur because of the smaller population size. Marshall and Spiegelhalter have emphasised the importance of accompanying any performance indicators with measures of sampling variability.²³

Any single performance indicator may be a misleading guide to the overall performance of an organisation as it covers only one dimension of that performance, and concentration on one aspect of care may produce perverse incentives to ignore other aspects of performance. If performance indicators are to be used, it is important that they cover the full range of outputs and inputs for the sector in question. While health outcome should be related to crude rates of adverse events in the population, performance indicators should relate only to those aspects of care that can be altered by the staff whose performance is being measured.

	Acknowledgments

   Contributors: AG, HG, and MR developed the study and wrote the paper jointly, with AG contributing particularly to the econometrics, HG to the modelling, and MR to the clinical input. The paper is guaranteed by all authors. Discussions with Dr Tom Ricketts at the Sheps Center, University of North Carolina, gave us the original idea for carrying out these analyses.

	Footnotes

Funding: NPCRDC is funded by the Department of Health. The views expressed in this paper are those of the authors and not necessarily those of the Department of Health.

Competing interests: None declared.

	Appendix

Top Abstract Introduction Method Results Discussion Appendix References

	References

Top Abstract Introduction Method Results Discussion Appendix References

(Accepted 26 March 1999)