Papers

Effect of correcting outcome data for case mix: an example from stroke medicine

^a University of Edinburgh, Department of Clinical Neurosciences, Western General Hospital, Edinburgh EH4 2XU

Correspondence to: Dr Davenport.

Abstract

Objective: To show the influence of variations in case mix on clinical outcome indicators for patients admitted to hospital with acute stroke.
Design: "Before and after" cohort study, with prospective, consecutive identification of patients and prospective follow up; multiple logistic regression analyses to correct for case mix variations.
Setting: University teaching hospital.
Subjects: 216 patients with stroke identified before the introduction of an organised stroke service, and 252 patients with stroke identified after its introduction.
Main outcome measures: Case fatality at 30 days and 12 months; for survivors at 12 months, proportions of patients who were independent (according to the Oxford handicap scale) and of those living at home.
Results: Crude outcome data suggested that patients in the cohort identified after the introduction of the stroke service were significantly more likely to be alive, independent, and living at home than patients managed before the stroke service. After adjustment for age and sex these "improvements" were less impressive but still significant. After adjustment for many other possible prognostic indicators, however, the differences between the two groups for all four outcomes were non-significant, suggesting that the "improvements" may have been entirely due to differences in case mix between the two cohorts, rather than the new stroke service.
Conclusions: Variations in case mix have a crucial influence on the interpretation of outcome data, and this is particularly important in non-randomised comparative studies. Such studies, comparing performance within and between different provider units, are likely to become increasingly common in the new reformed NHS. To allow meaningful interpretation, these studies must try to correct for case mix.

Introduction

The confounding influence that variations in case mix may exert on clinical outcome is well recognised.^{1 2 3} Despite this, the government has published for Scottish hospitals clinical outcome indicators for several diseases--including stroke--that have only been corrected for age and sex.⁴ These data have been widely interpreted by the media (and even by some doctors and managers) as league tables of hospital performance. To assess the impact of case mix on the outcome after acute stroke, we compared the outcome of patients managed before and after the introduction of an organised stroke service in our hospital. This service included an inpatient stroke unit, an intervention that has been shown in a systematic review of randomised controlled trials to reduce case fatality and dependence.⁵

Methods

We prospectively identified consecutive stroke patients who needed inpatient management in our hospital (excluding those with subarachnoid haemorrhage and those admitted to the regional neurosciences unit) by daily consultation with the admitting medical and nursing teams and perusal of admissions records. All patients were assessed by a stroke physician who completed a standard data form; the data were then transferred to an electronic database (stroke register). We followed up survivors either with a face to face or telephone interview or with a postal questionnaire. We measured four important outcomes: case fatality at 30 days and 12 months; the proportion of survivors living at home at 12 months after their stroke; and the proportion of survivors who were independent (<3 on the Oxford handicap scale⁶) at 12 months. We calculated uncorrected odds ratios and 95% confidence intervals for these outcomes. We then performed multiple logistic regression, forcing in 19 different indicators of case mix (box); we chose these variables on the basis of likely clinical importance and from our previous experience with prognostic models derived from large community and hospital based stroke registers (C Counsell, personal communication). This provided corrected odds ratios for the outcomes. We also corrected the data for age and sex alone.

Results

We identified 216 strokes before and 252 strokes after the introduction of the stroke unit, over a period of 27 months. Three patients were lost to follow up at 30 days and four at 12 months; all were overseas visitors. Independence data for six survivors were unavailable (two in the cohort managed before the introduction of the stroke unit and four in the in the cohort managed after). Figure 1 shows the uncorrected odds ratios; patients admitted after the introduction of the stroke unit had very significantly better outcomes--for example, a 50% increase in the odds of living at home at 12 months. Correcting for age and sex alone reduced these estimates of treatment effect, but they remained significant. When the data were corrected for case mix, however, all the estimates became non-significant, with odds ratios approaching unity.

View larger version (26K): [in this window] [in a new window]   Fig 1--Odds ratios (95% confidence intervals) comparing standard ward care with stroke service care; data for four outcomes before and after correction for age and sex only, and for 19 indicators of case mix

Discussion

Because our study was a non-randomised, "before and after" comparison, we were aware that variations in case mix might significantly bias the outcomes. We therefore corrected the crude outcomes using multiple logistic regression analysis, which is a method of simultaneously adjusting for the effects of several different variables. Before discussing the implications of our results, we consider some of the problems of using these complicated mathematical models.⁸

Firstly, such models may "overcorrect" the data--that is, they may obscure a real but moderate treatment effect by including too many variables. Wasson et al suggested that for every variable there should be at least five patients with the outcome of particular interest.⁹ In our models we used 19 variables, and there were considerably more than 95 outcome events for each outcome we measured.

 Variables forced into multiple logistic regression modelling
 (measured at time of stroke)

 Variable                            Definition
 Age (years)                         Not applicable
 Able to lift both arms above        Not applicable
  gravity
 Diabetes mellitus                   Previous history recorded
 Employed                            Employed at time of stroke
 Assessed for stroke register        Not applicable
  within 48 hours of onset
 Normal eye opening                  Glasgow coma eye score 4
 Normal motor score                  Glasgow coma motor score 6
 Normal verbal score                 Glasgow coma verbal score 5
 Haemorrhagic stroke                 Haemorrhage on computed tomography,
                                       magnetic resonance imaging, or postmortem
 Prestroke independence              Oxford handicap scale <3
 High systolic blood pressure        > 159 mm Hg recorded since onset of stroke
 Low systolic blood pressure         < 121 mm Hg recorded since onset of stroke
 Living alone                        Not applicable
 Cohort                              Before or after stroke service was available
 Previous myocardial infarction      History of previous myocardial infarction from
                                      patient, general practitioner, or hospital notes
 Sex                                 Not applicable
 Total anterior circulation infarct  According to Oxfordshire Community Stroke
  (assumed or definite)              Project classification7
 Unable to walk                      Not applicable
 Urinary incontinence within         More than two episodes of incontinence or
  7 days of onset                     catheterised since onset of stroke service
                                      (recorded in nursing or hospital notes)

Secondly, models derived from one dataset may not perform well on another, independent dataset. In our department we have developed prognostic models for a variety of outcomes, including the four in the present study, which rely on five or six variables (C Counsell and M McDowall, personal communication). These models were derived from a community based stroke register (the Oxfordshire Community Stroke Project¹⁰) and have been validated on two further independent datasets (one community and one hospital based register). When we applied these validated models to the results of the current study we obtained almost identical corrected odds ratios as those in the figure, which was reassuring.

IMPLICATIONS OF FINDINGS

Our results confirm the dangers of comparing outcome measures that have not been adjusted for case mix; it was case mix, rather than a beneficial effect of our stroke unit, that seemed to be responsible for much of the observed "improvements" in outcome (although because of the imprecision of the corrected data, the results are still consistent with a moderate but non-significant beneficial effect). Several factors may have explained why case mix changed, but one factor was the closure of our hospital's casualty department during the study. Similar findings have been shown for other diseases.^{11 12 13} While epidemiologists and clinical trialists have been aware of this problem for some time, its importance in everyday clinical practice is perhaps less readily acknowledged.

Although randomised controlled trials remain the gold standard for assessing interventions, we believe that there are two areas where non-randomised comparative studies are likely to become increasingly common. Firstly, individual provider units are increasingly aware of the need to provide evidence of the quality of care that they deliver; because randomised controlled trials within single units are impractical and sometimes unethical, often the most feasible design for these studies is a "before and after" one. These non-randomised studies are susceptible to several biases, with the confounding effect of case mix variation one of the most potent; therefore, if providers wish to draw valid comparisons between outcomes, they must collect potentially important case mix variables as well as outcome measures. Secondly, the new NHS reforms encourage competition between provider units, with purchasers empowered to select the best treatment packages for their patients. Currently, little information is available to purchasers on which to base such decisions, as there are few comparative data between different hospitals for similar conditions.

Recently, the government published limited outcome indicators for several diseases, comparing the performance of hospitals in Scotland.⁴ In the foreword to this document the problems of case mix were acknowledged, and it was stated that the purpose of publication was not to provide an indication of the best and worst hospitals for general practitioners and patients but to improve overall standards of care. It was added that trusts that underperformed should immediately review their treatments in the relevant areas, and health boards and purchasers should be encouraged to question the quality of care delivered in these centres; yet these suggestions may be entirely unjustified as the data took no account of case mix. We have clearly shown that if the government and other agencies wish to provide meaningful comparative outcome data they must address the question of case mix before publishing any more potentially unreliable figures.

Addendum

After we submitted this paper for publication the government published an updated version of their outcome indicators.¹⁴ In addition to correction for age and sex, these data were also corrected for social deprivation with the Carstairs index,¹⁵ based on postcode sectors of residence according to the 1991 census. Data were also corrected for pre-existing morbidity, which was based on the principal Scottish Morbidity Record 1 (SMR1) diagnoses for the preceding five years, not on secondary diagnoses recorded at the time of discharge after stroke--that is, there was no correction for other conditions that had not been noted in previous hospital admissions. It was acknowledged that this system represented an extremely crude and only partial adjustment. Moreover, there was still no correction for crucially important case mix variables for stroke prognosis, such as level of consciousness on admission.

We are grateful to Mr Jim Slattery for advice regarding the statistical methods used in this paper.

Funding: RJD is funded by the Medical Research Council (UK); MSD and the stroke register were funded by the Stroke Association (UK).

Conflict of interest: None.

1. Orchard C. Comparing health care outcomes. BMJ 1994;308:1493-6. [Free Full Text]
2. Green J, Wintfeld N, Sharkey P, Passman LJ. The importance of severity of illness in assessing hospital mortality. JAMA 1990;263:241-6. [Abstract/Free Full Text]
3. Bion J. Outcomes in intensive care. BMJ 1993;307:953-4.
4. Scottish Office. Clinical outcome indicators. Edinburgh: HMSO, 1994.
5. Stroke Unit Trialists' Collaboration. A systematic review of specialist multidisciplinary team (stroke unit) care for stroke inpatients. In: Warlow C, Van Gijn J, Sandercock P, eds. Stroke module of the Cochrane database of systematic reviews. London: BMJ Publishing, 1995.
6. Bamford JM, Sandercock PAG, Warlow CP, Slattery J. Interobserver agreement for the assessment of handicap in stroke patients. Stroke 1989;20:828. [Medline]
7. Bamford J, Sandercock P, Dennis M, Burn J, Warlow C. Classification and natural history of clinically identifiable subtypes of cerebral infarction. Lancet 1991;337:1521-6. [Medline]
8. Wyatt JC, Altman DG. Prognostic models: clinically useful or quickly forgotten? BMJ 1995;311:1539-41.
9. Wasson JH, Sox HC, Neff RK, Goldman L. Clinical prediction rules. Applications and methodological standards. N Engl J Med 1985;313:793-9. [Abstract]
10. Bamford J, Sandercock P, Dennis M, Warlow C, Jones L, McPherson K, et al. A prospective study of acute cerebrovascular disease in the community: the Oxfordshire Community Stroke Project 1981-86. 1. Methodology, demography and incident cases of first-ever stroke. J Neurol Neurosurg Psychiatry 1988;51:1373-80. [Abstract/Free Full Text]
11. Rockall TA, Logan RFA, Devlin HB, Northfield TC, for the National Audit of Acute Upper Gastrointestinal Haemorrhage. Variation in outcome after acute upper gastrointestinal haemorrhage. Lancet 1995;346:346-50. [Medline]
12. Rowan KM, Kerr JH, Major E, McPherson K, Short A, Vessey MP. Intensive Care Society's APACHE II study in Britain and Ireland. I: Variations in case mix of adult admissions to general intensive care units and impact on outcome. BMJ 1993;307:972-7.
13. Keeler EB, Kahn KL, Draper D, Sherwood MJ, Rubenstein LV, Reinisch EJ, et al. Changes in sickness at admission following the introduction of the prospective payment system. JAMA 1990;264:1962-8. [Abstract/Free Full Text]
14. Scottish Office. Clinical outcome indicators. Edinburgh: HMSO, 1995.
15. Carstairs V, Morris R. Deprivation and health. Aberdeen: Aberdeen University Press, 1991.

CiteULike    Complore    Connotea    Del.icio.us    Digg    Reddit    StumbleUpon    Technorati    What's this?

This article has been cited by other articles:

• Zhu, H. F., Newcommon, N. N., Cooper, M. E., Green, T. L., Seal, B., Klein, G., Weir, N. U., Coutts, S. B., Watson, T., Barber, P. A., Demchuk, A. M., Hill, M. D., for the Calgary Stroke Program, (2009). Impact of a Stroke Unit on Length of Hospital Stay and In-Hospital Case Fatality. Stroke 40: 18-23 [Abstract] [Full text]  
• De Wit, L., Putman, K., Schuback, B., Komarek, A., Angst, F., Baert, I., Berman, P., Bogaerts, K., Brinkmann, N., Connell, L., Dejaeger, E., Feys, H., Jenni, W., Kaske, C., Lesaffre, E., Leys, M., Lincoln, N., Louckx, F., Schupp, W., Smith, B., De Weerdt, W. (2007). Motor and Functional Recovery After Stroke: A Comparison of 4 European Rehabilitation Centers. Stroke 38: 2101-2107 [Abstract] [Full text]  
• Seenan, P., Long, M., Langhorne, P. (2007). Stroke Units in Their Natural Habitat: Systematic Review of Observational Studies. Stroke 38: 1886-1892 [Abstract] [Full text]  
• Gray, L J, Sprigg, N, Bath, P M W, Sorensen, P, Lindenstrom, E, Boysen, G, De Deyn, P P, Friis, P, Leys, D, Marttila, R, Olsson, J-E, O'Neill, D, Ringelstein, B, van der Sande, J-J, Turpie, A G G, for the TAIST Investigators, (2006). Significant variation in mortality and functional outcome after acute ischaemic stroke between western countries: data from the tinzaparin in acute ischaemic stroke trial (TAIST). J. Neurol. Neurosurg. Psychiatry 77: 327-333 [Abstract] [Full text]  
• van Exel, N.J.A., Koopmanschap, M.A., Scholte op Reimer, W., Niessen, L.W., Huijsman, R. (2005). Cost-effectiveness of integrated stroke services. QJM 98: 415-425 [Abstract] [Full text]  
• Walsh, K, Gompertz, P H, Rudd, A G (2002). Stroke care: how do we measure quality?. Postgrad. Med. J. 78: 322-326 [Abstract] [Full text]  
• Wolfe, C., Rudd, A., Dennis, M., Warlow, C., Langhorne, P. (2001). Taking acute stroke care seriously. BMJ 323: 5-6 [Full text]  
• Grieve, R., Hutton, J., Bhalla, A., Rastenyte, D., Ryglewicz, D., Sarti, C., Lamassa, M., Giroud, M., Dundas, R., Wolfe, C.D.A. (2001). A Comparison of the Costs and Survival of Hospital-Admitted Stroke Patients Across Europe. Stroke 32: 1684-1691 [Abstract] [Full text]  
• Weir, N., Dennis, M. S. (2001). Towards a National System for Monitoring the Quality of Hospital-Based Stroke Services. Stroke 32: 1415-1421 [Abstract] [Full text]  
• McKevitt, C., Dundas, R., Wolfe, C. (2001). Two Simple Questions to Assess Outcome After Stroke : A European Study. Stroke 32: 681-686 [Abstract] [Full text]  
• Man-Son-Hing, M., Laupacis, A., O'Connor, A. M., Coyle, D., Berquist, R., McAlister, F. (2000). Patient Preference-based Treatment Thresholds and Recommendations: A Comparison of Decision-analytic Modeling with the Probability-tradeoff Technique. Med Decis Making 20: 394-402 [Abstract]  
• Williams, L. S., Yilmaz, E. Y., Lopez-Yunez, A. M. (2000). Retrospective Assessment of Initial Stroke Severity With the NIH Stroke Scale. Stroke 31: 858-862 [Abstract] [Full text]  
• Freburger, J. K (1999). An Analysis of the Relationship Between the Utilization of Physical Therapy Services and Outcomes for Patients With Acute Stroke. ptjournal 79: 906-918 [Abstract] [Full text]  
• Wolfe, C. D. A., Tilling, K., Beech, R., Rudd, A. G. (1999). Variations in Case Fatality and Dependency From Stroke in Western and Central Europe. Stroke 30: 350-356 [Abstract] [Full text]  
• Lane, R., Burrell, K., Dennis, C. (1997). Study confirms results of systematic review of care in stroke unit. BMJ 315: 881-881 [Full text]  
• Kendrick, S., Bain, M. (1996). Further studies are needed to assess current outcome indicators. BMJ 313: 1006-1006 [Full text]  
• Barer, D., Ellul, J., Watkins, C. (1996). Correcting outcome data for case mix in stroke medicine. BMJ 313: 1005c-1006 [Full text]  
• Mant, J., Hicks, N. R, Fletcher, J. (1996). Study should have had more patients or longer time scale. BMJ 313: 1006a-1006 [Full text]  
• Evans, R. (1996). American guidelines on managed care are not the answer. BMJ 313: 946b-946 [Full text]