Effect of correcting outcome data for case mix: an example from stroke medicine
BMJ 1996; 312 doi: https://doi.org/10.1136/bmj.312.7045.1503 (Published 15 June 1996) Cite this as: BMJ 1996;312:1503- Richard J Davenport, clinical research fellowa,
- Martin S Dennis, senior lecturer in stroke medicinea,
- Charles P Warlow, professor of medical neurologya
- a University of Edinburgh, Department of Clinical Neurosciences, Western General Hospital, Edinburgh EH4 2XU
- Correspondence to: Dr Davenport.
- Accepted 3 April 1996
Abstract
Objective: To show the influence of variations in case mix on clinical outcome indicators for patients admitted to hospital with acute stroke.
Design: “Before and after” cohort study, with prospective, consecutive identification of patients and prospective follow up; multiple logistic regression analyses to correct for case mix variations.
Setting: University teaching hospital.
Subjects: 216 patients with stroke identified before the introduction of an organised stroke service, and 252 patients with stroke identified after its introduction.
Main outcome measures: Case fatality at 30 days and 12 months; for survivors at 12 months, proportions of patients who were independent (according to the Oxford handicap scale) and of those living at home.
Results: Crude outcome data suggested that patients in the cohort identified after the introduction of the stroke service were significantly more likely to be alive, independent, and living at home than patients managed before the stroke service. After adjustment for age and sex these “improvements” were less impressive but still significant. After adjustment for many other possible prognostic indicators, however, the differences between the two groups for all four outcomes were non-significant, suggesting that the “improvements” may have been entirely due to differences in case mix between the two cohorts, rather than the new stroke service.
Conclusions: Variations in case mix have a crucial influence on the interpretation of outcome data, and this is particularly important in non-randomised comparative studies. Such studies, comparing performance within and between different provider units, are likely to become increasingly common in the new reformed NHS. To allow meaningful interpretation, these studies must try to correct for case mix.
Key messages
Variations in case mix wield a vital influence over outcome
The government has published clinical outcome indicators for several diseases in Scottish hospitals that have not been adequately corrected for case mix
Interpretation of outcome data from non- randomised studies may be confounded by case mix, and researchers must try to adjust their data for case mix variables to allow meaningful comparisons
Introduction
The confounding influence that variations in case mix may exert on clinical outcome is well recognised.1 2 3 Despite this, the government has published for Scottish hospitals clinical outcome indicators for several diseases—including stroke—that have only been corrected for age and sex.4 These data have been widely interpreted by the media (and even by some doctors and managers) as league tables of hospital performance. To assess the impact of case mix on the outcome after acute stroke, we compared the outcome of patients managed before and after the introduction of an organised stroke service in our hospital. This service included an inpatient stroke unit, an intervention that has been shown in a systematic review of randomised controlled trials to reduce case fatality and dependence.5
Methods
We prospectively identified consecutive stroke patients who needed inpatient management in our hospital (excluding those with subarachnoid haemorrhage and those admitted to the regional neurosciences unit) by daily consultation with the admitting medical and nursing teams and perusal of admissions records. All patients were assessed by a stroke physician who completed a standard data form; the data were then transferred to an electronic database (stroke register). We followed up survivors either with a face to face or telephone interview or with a postal questionnaire. We measured four important outcomes: case fatality at 30 days and 12 months; the proportion of survivors living at home at 12 months after their stroke; and the proportion of survivors who were independent (<3 on the Oxford handicap scale6) at 12 months. We calculated uncorrected odds ratios and 95% confidence intervals for these outcomes. We then performed multiple logistic regression, forcing in 19 different indicators of case mix (box); we chose these variables on the basis of likely clinical importance and from our previous experience with prognostic models derived from large community and hospital based stroke registers (C Counsell, personal communication). This provided corrected odds ratios for the outcomes. We also corrected the data for age and sex alone.
Results
We identified 216 strokes before and 252 strokes after the introduction of the stroke unit, over a period of 27 months. Three patients were lost to follow up at 30 days and four at 12 months; all were overseas visitors. Independence data for six survivors were unavailable (two in the cohort managed before the introduction of the stroke unit and four in the in the cohort managed after). Figure 1 shows the uncorrected odds ratios; patients admitted after the introduction of the stroke unit had very significantly better outcomes—for example, a 50% increase in the odds of living at home at 12 months. Correcting for age and sex alone reduced these estimates of treatment effect, but they remained significant. When the data were corrected for case mix, however, all the estimates became non-significant, with odds ratios approaching unity.
Discussion
Because our study was a non-randomised, “before and after” comparison, we were aware that variations in case mix might significantly bias the outcomes. We therefore corrected the crude outcomes using multiple logistic regression analysis, which is a method of simultaneously adjusting for the effects of several different variables. Before discussing the implications of our results, we consider some of the problems of using these complicated mathematical models.8
Firstly, such models may “overcorrect” the data—that is, they may obscure a real but moderate treatment effect by including too many variables. Wasson et al suggested that for every variable there should be at least five patients with the outcome of particular interest.9 In our models we used 19 variables, and there were considerably more than 95 outcome events for each outcome we measured.
Secondly, models derived from one dataset may not perform well on another, independent dataset. In our department we have developed prognostic models for a variety of outcomes, including the four in the present study, which rely on five or six variables (C Counsell and M McDowall, personal communication). These models were derived from a community based stroke register (the Oxfordshire Community Stroke Project10) and have been validated on two further independent datasets (one community and one hospital based register). When we applied these validated models to the results of the current study we obtained almost identical corrected odds ratios as those in the figure, which was reassuring.
IMPLICATIONS OF FINDINGS
Our results confirm the dangers of comparing outcome measures that have not been adjusted for case mix; it was case mix, rather than a beneficial effect of our stroke unit, that seemed to be responsible for much of the observed “improvements” in outcome (although because of the imprecision of the corrected data, the results are still consistent with a moderate but non-significant beneficial effect). Several factors may have explained why case mix changed, but one factor was the closure of our hospital's casualty department during the study. Similar findings have been shown for other diseases.11 12 13 While epidemiologists and clinical trialists have been aware of this problem for some time, its importance in everyday clinical practice is perhaps less readily acknowledged.
Although randomised controlled trials remain the gold standard for assessing interventions, we believe that there are two areas where non-randomised comparative studies are likely to become increasingly common. Firstly, individual provider units are increasingly aware of the need to provide evidence of the quality of care that they deliver; because randomised controlled trials within single units are impractical and sometimes unethical, often the most feasible design for these studies is a “before and after” one. These non-randomised studies are susceptible to several biases, with the confounding effect of case mix variation one of the most potent; therefore, if providers wish to draw valid comparisons between outcomes, they must collect potentially important case mix variables as well as outcome measures. Secondly, the new NHS reforms encourage competition between provider units, with purchasers empowered to select the best treatment packages for their patients. Currently, little information is available to purchasers on which to base such decisions, as there are few comparative data between different hospitals for similar conditions.
Recently, the government published limited outcome indicators for several diseases, comparing the performance of hospitals in Scotland.4 In the foreword to this document the problems of case mix were acknowledged, and it was stated that the purpose of publication was not to provide an indication of the best and worst hospitals for general practitioners and patients but to improve overall standards of care. It was added that trusts that underperformed should immediately review their treatments in the relevant areas, and health boards and purchasers should be encouraged to question the quality of care delivered in these centres; yet these suggestions may be entirely unjustified as the data took no account of case mix. We have clearly shown that if the government and other agencies wish to provide meaningful comparative outcome data they must address the question of case mix before publishing any more potentially unreliable figures.
Addendum
After we submitted this paper for publication the government published an updated version of their outcome indicators.14 In addition to correction for age and sex, these data were also corrected for social deprivation with the Carstairs index,15 based on postcode sectors of residence according to the 1991 census. Data were also corrected for pre-existing morbidity, which was based on the principal Scottish Morbidity Record 1 (SMR1) diagnoses for the preceding five years, not on secondary diagnoses recorded at the time of discharge after stroke—that is, there was no correction for other conditions that had not been noted in previous hospital admissions. It was acknowledged that this system represented an extremely crude and only partial adjustment. Moreover, there was still no correction for crucially important case mix variables for stroke prognosis, such as level of consciousness on admission.
We are grateful to Mr Jim Slattery for advice regarding the statistical methods used in this paper.
Footnotes
-
Funding RJD is funded by the Medical Research Council (UK); MSD and the stroke register were funded by the Stroke Association (UK).
-
Conflict of interest None.