CCBYNC Open access
Research

Evaluation of modernisation of adult critical care services in England: time series and cost effectiveness analysis

BMJ 2009; 339 (Published 12 November 2009) Cite this as: BMJ 2009;339:b4353
1. Andrew Hutchings, lecturer1,
2. Mary Alison Durand, research fellow1,
3. Richard Grieve, senior lecturer in health economics1,
4. David Harrison, statistician2,
5. Kathy Rowan, director2,
6. Judith Green, reader in medical sociology1,
7. John Cairns, professor of health economics1,
8. Nick Black, professor of health services research1
1. 1Health Services Research Unit, London School of Hygiene and Tropical Medicine, London WC1E 7HT
2. 2Intensive Care National Audit and Research Centre, Tavistock House, London WC1H 9HR
1. Correspondence to: A Hutchings andrew.hutchings{at}lshtm.ac.uk
• Accepted 26 June 2009

Abstract

Objective To evaluate the impact and cost effectiveness of a programme to transform adult critical care throughout England initiated in late 2000.

Design Evaluation of trends in inputs, processes, and outcomes during 1998-2000 compared with last quarter of 2000-6.

Setting 96 critical care units in England.

Participants 349 817 admissions to critical care units.

Interventions Adoption of key elements of modernisation and increases in capacity. Units were categorised according to when they adopted key elements of modernisation and increases in capacity.

Association with organisational changes

We used similar regression models to evaluate the impact of three organisational changes after 2000: increase in capacity, adoption of clinical networks, and adoption of outreach and ventilator care bundle. Interactions were fitted between each type of change to test whether trends in processes and outcomes were associated with any particular variable.

Results

Trends in inputs

The annual expenditure on critical care increased in real terms from £700m (1999-2000) to £1bn (2005-6). This was associated with a 35% increase in the number of staffed beds in general intensive care units (fig 2), with more of the increase in high dependency (106%) than in intensive care beds (23%). The main increase occurred during winter 2000-1, when high dependency beds increased by 57.5% and intensive care beds by 7.2%, after which there was an average 9.0% rise per year for high dependency and 1.4% rise per year for intensive care beds. Over the period 1999-2006, the mean cost of an intensive care bed day rose slightly from £1551 to £1647 (2006-7 prices).

Fig 2 Number of critical care beds in England located in general units providing intensive care 1999-2006

Trends in processes

Case mix of admissions

From 1998 to 2006 the proportion of women in the case mix increased, as did the mean age of those admitted (from 59.6 to 60.5) (table 3). Although there was no consistent change either in the proportion with at least one chronic condition or in the mean physiology score, the mean predicted risk of mortality rose from 30.5% in 1998 to 32.1% in 2000 but subsequently fell to 31.4% in 2006, indicating that less severe cases were being admitted. Analysis by 10ths of predicted risk of mortality in England showed no widening in the distribution of cases.

Table 3

Characteristics of patients admitted to case mix programme units in England (1998-2006)

View this table:

Transfers and discharges

Transfers out of units to another unit to receive the same level of care declined by 11.0% a year after 2000 (table 4) compared with annual increases of 2.6% before 2000 (P<0.001). Similarly, transfers into units declined 8.7% a year after 2000 compared with an annual 3.8% increase beforehand (P<0.001) (fig 3).

Fig 3 Critical care transfers as proportion of admissions (transfers in) and discharges (transfers out) 1998-2006

Table 4

Comparison of average annual changes (adjusted for case mix) in transfers and discharges between 1998-2000 and 2000-6 (96 units)

View this table:

Early discharges because of a shortage of critical care beds declined from 7.1% in 1998 to 3.3% in 2006 (fig 4). Although the rate of night discharges (midnight to 4:59 am) steadily increased from 2.8% in 1998 to 4.2% in 2006 (fig 5), the proportion reported as being because of a shortage of critical care beds declined (44.5% to 21.8%), suggesting the proportion deemed “unplanned” fell from 1.2% to 0.9%. Before 2000, unplanned night discharges were increasing by 3.1% a year whereas afterwards they decreased by 7.7% a year (P=0.008) (table 4). A similar pattern emerged when we considered discharges between 10 pm and 6:59 am.

Fig 4 Early discharges, reported delayed discharges, and discharges directly to normal place of residence as proportion of all discharges 1998-2006

Fig 5 Night discharges and unplanned night discharges as proportion of all discharges 1998-2006

In contrast, the number of discharges delayed due to a shortage of ward beds rose steadily (fig 4) from 2.7% in 1998 to 14.2% by 2006, though the average annual increase after 2000 (16.2%) was lower than before (38.0%) (P<0.001) (table 4). There was an increase in discharges directly to patients’ normal place of residence (from 0.6% to 2.1%). The trend was not significantly greater after 2000 than before.

Length of stay and readmissions

After adjustment for differences in case mix, the mean length of stay increased before 2000 by 0.243 days a year but by only 0.036 days a year afterwards (P<0.001) (table 5). There was no significant difference in the decline in readmissions within 24 hours before and after 2000 (table 6). There was, however, a faster decline in readmissions within 48 hours after 2000 (P=0.006) (fig 6).

Fig 6 Readmissions within 24 and 48 hours as proportion of all discharges 1998-2006

Table 5

Comparison of average annual changes in mean length of stay (days) before and after 2000 (n=96 units)

View this table:
Table 6

Comparison of average annual changes in readmission rates before and after 2000 (n=96 units)

View this table:

Trends in outcomes

While unit mortality adjusted for case mix did not change between 1998 and 2000, subsequently it fell by an average of 2.0% a year (P<0.001) (table 7), indicating a total fall of 11.3% in the six years after the final quarter of 2000. A similar pattern was observed for hospital mortality adjusted for case mix, with an average annual decline after 2000 of 2.4%, indicating a total fall of 13.4% over six years. Annual unit and hospital mortality adjusted for case mix also show a similar pattern (fig 7).

Fig 7 Relative risk (95% confidence interval) of hospital mortality and unit mortality adjusted for case mix, 1998-2006

Table 7

Relative risk for annual change in unit and hospital mortality adjusted for case mix before and after 2000 (n=96 units)

View this table:

Cost effectiveness of changes in critical care

The decrease in the annual change in the mean length of stay, both in critical care (by 0.18 days, P<0.001) and for the subsequent ward stay (by 0.35 days, P<0.001), meant the mean annual cost increases after 2000 (£196) were smaller than before (£391). The mean incremental cost (adjusted for case mix) therefore fell by £195 (P<0.001).

Annual improvements in the mean lifetime QALYs (adjusted for case mix) were slightly greater after 2000 and so the mean incremental QALYs were positive (0.025), though not significant (P=0.06) (table 8). Valuing a QALY at £20 000, coupled with the decline in incremental costs (−£195), resulted in a positive incremental net monetary benefit of £692 (P=0.008). Hence, if the differences in costs and QALYs can be attributed to the interventions, then they were relatively cost effective.

Table 8

Incremental cost effectiveness (mean predicted costs*, QALYs, and net monetary benefit) for 2000-6 compared with 1998-2000

View this table:

Sensitivity analysis had little effect on the main findings. Firstly, the effect of taking mean unit costs from all English NHS trusts, rather than just participants from the case mix programme, led to a relative decrease in costs after 2000, lower incremental costs, and a higher incremental net monetary benefit (table 9). Secondly, re-weighting mean unit costs to include a 20% high dependency component led to a relative reduction in mean costs after 2000 and a higher incremental net monetary benefit. Re-weighting to reflect the gradual increase in the size of the high dependency component resulted in similar results to the base case. Thirdly, when the summary baseline probability of death was used for case mix adjustment rather than the separate components of the ICNARC model, this led to greater QALY gains, smaller incremental costs, and a larger incremental net monetary benefit. Fourthly, assuming the long term survival and the quality of life of patients was the same as for the general population23 yielded a higher incremental net monetary benefit. Finally, extending the analysis from 96 to all 159 units in the case mix programme also led to a higher incremental net monetary benefit as did valuing a QALY at £30 000.

Table 9

Sensitivity analyses reporting incremental costs (£), incremental QALYs, and incremental net monetary benefit (NMB) (£20 000 per QALY) for different scenarios

View this table:

Impact of organisational changes

The benefit of being in a “later adopter” network was greater than being in a “earlier adopter” network: faster declines in the proportion of admissions that were transfers in (10.0% v 6.2% a year), transfers out (13.4% v 4.4% a year), early discharges (12.6% v 4.5% a year), and unplanned night discharges (8.6% v 1.4% a year); smaller increases in delayed discharges (12.2% v 23.6% a year); less increase in mean length of stay (0.015 v 0.089 days a year); and a faster decline in unit (2.5% v 1.3% a year) and hospital mortality (2.9% v 1.9% a year).

The effects of the time at which units adopted outreach and the ventilator care bundle were less clear. Later adopter units were associated with some benefits: faster declines in transfers in (11.8% v 8.2% a year) and unplanned night discharges (8.9% v 3.0% a year); slower increase in delayed discharges (8.1% v 18.4% a year); less increase in length of stay (0.024 v 0.072 days a year); faster decline in hospital mortality (2.8% v 2.2% a year). These units, however, experienced a slower decline in transfers out (9.6% v 13.0% a year) and in early discharges (7.7% v 11.8% a year).

Units with the largest increases in capacity were associated with faster declines in transfers in (13.6% v 5.8% a year), transfers out (17.9% v 8.5% a year), early discharges (17.5% v 6.5% a year), and unplanned night discharges (12.1% v 6.1% a year); and slower increases in delayed discharges (14.4% v 17.7% a year). There were, however, smaller declines in unit mortality (1.8% v 2.7%), though no significant differences for hospital mortality.

Discussion

Main findings

After 2000, unit mortality adjusted for case mix in England fell dramatically by 2.0% a year and hospital mortality by 2.4% a year (compared with no change between 1998 and 2000). This was accompanied by a decrease of 11.0% a year in transfers out (for the same level of care) to other units and a fall of 8.7% a year in transfers in, whereas previously both proportions had been rising. In addition, the proportion of unplanned night discharges declined by 7.7% a year. Despite small increases in average unit costs, the cost effectiveness of critical care increased after 2000, partly as a result of the improvements in outcome and partly because of smaller increases in the mean length of stay.

If such changes can be attributed to the initiatives to transform critical care (increased capacity, clinical networks, outreach services, and the ventilator care bundle) this package can be regarded as a more cost effective intervention than many other healthcare interventions. Improvements were associated with later, rather than earlier, adoption of organisational changes. There are two possible explanations for what seems to be a counterintuitive finding: later adopters might have benefited from the experiences of earlier adopters and also might have spent longer preparing for change and thus established greater commitment from staff. While increased capacity was not directly associated with improvements in outcomes, it was associated with declines in the rate of transfers and early discharges and might have contributed indirectly via unmeasured effects, such as improvements in staff morale.

Despite striking improvements in processes, outcomes, and cost effectiveness in England, the contribution of the explicit programme promoted by the Modernisation Agency is unclear. It is possible that concurrent changes—such as the introduction of “hospital at night,”24 new staff contracts,25 and nurse consultants26—which were also being implemented during this period contributed to the improvements observed.

Methodological limitations

Defining and measuring interventions

Inclusion of interventions had to be restricted to those that were feasible to measure. While this included the principal elements, it was not comprehensive. Creating simple dichotomies (earlier and later adopters) for networks and units might have masked some associations, and data on when interventions were adopted might have been subject to recall bias. Both limitations, however, would have reduced the likelihood of identifying associations rather than created spurious ones.

Measuring inputs

Given the increasing proportion of lower cost high dependency beds over time, use of overall bed numbers would have overestimated costs in the later period and thus underestimated cost effectiveness. The use of average NHS reference costs is justified by their similarity to deriving average unit costs for critical care with micro-costing techniques.27 Changes in the reporting of unit costs in 2004 made little difference given that the main determinants of the incremental net monetary benefit were the relative reduction in length of stay and the gain in QALYs.

Analysis

While the final quarter of 2000 represented the optimum time point to mark the start of the interventions, adoption did not occur until later in many units. This will have tended to reduce the observed impact. Our approach also assumes that any changes in processes and outcomes observed after 2000 are linear.

There is some uncertainty as to the longevity and quality of life of survivors of critical care. Recent research suggests that the assumption of a 20% deficit might overstate the relative decrement,28 and other analyses have assumed the same survival and quality of life as the general population.29 The sensitivity analysis, however, showed that our findings were robust.

It was important to recognise the hierarchical nature of the data in the analysis.30 The main analyses allowed for clustering within critical care units by fitting units as random effects. Networks were not treated as a separate level because of potential biases in those with a lower participation in the case mix programme. Some caution is therefore required in interpreting the findings for network adoption.

Conclusions

This attempt at a nationwide evaluation of policies pursued since the publication of the NHS Plan in 2000, albeit limited to one specific part of health care, suggests that the interventions represent a highly cost effective use of resources. In any time series analysis the attribution of causality is a challenge. This is particularly so for evaluations of complex interventions at a national level. The ability to make causal inferences about the benefits of “modernisation” is limited by the lack of a comparable control group of critical care units in which there were no modernisation interventions. Attribution is also challenged by the presence of other concurrent interventions, such as the introduction of hospital at night and a myriad local modernisation projects. As a consequence, conclusions must be restricted to the observation that significant improvements in outcomes and processes occurred in 2000-6 without being able to link such changes to specific interventions.

Whatever the reasons for the improvements, considerable additional expenditure on critical care combined with an explicit centrally driven programme of modernisation has resulted in dramatic improvements in outcomes.

What is already known on this topic

• Since 2000, there have been considerable increases in funding for adult critical care in the NHS in England

• A specific attempt has been made to “modernise” services by establishing critical care networks, introducing outreach services in hospitals, and adopting clinical guidelines in the form of care bundles

• There have been conflicting claims as to the impact of these developments on the outcome of care (intensive care unit and hospital mortality) and the processes of care (transfers, early discharges, delayed discharges)

What this study adds

• In the six years since 2000 there have been major improvements in outcome, with a 13.4% fall in hospital mortality and in processes of care (fewer transfers, fewer early discharges)

• Collectively the interventions associated with the modernisation of critical care represent a highly cost effective use of NHS resources

Notes

Cite this as: BMJ 2009;339:b4353

Footnotes

• We thank all patients and staff in the participating units in the ICNARC case mix programme, members of the adult critical care stakeholders’ forum, and members of the study reference group: Deborah Dawson, Lisa Hinton, Sue MacFarlane, Mick Nielsen, Mike Pepperman, Sue Shepherd, and Keith Young. We are grateful for administrative support and assistance provided by Carolyn Knipe, Emma North, Keryn Vella, Cathy Welch, and Emma Wood.

• Contributors: All authors took part in planning the study. AH and RG analysed the data. All authors took part in interpreting the results and reporting the research. AH is guarantor.

• Funding: The study was funded by the NIHR SDO research and development programme (grant SDO/133/2006). AH was supported by a MRC special training fellowship in health services research. The study funders had no role in study design; in the collection, analysis, and interpretation of data; in the writing of the report; and in the decision to submit the article for publication. The researchers were independent from the funders

• Competing interests: None declared.

• Ethical approval: Not required

• Data sharing: No additional data available.

This is an open-access article distributed under the terms of the Creative Commons Attribution Non-commercial License, which permits use, distribution, and reproduction in any medium, provided the original work is properly cited, the use is non commercial and is otherwise in compliance with the license. See: http://creativecommons.org/licenses/by-nc/2.0/ and http://creativecommons.org/licenses/by-nc/2.0/legalcode.

View Abstract