Evaluation of modernisation of adult critical care services in England: time series and cost effectiveness analysisBMJ 2009; 339 doi: https://doi.org/10.1136/bmj.b4353 (Published 12 November 2009) Cite this as: BMJ 2009;339:b4353
- Andrew Hutchings, lecturer1,
- Mary Alison Durand, research fellow1,
- Richard Grieve, senior lecturer in health economics1,
- David Harrison, statistician2,
- Kathy Rowan, director2,
- Judith Green, reader in medical sociology1,
- John Cairns, professor of health economics1,
- Nick Black, professor of health services research1
- 1Health Services Research Unit, London School of Hygiene and Tropical Medicine, London WC1E 7HT
- 2Intensive Care National Audit and Research Centre, Tavistock House, London WC1H 9HR
- Correspondence to: A Hutchings
- Accepted 26 June 2009
Objective To evaluate the impact and cost effectiveness of a programme to transform adult critical care throughout England initiated in late 2000.
Design Evaluation of trends in inputs, processes, and outcomes during 1998-2000 compared with last quarter of 2000-6.
Setting 96 critical care units in England.
Participants 349 817 admissions to critical care units.
Interventions Adoption of key elements of modernisation and increases in capacity. Units were categorised according to when they adopted key elements of modernisation and increases in capacity.
Main outcome measures Trends in inputs (beds, costs), processes (transfers between units, discharge practices, length of stay, readmissions), and outcomes (unit and hospital mortality), with adjustment for case mix. Differences in annual costs and quality adjusted life years (QALYs) adjusted for case mix were used to calculate net monetary benefits (valuing a QALY gain at £20 000 ($33 170, €22 100)). The incremental net monetary benefits were reported as the difference in net monetary benefits after versus before 2000.
Results In the six years after 2000, the risk of unit mortality adjusted for case mix fell by 11.3% and hospital mortality by 13.4% compared with the steady state in the three preceding years. This was accompanied by substantial reductions both in transfers between units and in unplanned night discharges. The mean annual net monetary benefit increased significantly after 2000 (from £402 ($667, €445) to £1096 ($1810, €1210)), indicating that the changes were relatively cost effective. The relative contribution of the different initiatives to these improvements is unclear.
Conclusion Substantial improvements in NHS critical care have occurred in England since 2000. While it is unclear which factors were responsible, collectively the interventions represented a highly cost effective use of NHS resources.
Politicians, managers, clinicians, and the public have strongly held views as to the costs and benefits (or lack of benefits) of the additional investment and the attempts to “modernise” the English National Health Service (NHS) since 2000. There is little rigorous evidence to inform such debates.
In 2000 the Department of Health in England advocated the “modernisation” of adult critical care1 and, as a parallel initiative, funded a 35% increase in the number of beds.2 Responsibility for leading the transformation of services lay with the newly created NHS Modernisation Agency and focused on integrating critical care with other acute services in the hospital, thus creating “comprehensive critical care.”
Key elements were to be the creation of 29 clinical networks, covering the whole country and sharing agreed protocols, and, within hospitals, the establishment of outreach services and the adoption of care bundles. Care bundles are groups of clinical guidelines that, when implemented and monitored together, are believed to produce better outcomes than would be the case with individual implementation.3 4 Implementation of one—the ventilator care bundle, which has four components that aim to prevent ventilator acquired pneumonia—was supported by the Modernisation Agency. Others include the central venous line bundle to prevent infection5 and the sepsis, resuscitation, and management bundles.6 In addition, hospitals were encouraged to undertake local improvement projects, which resulted in many diverse activities.
Although there has been no previous attempt to evaluate the impact of the changes instituted since 2000, some constituent elements have been investigated. There is mixed evidence about the impact of outreach services on mortality, unplanned admissions, and length of stay.7 8 Four studies of the ventilator care bundle reported reductions in the rate of ventilator associated pneumonia,4 9 10 11 another reported a reduction in mortality,12 and one reported shorter lengths of stay and duration of ventilation but no impact on crude mortality.13
To evaluate the impact of these initiatives it was necessary to define a point in time when transformation started. The Modernisation Agency’s critical care programme got under way at the end of 2000, which coincided with the substantial increase in investment (fig 1)⇓. We therefore compared the periods before and after the final quarter of 2000.
We compared trends in inputs, organisation, processes, and outcomes of critical care in a representative sample of English hospitals before and after the final quarter of 2000; explored associations of trends with each component intervention; and estimated the cost effectiveness of the interventions overall.
The Department of Health provided data on the number of available beds since 1999,14 the NHS reference costs provided the number of bed days and unit costs,15 and the case mix programme database (CMPD) from the Intensive Care National Audit and Research Centre (ICNARC) provided estimates of the number of occupied beds since 1998.16 Units were categorised by their increase in capacity from 2000 to 2004: <20% growth (40% of units), 20-50% (25% of units), or >50% (35% of units).
Classification of critical care networks
The Modernisation Agency categorised networks on five occasions during 2002-3 as excellent, ready to receive, developing, at risk, or at significant risk. We classified seven networks (24%) that achieved “ready to receive” or “excellent” status at the first assessment as “earlier adopters” and 16 (55%) as “later adopters”—those that did not achieve this status until later assessments. Six networks that had not achieved the “ready to receive” threshold by the final assessment were classified as “other.”
Classification of critical care units
A previous survey in 2005 showed that by the end of 2001 half the units had established outreach services (table 1)⇓.17 We conducted a second survey in November 2007 to identify whether and when a unit had implemented the ventilator care bundle, with the level of compliance for each year categorised as >90%, 75-90%, or <75% of eligible patients. By 2004 half of the units reported they had implemented the bundle (table 2)⇓. Three quarters of units that implemented the bundle reported compliance levels above 90%, with only seven (6%) below 75%.
Of the 96 English units with case mix programme data, 31 (32%) were classified as “later adopters”—they had not implemented an outreach service by the end of 2003 or had not implemented the ventilator care bundle by the end of 2005, or both—and 58 (60%) were classified as “earlier adopters”—they had implemented both by these dates.
There was no strong association between earlier adopter units and earlier adopter networks. In earlier adopter networks, 19 (73%) units were earlier adopters whereas in later adopter networks 30 (59%) were so classified (P=0.12). There was also no strong association between a unit’s classification and its change in bed capacity between 2000 and 2004.
Processes and outcomes
The case mix programme provided data on case mix, transfers, discharges, length of stay, readmissions, and unit and hospital mortality for patients admitted to critical care units. Data for 1998-2006 were available for 349 817 admissions in 96 English units.
Comparison of trends
We compared data for 1998-2000 with data for 2000-6. We used regression analysis to compare trends in processes: Poisson regression for dichotomous dependent variables to estimate risk ratios for the average annual change in each period and linear regression for continuous dependent variables to estimate mean change per year and its 95% confidence intervals. The two time periods were fitted as an interaction to test if the trends differed between them. The time periods were based on quarterly data adjusted for quarterly seasonality with the final quarter of 2000 representing the end of the first time period and the beginning of the second. We used predicted mortality from the ICNARC model to adjust for case mix.18 Units were included as random effects to allow for clustering of admissions within units. For the Poisson regression analysis we estimated bias corrected 95% confidence intervals using bootstrap resampling with 1000 replications.19
Cost effectiveness analysis
We adopted a hospital perspective that included all costs for hospital admissions during an episode, including the index admission and any readmissions. Length of stay was valued using NHS reference costs (1998-2005) to give an episode cost for each patient. For 1998-2003, a single unit cost for intensive care unit bed days was available for each trust, whereas for 2004 and 2005 the unit costs were taken as the mean across three levels of care weighted by the relative number of bed days in each category. The unit cost for general wards was the weighted mean for all healthcare resource groups for non-elective bed days from NHS reference costs. All unit costs were inflated to 2006-7 prices.20
Mortality data were extrapolated to calculate lifetime QALYs. We assumed that the life expectancy for survivors was 80% that of the general population matched for age and sex21 with their quality of life downweighted by 20%. Those patients who died in hospital were assigned a QALY of zero. Each individual’s lifetime QALYs (discounted at 3.5%) were valued at £20 000, (€22 100, $33 170) and their costs subtracted to give a net monetary benefit.22 Similar regression models (to those described above) were used with annual costs, QALYs, and net monetary benefit as dependent variables and separate components of the ICNARC risk prediction model as independent variables.
Association with organisational changes
We used similar regression models to evaluate the impact of three organisational changes after 2000: increase in capacity, adoption of clinical networks, and adoption of outreach and ventilator care bundle. Interactions were fitted between each type of change to test whether trends in processes and outcomes were associated with any particular variable.
Trends in inputs
The annual expenditure on critical care increased in real terms from £700m (1999-2000) to £1bn (2005-6). This was associated with a 35% increase in the number of staffed beds in general intensive care units (fig 2)⇓, with more of the increase in high dependency (106%) than in intensive care beds (23%). The main increase occurred during winter 2000-1, when high dependency beds increased by 57.5% and intensive care beds by 7.2%, after which there was an average 9.0% rise per year for high dependency and 1.4% rise per year for intensive care beds. Over the period 1999-2006, the mean cost of an intensive care bed day rose slightly from £1551 to £1647 (2006-7 prices).
Trends in processes
Case mix of admissions
From 1998 to 2006 the proportion of women in the case mix increased, as did the mean age of those admitted (from 59.6 to 60.5) (table 3)⇓. Although there was no consistent change either in the proportion with at least one chronic condition or in the mean physiology score, the mean predicted risk of mortality rose from 30.5% in 1998 to 32.1% in 2000 but subsequently fell to 31.4% in 2006, indicating that less severe cases were being admitted. Analysis by 10ths of predicted risk of mortality in England showed no widening in the distribution of cases.
Transfers and discharges
Transfers out of units to another unit to receive the same level of care declined by 11.0% a year after 2000 (table 4)⇓ compared with annual increases of 2.6% before 2000 (P<0.001). Similarly, transfers into units declined 8.7% a year after 2000 compared with an annual 3.8% increase beforehand (P<0.001) (fig 3)⇓.
Early discharges because of a shortage of critical care beds declined from 7.1% in 1998 to 3.3% in 2006 (fig 4)⇓. Although the rate of night discharges (midnight to 4:59 am) steadily increased from 2.8% in 1998 to 4.2% in 2006 (fig 5)⇓, the proportion reported as being because of a shortage of critical care beds declined (44.5% to 21.8%), suggesting the proportion deemed “unplanned” fell from 1.2% to 0.9%. Before 2000, unplanned night discharges were increasing by 3.1% a year whereas afterwards they decreased by 7.7% a year (P=0.008) (table 4).⇑ A similar pattern emerged when we considered discharges between 10 pm and 6:59 am.
In contrast, the number of discharges delayed due to a shortage of ward beds rose steadily (fig 4) from 2.7% in 1998 to 14.2% by 2006, though the average annual increase after 2000 (16.2%) was lower than before (38.0%) (P<0.001) (table 4).⇑ There was an increase in discharges directly to patients’ normal place of residence (from 0.6% to 2.1%). The trend was not significantly greater after 2000 than before.
Length of stay and readmissions
After adjustment for differences in case mix, the mean length of stay increased before 2000 by 0.243 days a year but by only 0.036 days a year afterwards (P<0.001) (table 5)⇓. There was no significant difference in the decline in readmissions within 24 hours before and after 2000 (table 6)⇓. There was, however, a faster decline in readmissions within 48 hours after 2000 (P=0.006) (fig 6)⇓.
Trends in outcomes
While unit mortality adjusted for case mix did not change between 1998 and 2000, subsequently it fell by an average of 2.0% a year (P<0.001) (table 7)⇓, indicating a total fall of 11.3% in the six years after the final quarter of 2000. A similar pattern was observed for hospital mortality adjusted for case mix, with an average annual decline after 2000 of 2.4%, indicating a total fall of 13.4% over six years. Annual unit and hospital mortality adjusted for case mix also show a similar pattern (fig 7)⇓.
Cost effectiveness of changes in critical care
The decrease in the annual change in the mean length of stay, both in critical care (by 0.18 days, P<0.001) and for the subsequent ward stay (by 0.35 days, P<0.001), meant the mean annual cost increases after 2000 (£196) were smaller than before (£391). The mean incremental cost (adjusted for case mix) therefore fell by £195 (P<0.001).
Annual improvements in the mean lifetime QALYs (adjusted for case mix) were slightly greater after 2000 and so the mean incremental QALYs were positive (0.025), though not significant (P=0.06) (table 8)⇓. Valuing a QALY at £20 000, coupled with the decline in incremental costs (−£195), resulted in a positive incremental net monetary benefit of £692 (P=0.008). Hence, if the differences in costs and QALYs can be attributed to the interventions, then they were relatively cost effective.
Sensitivity analysis had little effect on the main findings. Firstly, the effect of taking mean unit costs from all English NHS trusts, rather than just participants from the case mix programme, led to a relative decrease in costs after 2000, lower incremental costs, and a higher incremental net monetary benefit (table 9)⇓. Secondly, re-weighting mean unit costs to include a 20% high dependency component led to a relative reduction in mean costs after 2000 and a higher incremental net monetary benefit. Re-weighting to reflect the gradual increase in the size of the high dependency component resulted in similar results to the base case. Thirdly, when the summary baseline probability of death was used for case mix adjustment rather than the separate components of the ICNARC model, this led to greater QALY gains, smaller incremental costs, and a larger incremental net monetary benefit. Fourthly, assuming the long term survival and the quality of life of patients was the same as for the general population23 yielded a higher incremental net monetary benefit. Finally, extending the analysis from 96 to all 159 units in the case mix programme also led to a higher incremental net monetary benefit as did valuing a QALY at £30 000.
Impact of organisational changes
The benefit of being in a “later adopter” network was greater than being in a “earlier adopter” network: faster declines in the proportion of admissions that were transfers in (10.0% v 6.2% a year), transfers out (13.4% v 4.4% a year), early discharges (12.6% v 4.5% a year), and unplanned night discharges (8.6% v 1.4% a year); smaller increases in delayed discharges (12.2% v 23.6% a year); less increase in mean length of stay (0.015 v 0.089 days a year); and a faster decline in unit (2.5% v 1.3% a year) and hospital mortality (2.9% v 1.9% a year).
The effects of the time at which units adopted outreach and the ventilator care bundle were less clear. Later adopter units were associated with some benefits: faster declines in transfers in (11.8% v 8.2% a year) and unplanned night discharges (8.9% v 3.0% a year); slower increase in delayed discharges (8.1% v 18.4% a year); less increase in length of stay (0.024 v 0.072 days a year); faster decline in hospital mortality (2.8% v 2.2% a year). These units, however, experienced a slower decline in transfers out (9.6% v 13.0% a year) and in early discharges (7.7% v 11.8% a year).
Units with the largest increases in capacity were associated with faster declines in transfers in (13.6% v 5.8% a year), transfers out (17.9% v 8.5% a year), early discharges (17.5% v 6.5% a year), and unplanned night discharges (12.1% v 6.1% a year); and slower increases in delayed discharges (14.4% v 17.7% a year). There were, however, smaller declines in unit mortality (1.8% v 2.7%), though no significant differences for hospital mortality.
After 2000, unit mortality adjusted for case mix in England fell dramatically by 2.0% a year and hospital mortality by 2.4% a year (compared with no change between 1998 and 2000). This was accompanied by a decrease of 11.0% a year in transfers out (for the same level of care) to other units and a fall of 8.7% a year in transfers in, whereas previously both proportions had been rising. In addition, the proportion of unplanned night discharges declined by 7.7% a year. Despite small increases in average unit costs, the cost effectiveness of critical care increased after 2000, partly as a result of the improvements in outcome and partly because of smaller increases in the mean length of stay.
If such changes can be attributed to the initiatives to transform critical care (increased capacity, clinical networks, outreach services, and the ventilator care bundle) this package can be regarded as a more cost effective intervention than many other healthcare interventions. Improvements were associated with later, rather than earlier, adoption of organisational changes. There are two possible explanations for what seems to be a counterintuitive finding: later adopters might have benefited from the experiences of earlier adopters and also might have spent longer preparing for change and thus established greater commitment from staff. While increased capacity was not directly associated with improvements in outcomes, it was associated with declines in the rate of transfers and early discharges and might have contributed indirectly via unmeasured effects, such as improvements in staff morale.
Despite striking improvements in processes, outcomes, and cost effectiveness in England, the contribution of the explicit programme promoted by the Modernisation Agency is unclear. It is possible that concurrent changes—such as the introduction of “hospital at night,”24 new staff contracts,25 and nurse consultants26—which were also being implemented during this period contributed to the improvements observed.
Defining and measuring interventions
Inclusion of interventions had to be restricted to those that were feasible to measure. While this included the principal elements, it was not comprehensive. Creating simple dichotomies (earlier and later adopters) for networks and units might have masked some associations, and data on when interventions were adopted might have been subject to recall bias. Both limitations, however, would have reduced the likelihood of identifying associations rather than created spurious ones.
Given the increasing proportion of lower cost high dependency beds over time, use of overall bed numbers would have overestimated costs in the later period and thus underestimated cost effectiveness. The use of average NHS reference costs is justified by their similarity to deriving average unit costs for critical care with micro-costing techniques.27 Changes in the reporting of unit costs in 2004 made little difference given that the main determinants of the incremental net monetary benefit were the relative reduction in length of stay and the gain in QALYs.
While the final quarter of 2000 represented the optimum time point to mark the start of the interventions, adoption did not occur until later in many units. This will have tended to reduce the observed impact. Our approach also assumes that any changes in processes and outcomes observed after 2000 are linear.
There is some uncertainty as to the longevity and quality of life of survivors of critical care. Recent research suggests that the assumption of a 20% deficit might overstate the relative decrement,28 and other analyses have assumed the same survival and quality of life as the general population.29 The sensitivity analysis, however, showed that our findings were robust.
It was important to recognise the hierarchical nature of the data in the analysis.30 The main analyses allowed for clustering within critical care units by fitting units as random effects. Networks were not treated as a separate level because of potential biases in those with a lower participation in the case mix programme. Some caution is therefore required in interpreting the findings for network adoption.
This attempt at a nationwide evaluation of policies pursued since the publication of the NHS Plan in 2000, albeit limited to one specific part of health care, suggests that the interventions represent a highly cost effective use of resources. In any time series analysis the attribution of causality is a challenge. This is particularly so for evaluations of complex interventions at a national level. The ability to make causal inferences about the benefits of “modernisation” is limited by the lack of a comparable control group of critical care units in which there were no modernisation interventions. Attribution is also challenged by the presence of other concurrent interventions, such as the introduction of hospital at night and a myriad local modernisation projects. As a consequence, conclusions must be restricted to the observation that significant improvements in outcomes and processes occurred in 2000-6 without being able to link such changes to specific interventions.
Whatever the reasons for the improvements, considerable additional expenditure on critical care combined with an explicit centrally driven programme of modernisation has resulted in dramatic improvements in outcomes.
What is already known on this topic
Since 2000, there have been considerable increases in funding for adult critical care in the NHS in England
A specific attempt has been made to “modernise” services by establishing critical care networks, introducing outreach services in hospitals, and adopting clinical guidelines in the form of care bundles
There have been conflicting claims as to the impact of these developments on the outcome of care (intensive care unit and hospital mortality) and the processes of care (transfers, early discharges, delayed discharges)
What this study adds
In the six years since 2000 there have been major improvements in outcome, with a 13.4% fall in hospital mortality and in processes of care (fewer transfers, fewer early discharges)
Collectively the interventions associated with the modernisation of critical care represent a highly cost effective use of NHS resources
Cite this as: BMJ 2009;339:b4353
We thank all patients and staff in the participating units in the ICNARC case mix programme, members of the adult critical care stakeholders’ forum, and members of the study reference group: Deborah Dawson, Lisa Hinton, Sue MacFarlane, Mick Nielsen, Mike Pepperman, Sue Shepherd, and Keith Young. We are grateful for administrative support and assistance provided by Carolyn Knipe, Emma North, Keryn Vella, Cathy Welch, and Emma Wood.
Contributors: All authors took part in planning the study. AH and RG analysed the data. All authors took part in interpreting the results and reporting the research. AH is guarantor.
Funding: The study was funded by the NIHR SDO research and development programme (grant SDO/133/2006). AH was supported by a MRC special training fellowship in health services research. The study funders had no role in study design; in the collection, analysis, and interpretation of data; in the writing of the report; and in the decision to submit the article for publication. The researchers were independent from the funders
Competing interests: None declared.
Ethical approval: Not required
Data sharing: No additional data available.
This is an open-access article distributed under the terms of the Creative Commons Attribution Non-commercial License, which permits use, distribution, and reproduction in any medium, provided the original work is properly cited, the use is non commercial and is otherwise in compliance with the license. See: http://creativecommons.org/licenses/by-nc/2.0/ and http://creativecommons.org/licenses/by-nc/2.0/legalcode.