Implementation of computerised physician order entry (CPOE) and picture archiving and communication systems (PACS) in the NHS: quantitative before and after study

Impact of CPOE on primary outcomes

The reduction in outpatient tests could plausibly be attributed to implementation of CPOE. The CPOE system enabled clinicians to access the patient’s pathology test history during an outpatient appointment, which could have reduced the number of tests ordered because of a missing previous test result. This argument is strengthened by the reduction in repeat full blood counts ordered at consecutive outpatient appointments. Reasons for the large increase in urea and electrolytes test ordering across all trusts, and for the greater relative increase in ordering of this test in the intervention trust, were not apparent.

In a recent systematic review, CPOE was associated with a reduced volume of pathology tests in seven out of 11 studies, with no change in three studies, and with an increase in one study.2 Only one of the studies (showing reduced volume) was performed in outpatient departments, and the intervention evaluated in this study was a module added to an existing CPOE system to display test charges.9 We found one additional study in a US primary care setting, which reported a reduction in ordering of six types of pathology tests (including full blood count, urea and electrolytes, and urine culture) if previous test results were displayed.10 This result was for all tests combined; the slight decrease in orders for full blood count and urea and electrolytes tests was not analysed separately and orders for urine culture test showed a slight increase.10

Impact of PACS on primary outcomes

Implementation of PACS was associated with fewer computed tomograms being requested for inpatients but more scans requested at outpatient appointments. PACS was also associated with an increase in computed tomography repeated within 48 hours during inpatient stay. Possible explanations for the relatively large increases in outpatient computed tomography and repeat inpatient scans in the intervention trust, and the large increase in outpatient scans in trust 1, were not forthcoming from the trusts. New scanners were installed in the intervention trust in 2000 and 2006. A new scanner was installed in trust 1 in 2003, which replaced an existing machine. These results suggest that implementation of PACS in the intervention trust allowed an increasing demand for computed tomography to be met through outpatient appointments, rather than through inpatient admissions, possibly as a consequence of shorter turnaround times (from examination request to image availability).11 It is then plausible that those patients who still required hospital admission would be those patients who needed repeat scans. A large increase in outpatient computed tomography, however, was also seen in one of the control trusts, hence attribution of these effects to implementation of PACS is questionable. PACS was also associated with fewer repeat plain x ray film exams at consecutive outpatient appointments. As with repeat full blood counts at consecutive outpatient appointments, attribution of this effect to implementation of PACS is plausible if PACS enables the clinician to access the patient’s radiological examination history during the outpatient appointment. Fewer repeats could also be related to lower rejection rates.12

Impact of CPOE and PACS on secondary outcomes

There seemed to be a consistent reduction in the proportion of patients discharged at outpatient appointments after CPOE and PACS were implemented, but we found no explanation for this apparently detrimental effect. Neither could we explain the reductions in deaths or outpatient appointment attendances associated with CPOE. Attribution of changes in secondary outcomes due to implementation of CPOE or PACS was problematic because the hypothesised chain of causality linking implementation of CPOE or PACS to secondary outcomes was more tenuous than for primary outcomes. Studies on length of stay found no impact of CPOE2 or PACS,13 14 15 except for one that found a reduction associated with implementation of PACS.11 Secondary outcomes were more likely to be influenced by major process changes within each trust. For example, deployment of CPOE within trust 1 coincided with construction of a new hospital under a government private finance initiative (PFI). Such trust-wide changes would influence our estimates if they occurred differentially in intervention and control trusts and if they coincided with implementation of CPOE and PACS. The absence of effects of CPOE and PACS on most of our secondary outcomes could be interpreted as evidence that the effects that we observed on our primary outcomes were not confounded by process changes throughout the trust.

Study in context

CPOE in trust 1 and PACS in trust 4 were considered by managers and end users to have been successful implementations of these types of healthcare IT system,16 preceding by several years the rollout of similar systems under NPfIT. The NHS is leading the way in terms of the scale and homogeneity of its healthcare IT programme and, although running behind schedule and over budget, the programme continues to receive the support of managers and clinicians alike.5 6 CPOE and PACS, when fully integrated with the other information technology systems that comprise NPfIT (national electronic health records, patient administration systems, electronic referral, etc), might contribute to more dramatic quantitative changes.

Our study has shown that it is possible to use routinely collected patient level data from disparate sources within large healthcare institutions as a basis for assessing the impact of technological changes on indicators of clinical activity and operational efficiency. In the context of future research, the transmission of local (trust level) patient identifiers in commissioning datasets to the NHS-wide clearing service and the ready availability of datasets from specialist departments within trusts suggest that our technique of joining data from commissioning datasets with these specialist datasets could form the basis for operational research on a nationwide scale. Use of NHS patient numbers to identify patients uniquely might improve the quality of such an exercise. The use of intervention and control groups comprising a much larger number of trusts would improve the stability of effect estimates and would increase confidence in attributing causality where there was evidence of an effect.

Our difficulty in attributing causality to observed effects, however, also suggests that novel or complementary approaches to quantitative research are required. Smaller studies can be designed to measure effects at a much finer level of detail and with more specific contextual information. For example, a recent study involving the same CPOE system as was deployed in our intervention trust (but looking at inpatient tests for liver function and plasma gentamicin and vancomycin concentrations) found no change in the volume of test orders but did find other changes (in turnaround time, in information provided with specimens, and in ordering of tests removed from an order set), which suggest that changes might occur at a level of detail that was unobtainable from our data.17 Another study of test use in a coronary care unit found reductions in tests ordered when compared with a similar unit and detected no differences in clinical outcomes.18 Such smaller studies could be nested within a hospital-wide study.

Much closer linkage with qualitative research would enable a better understanding of quantitative findings. For example, we were not surprised that our study, like almost all previous studies, failed to detect any consistent or plausible beneficial impact of CPOE or PACS on outcomes such as length of inpatient stay. We could interpret the lack of effects on secondary outcomes to mean that, while CPOE systems and PACS might bring important qualitative improvements to the process of clinical care (particularly by making life easier for clinicians once they become familiar with systems),16 these benefits are difficult to quantify and detect on a macroscopic (hospital-wide or trust-wide) scale. Our original intention had been to conduct qualitative and quantitative research in parallel, but this was frustrated by delays in obtaining quantitative data. As a consequence, our interviews with users of the systems could not refer to preliminary quantitative findings to elicit explanations for observed effects.

Furthermore, one of our biggest difficulties was in obtaining detailed information on the rollout and use of IT systems during the study period, particularly in the control trusts. Frontline staff in pathology and radiology departments were too heavily burdened with work to respond to requests for information. Higher level staff (managers and consultants) expressed interest in the aims and ultimate success of our study but lacked sufficiently detailed historical knowledge of systems in these departments and referred us to the same frontline staff who had been unable to respond originally. These shortcomings were compounded by “institutional amnesia” resulting from high staff turnover and the demands of more immediate issues.5 6

In evaluating the impact of healthcare IT systems, it is conceivable that important qualitative benefits for staff, such as ease of working and reduced workplace stress, are not readily quantifiable. Conversely, quantifiable effects that are not closely linked with qualitative information are difficult to interpret. Healthcare IT has been described as a “diffuse” technology, meaning complex systems comprising multiple components, the implementation of which is as important as their function and the evaluation of which is inherently difficult.19 Furthermore, it has been argued that the implementation of such systems must be viewed as a process involving organisational change.20 Recent qualitative studies of CPOE21 22 23 24 25 26 and PACS27 28 29 30 implementations have successfully adopted this “sociotechnical” approach but were not supported by quantitative methods. We have shown the potential of our quantitative methods, but future application must be closely synchronised with qualitative methods.

Limitations

Although our study benefited from a large number of observations, adjustment for clustering by site and specialty gave rise to large standard errors. Hence, although there seemed to be evidence of potentially important effects for many outcomes, few could be measured with sufficient precision in our final analysis. We restricted our analyses to specialties common to all of the participating trusts, but our results remain susceptible to residual confounding because of differences in case mix between trusts. Confounding was a particular concern in the few instances where the indicator data showed substantial differences between trusts. Inclusion of specialty as a covariate in our regression models to control for differences in case mix, however, did not tend to change our point estimates.

We could not verify data quality, although outpatient data from the commissioning datasets have been assessed as reliable.31 Data on pathology tests and radiology examinations were unlikely to contain important omissions as these were obtained directly from pathology and radiology information systems used routinely to manage all requests. Some omissions might have arisen by using local patient identifiers to join these data with data from the commissioning datasets. We had no means to verify the reliability of this process, but the reasonable consistency of our outcome measures between trusts was reassuring.

Our “repeat investigation” measure was a proxy for redundant tests and was dictated by the data available from routine sources. We did not have the level of detail necessary to determine whether tests repeated within this interval were redundant (for example, redundant tests have typically been identified by chart review) or whether they reflected good clinical practice. Our method might not be equally applicable across specialties, but we found no evidence to the contrary by comparing the distributions of times to retest within specialties. Our choice of interval (48 hours) was consistent with that used in other studies.32 33 34 Misclassification of tests as redundant when clinically required, and vice versa, would probably have been non-differential and hence would have caused underestimation of underlying effects.

In addition to improved efficiency of delivery of care, modernisation of IT could improve patients’ health outcomes, most obviously through better patient safety. Because our study was designed to take advantage of routinely collected data, we were unable to investigate the impact of CPOE and PACS on health outcomes other than death and overall length of stay.

Conclusions

Efficiency gains from healthcare IT systems are difficult to quantify. Changes in routinely derived indicators are difficult to interpret and measure. We observed both beneficial and detrimental, or at least unexpected, changes so assumptions of substantial efficiency gains from healthcare IT systems might be unrealistic. Given the large overall benefit that would accrue from small efficiency gains occurring in all trusts across the NHS, further research is justified. Although our underlying methods are promising, quantitative research must be closely allied with qualitative research to provide context and to explain observed changes.