What value do computers provide to NHS hospitals?
BMJ 1996; 312 doi: https://doi.org/10.1136/bmj.312.7043.1407 (Published 01 June 1996) Cite this as: BMJ 1996;312:1407
- Chris Lock, principal research fellowa
- a Clinical Operational Research Unit, Department of Statistical Science, University College London, London WC1E 6BT
- Accepted 18 March 1996
As the NHS spends around £220 million a year on information technology for use by acute hospitals that are hard pressed for resources, it is reasonable to ask what value is provided. A review of rigorous scientific evidence for the value of information technology to NHS hospitals found that published evidence is scarce and far from conclusive. Information technology in NHS hospitals needs further assessment so that future decisions on such necessary and important investments are based on clear, well documented experience and research.
Recent reports from the Audit Commission have criticised the use of computer systems in NHS hospitals1 2 and have sparked a debate regarding the value of information technology to the NHS and to acute hospitals in particular. The costs are substantial (£220m a year), and some argue that the benefits are nebulous and diffuse. But where is the evidence? Assessing the value arising from investments in information technology, rather than merely their cost, is not straightforward. Much information technology investment has been in administrative systems that might be regarded as successful when they save money.
Recent thinking suggests that this is insufficient, and that a more comprehensive assessment should consider whether information technology supports the commercial and strategic objectives of the organisation.3 4 For the NHS, the impact of computer systems on patient care as well as on the “business” objectives of hospitals should be considered.
When computer systems are assessed against patient care objectives, outcomes should be expressed as would those arising from other clinical interventions. The range of outcomes that might arise from computer systems is potentially huge, and as yet no measure of outcome has been universally recognised. The Audit Commission offered a list of potential clinical benefits of information technology to patient care (box), which illustrates the breadth of potential benefits.
Method
By using computer and manual methods, searches were conducted for published information about hospital information technology systems that had undergone refereeing or peer review and contained sufficient detail to enable the value of the investment to be considered. The search included papers from medical and scientific journals, the proceedings of the annual British healthcare computing conference, and assessments published by the NHS Executive Information Management Group5 and individual hospitals.
Appropriate publications, from 1990 to July 1995, that described hospital or departmental systems in use in the NHS were identified by a methodical keyword search procedure. Full details of the method used are available on request.
ANALYSIS
A structured method of analysis was developed to assess the level of detail presented in the paper in an objective and replicable manner. Papers that gave sufficient detail to allow at least a provisional assessment of the value of the information technology system were separated from those that were essentially discussions or descriptions of systems or commentaries on such systems.
Potential benefits from information technology systems in hospitals
Referral to or attendance at casualty department
Easier access to medical history
Outpatient clinic
Improved access to summary patient details
Easier production of clinic letter to general practitioner
Admission
Improved access to summary patient details
Support for protocols or guidelines
Easier access to results of investigations
Quicker reporting of results of treatment
Discharge
Easier generation of discharge summary for general practitioner
More reliable data to support audit
Shared care better supported with community based professionals
The criteria in table 1 were used to assess the level of detail. The criteria focus on the investment made or resources consumed by the information technology project and on the advantages or improvements realised as a result of the project or system, whether clinical, administrative, or financial. The criteria recognise that benefits are generally more difficult to quantify than costs, and that benefits are even harder to translate into financial terms. Also, the criteria accommodate statements of benefits that arise from qualitative assessments. Reports that contained cost or benefit information that was sufficiently quantified (level 5 or 6) were identified for further analysis.
Criteria for assessing level of detail provided in published reports of information technology systems in hospitals
Results
Only 55 papers were found when the criteria for level of detail were applied to the data in papers found through Medline and the Bath Information and Data Service (BIDS)(table 2). Of these, just six concerned hospital systems in use in the NHS.6 7 8 9 10 11 Only one of these papers—from the Royal Hampshire County Hospital, Winchester7—gave details of costs or benefits to the required detail (level 5 or 6). Of the 465 papers given at the annual conferences on healthcare computing, 40 described hospital or departmental systems in use12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51; of these, seven papers described six cases with details to the required level (table 2).22 35 36 39 41 42 45
Cost-benefit evidence in publications found on Medline,* BIDS,+ and reports of healthcare computing conferences, 1990-5
A further nine cases were identified in five publications from the NHS Executive and other sources.52 53 54 55 56 After duplicates were removed, published information on the value of information technology consisted of cases from 12 hospitals. Table 3 summarises the key data on costs and benefits given in these cases.
Summary of evidence regarding value of information technology systems in hospitals
Discussion
EVIDENCE IS ELUSIVE
Published evidence on the value of information technology in hospitals is scarce and far from conclusive. The systems in these 12 hospitals represent an investment of £54m. It is estimated that, from 1990 to July 1995, the NHS invested £220m a year—a total of over £1000m—in information technology for use by NHS hospitals.1 These reports describe about 5% of this expenditure, which is a poor indicator of the true value of a large and important national investment.
Such elusiveness of evidence regarding information technology investments is not unusual. A recent review of the European literature also found few evaluation studies of automated information systems in health care.57 Only 13 of the 108 studies that were identified used any economic analysis. The author found it “astonishing” that so few studies attempted to show cost effectiveness.
In their review of the use of computers by general practioners, Sullivan and Mitchell reported that although 90% of general practitioners in the United Kingdom used computers, and 55% used them during consultations, only 30 studies had been carried out worldwide (between 1984 and 1994). The results were described as not fully conclusive.58
Powell noted that evaluation of information technology investments is problematic and complex in a range of industries.59 Examining the effect of information technology on the economic wellbeing of businesses, Landauer observed that, although service related organisations are those where the greatest productivity gains can be made, they are also where computer technology is sometimes least effective.60
BENEFITS ARE DIFFICULT TO ASSESS
Where specific productivity or cost effectiveness gains are sought through the use of information technology in hospitals, these can generally be assessed with health economics techniques. When clinical benefits are monitored, techniques such as randomised controlled trials can be considered, but these are inappropriate for assessing most information technology systems and there are difficulties in quantifying improvements and associating them directly with the use of the information technology system. New methods combining economic and clinical evaluation are starting to emerge and should inform the development of methods for assessing information technology systems.61
A further difficulty is the lack of clear, specific, care related objectives, which should be identified at the start of an information technology project and against which actual performance can be assessed. This lack of objectives may be one reason that information technology has not been included in the considerations of the NHS's health technology assessment programme. Despite these difficulties, attempts at assessment are made, as the evidence identified by this paper has shown, and these provide a measure of justification for the use of computers in hospitals.
NEED FOR MORE RESEARCH
If the NHS is to invest effectively in information technology a significant body of evidence must be developed to establish the value to healthcare delivery of different types of system and the most appropriate areas of application. That body of evidence does not currently exist. Wyatt recently pointed out the need for research and evaluation of information technology in hospitals,62 and these findings support this.
The Audit Commission has estimated that about a quarter of doctors' and nurses' time is spent collecting data and using information, which means that about 15% of hospital running costs—totalling around £3bna year—is expended on handling information.1 In this light, the use of modern information technology seems not only appropriate but unavoidable, and annual expenditure of £220m on hospital information technology seems modest.
Information technology professionals generally take the view that the need for improved systems is self evident in such complex, information intensive organisations as hospitals. Unless the benefits are proved, however, resources may be expended ineffectively. The process should take account of the best previous evidence, which until now has not been identified.
As proposals for information technology systems increasingly seek to produce improvements in patient care, clinicians have a greater part to play in these decisions, which compete for resources directly with investments in treatment and clinical facilities. This emerging requirement can be regarded as parallel to the current debate regarding evidence based medicine.63 64 Guidelines for the systematic review of research that are being developed65 may help inform the methodologies for assessment across a wide range of clinical interventions, including the use of information technology systems. Clinicians should seek evidence when they consider investing in information technology as they would for other initiatives.
I gratefully acknowledge the support and assistance of colleagues at the Clinical Operational Research Unit, University College London.
Footnotes
-
Funding Financial support for this research was provided by the Department of Health Research and Development Division and Siemens Nixdorf Medical Portfolio.
-
Conflict of interest The author is seconded from Siemens Nixdorf Medical Portfolio, which is a major supplier of computer systems to the NHS.