Evaluating computerised health information systems: hard lessons still to be learnt

Sir
As a former IBM employee who was intimately involved in the Limpopo
project, I would like (in my private capacity) to expand on some of the
points made by Littlejohns et al. I fear that the attempt to re-affirm
universals, has obscured some of the particulars. The result is
prejudicial to all of us who were involved.
Regrettably, the article conveys the impression of an incompetent client
and an overzealous vendor.

In practice, all of the “lessons” mentioned were identified during
the project (independently of the evaluation), but could not be applied,
for contractual and “real life” reasons
1. I believe the authors have misinterpreted the role of culture.
Irrespective of other concerns, the introduction of a computerised
information system amounted to a large-scale culture change. Hospitals
which had traditionally operated on paper-based, “batch mode” principles,
with a degree of laissez faire, had to suddenly migrate to “on-line, real-
time” principles, with control measures and instantaneous reporting. In
the best of circumstances, such a culture change might take many months
(witness the reluctance of many clinicians to work on-line). In ours, it
might take several years. By our internal assessments, certain hospitals
which had already had some experience of computerised systems, experienced
less upheaval. But how much preparatory education is sufficient for such
large-scale change, and how effective is it, if it is not temporally
proximate to the change?
2. As regards evaluations, a measuring instrument which is applied at six
months (say), might not measure a slow, evolutional change. I say this
with hindsight, as I was one of the “special advisory group of experts”.
3. Surfeit of change. In Limpopo, four previous administrations, of
varying efficiency and probity, were being amalgamated. Hospitals and
Departmental management were suffering from “change fatigue”. Many of the
managers were recent appointees, grappling with multiple challenges.
Perhaps such projects should not be undertaken during times of great
transition, but that would leave developing countries awaiting Godot.
4. In our latitudes there is often an inappropriate expectation, no doubt
fuelled by the computer industry, that Information Technology (IT) is the
answer to intractable organisational problems. In practice, IT amplifies
process and managerial deficiencies. The lack of satisfaction with the
subsequent, less sophisticated system, is therefore not surprising.
5. There is a perverse “efficiency” in the case of a shortage of skilled
staff – fewer people deal with a constant or increasing workload.
Injection of investment often has no apparent effect on efficiency,
because it allows a return to a more “normal” state.
6. Inefficient processes generally have one or more rate-limiting steps.
In Limpopo at the time, there were some three hundred-odd doctors manning
the 42 hospitals. It would therefore be naïve to expect a decrease in
patient waiting time. As the research completed by Mbananga et al
demonstrated, all we achieved was to shift the queues from the
administrative waiting area to the consultation waiting area (but at
least, few files were lost).
7. The distortions introduced by public tenders, are significant. In
general, the scope and time-scale cannot be changed (the subsequent
criticism of the Ethniks tender refers). This meant that
a. Our suggestions to de-scope the project, to a scale assimilable by the
hospitals, were not accepted.
b. Our attempts to introduce a formal change management programme (not
part of the original tender), were unsuccessful.
c. The functional modules stipulated in the tender were not necessarily
aligned with the achievement of some of the objectives. Offers to
substitute more suitable modules were not accepted.
d. Even a request by the head of Department to retard the project for
budgetary reasons, fell foul of regulations.
e. We were not in a position to influence the turn-around time of other
hardware suppliers, operating under different tenders.
8. The pharmacy system was supposedly the remit of a different vendor,
under another tender, and we did not therefore supply pharmacy management.
9. Half of the laboratories in the province were owned by an external
organisation, with its own information system. The other half were
provincially-owned, but the Department chose not to implement our Lab
modules.
10. All the tendered modules were available at time of purchase. However,
some necessary adaptations took time to complete.
11. It is surprising to learn that management reporting should receive
priority over the basic functioning of the system. Our decision makers
generally still do not have experience in the use of information in
institutional management. Unreliable information is unlikely to further
the cause.
12. Our training programmes (“how-to”) have been variously criticised as
occurring too early or too late. In reality, they occurred whenever
circumstances permitted.

The authors point out that enormous IT investments have had no
discernible effect on the productivity of health professionals.
Paradoxically, such investments continue. Perhaps increased productivity
of professionals is not the primary aim. Absent folie en masse of
healthcare decision makers, one presumes that other benefits accrue.
Measurement has been deucedly difficult – perhaps because of simplistic
yardsticks. In today’s tight financial conditions, many decision-makers
demand a measurable Return-on-Investment (ROI) before committing funds to
an IT project. Yet there are few convincing ways to measure ROI. The
Gartner Group, a respected IT consultancy, now proposes other measures.
Perhaps we in health care are also measuring the wrong things?

Since so many projects fail, perhaps the real question is when to
admit failure, and withdraw funding. There are no easy answers – many
successful complex projects at some point looked like failures. I have
been associated with another large healthcare information system project
in Gauteng Province, South Africa. At the end of the third year, the six
hospitals and five clinics were largely in no better shape than when we
started. In some ways, they were worse off, because of staff attrition.
Yet in the current (fifth) year, there is a significant difference. None
of the hospitals would now agree to the abandonment of the system. When I
recently mooted the discontinuation of nursing care plans (notoriously
difficult to implement successfully), I was greeted with outrage, and told
to improve the functionality instead. The users had already incorporated
usage of the module into a new system of performance measurement.
It is only now that we are getting to grips with information-assisted
management.

Hardly a paragon of success, but indubitably a small victory – made
possible by the tenacity of the client and the project team. Should the
plug have been pulled after year three?

Perhaps the money would have been better spent on aspirin. However,
it could be argued that there is a necessary first step in moving into the
information age. It is instructive to compare the amount wasted here,
with the amount quoted by the authors for one large hospital. Perhaps some
lessons have been learnt, after all.

Competing interests:  
B. Longano was an employee of IBM South Africa from Feb 1996 to Feb 2002. He has a collaborative relationship with IBM and is an expert on the Medicom Health Care Information System.