BMJ No 7111 Volume 315 Information in practice Saturday 27 September 1997
Evaluation of computer support for prescribing (CAPSULE) using
simulated cases
R T Walton, C Gierl, P Yudkin, H Mistry, M P Vessey, J
Fox
Abstract Objective: To evaluate the potential effect of
computer support on general practitioners' prescribing, and to compare
the effectiveness of three different support levels.
Design: Crossover experiment with balanced block
design.
Subjects: Random sample of 50 general practitioners
(42 agreed to participate) from 165 in a geographically defined area of
Oxfordshire.
Interventions: Doctors prescribed for 36 simulated
cases constructed from real consultations. Levels of computer support
were control (alphabetical list of drugs), limited support (list of
preferred drugs), and full support (the same list with explanations
available for suggestions).
Main outcome measures: Percentage of cases where
doctors ignored a cheaper, equally effective drug; prescribing score (a
measure of how closely prescriptions matched expert recommendations);
interview to elicit doctors' views of support system.
Results: Computer support significantly improved the
quality of prescribing. Doctors ignored a cheaper, equally effective
drug in a median 50% (range 25%-75%) of control cases, compared with
36% (8%-67%) with limited support and 35% (0-67%) with full
support (P<0.001). The median prescribing score rose from 6.0 units
(4.2-7.0) with control support to 6.8 (5.8 to 7.7) and 6.7 (5.6 to 7.8)
with limited and full support (P<0.001). Of 41 doctors, 36 (88%)
found the system easy to use and 24 (59%) said they would be likely to
use it in practice.
Conclusions: Computer support improved compliance
with prescribing guidelines, reducing the occasions when doctors
ignored a cheaper, equally effective drug. The system was easy to
operate, and most participating doctors would be likely to use it in
practice. Introduction
Effective drug use is an essential part of medical practice, but a
large and rapidly expanding body of information is needed to make a
rational choice of treatment. Therapeutic decisions are not always
based on best evidence.(1) The economic importance of
prescribing was highlighted by the Audit Commission, which suggested
that, although costs were lower in the United Kingdom than in other
European countries, savings of nearly £600m each year could be
achieved.(2)
Computer support can improve the quality of general medical
care.(3-8) Benefits have also been seen in patients taking
drugs with a narrow therapeutic range - warfarin,(9)
digoxin,(10) and aminophylline.(11) Previous
research has been in hospitals, and there is no evidence that computers
could improve the quality of prescribing in general practice in the
United Kingdom. The aim of this study was to examine the potential
effects of computer support in general practice and to determine the
most effective way of presenting advice.
| Scoring of drug prescriptions | | Choice of drug | | 4 - Top of expert panel's
list
3 - Second on expert panel's list
2 - Third on expert
panel's list
1 - Fourth or lower or not on list but still
effective and safe
0 - Ineffective or unsafe |
| Dose and frequency of administration | | 2 - Same as
expert panel's
1 - Not as good as above but still effective and
safe
0 - Ineffective or unsafe | | Duration of
treatment | | 2 - Same as expert panel's
1 - Not as good as
above but still effective and safe
0 - Ineffective or unsafe |
Methods
We used simulated cases to examine doctors' prescribing when
aided by computer support. The cases, computer programs, and scoring
system were developed and refined in a pilot project.(12)
Three support levels were compared: (a) an alphabetic list of drugs
(control level), (b) a short "pick list" of drugs in recommended
order (limited support), and (c) the same pick list with reasons for
the recommendations available for inspection (full support).
Simulated patient cases
We constructed the simulated cases from the notes of 43 patients
who presented consecutively to three doctors at a health centre during
one morning in May 1994. Ninety two problems were identified, of which
67 (73%) were treated with drugs. We selected the first 40 of these 67
problems for the experiment.
Recommended prescriptions
The cases were shown to an expert panel comprising two
general practitioners with an interest in drug treatment, two clinical
pharmacologists, and a pharmacist. The experts met to agree the optimum
treatment for the first 20 cases and achieved consensus on the
treatment for the remaining cases by correspondence. The experts
considered the benefit that might result from treatment, the likelihood
of unwanted effects, the presence of contraindications, the patient's
preference, and the drug's past effectiveness. Benefits other than the
main therapeutic effect were considered (for example, a sedative
antidepressant might help a patient with depression and disturbed
sleep). Four or five treatments were suggested for each case, in rank
order. Consensus was easily reached. In some cases the experts
recommended that no drug treatment should be given.
Prescribing support
The computer support system was constructed using logic
engineering(13) at the Imperial Cancer Research Fund
Advanced Computation Laboratory, London. A knowledge base of 800 facts
allowed the computer to mimic exactly the expert panel's decisions for
the 40 cases.(12)
The top half of the computer screen showed a patient's medical
history, social history, presenting problem, and current treatment.
(These same data were used by the computer to generate suggestions for
treatment.) The bottom half of the screen contained either an
alphabetical drug list or the computer support system's suggestions
for treatment. In the limited support mode the pick list of drugs was
ordered with the most favoured drug at the top. With full support, the
same list was given but the doctor had the option of inspecting the
computer's reasons for suggesting the drug - for example: "This is a
generic drug, recommended by the British National
Formulary and in the practice formulary; however, it is
relatively expensive." The doctor could switch easily from support to
the alphabetical list of drugs.
Participants
Using computer generated random numbers, we selected 50 general
practitioners from the 165 practising in the Oxford district. A
researcher asked them whether they would take part in an experiment on
the effects of computers on prescribing, and they were reassured that
their individual prescribing was not under scrutiny. Forty two doctors
agreed to participate.
Intervention
The system was set up on a laptop computer in each doctor's
consulting room. The doctors had access to their usual sources of
prescribing information, and copies of the British National
Formulary and Monthly Index of Medical Specialities
(MIMS) were provided. The doctors were familiarised with the
computer system by means of four test cases.
Study design
Each doctor prescribed for 36 cases, 12 at each level of support.
The doctors were also supplied with a summary of each case on paper;
the same amount of information was used as is normally stored in
general practice computer systems. The three support levels and the
three sets of cases were allocated in random order according to a
balanced block design (see table 1). The six different orders of
allocation were replicated seven times among the 42 doctors. The
study design ensured that comparisons between support levels would be
independent of any learning effect. A semistructured interview was
conducted to elicit the doctor's views about the support system and
suggestions for improvements.
| Table 1 - Order of allocation of the three sets of 12
cases and the three levels of computer support (the six different
orders of allocation were each replicated seven times among the 42
doctors) |
| Order | First
set | Second
set | Third set
|
|---|
| Level of support | Set of cases
| Level of support | Set of
cases | Level of support | Set of cases
|
| 1 | Full | Set
1 | Limited | Set
2 | Control* | Set 3
|
| 2 | Control* | Set
1 | Limited | Set
3 | Full | Set 2
|
| 3 | Control* | Set
2 | Full | Set
1 | Limited | Set 3
|
| 4 | Limited | Set
2 | Control* | Set
3 | Full | Set 1
|
| 5 | Limited | Set
3 | Full | Set
1 | Control* | Set 2
|
| 6 | Full | Set
3 | Control* | Set
2 | Limited | Set 1 |
| *Alphabetical listing of drugs. |
Outcome measures
For each set of 12 cases, we recorded the percentage of cases in
which each doctor ignored a cheaper, equally effective drug; the mean
prescribing score; the percentage of cases for which the doctor
achieved the maximum possible score; the mean time taken to prescribe;
and the percentage of cases in which a generic drug was chosen.
The drugs prescribed were scored independently by two doctors not
previously involved with the study. Scoring was blind to the
prescribing general practitioner and used a system devised and refined
in the pilot study. Scores (which could range from 0 to 8) were
allocated according to how closely the treatment agreed with that of
the experts (see box). For example, in treatment of skin infections
flucloxacillin 500 mg four times daily for one week would score 8
points, while Ceflex 250 mg four times daily for two weeks would score
3 points. We calculated the mean score for each doctor for each set of
12 cases.
Repeatability of scoring was checked by plotting the difference between
the two scores against their mean.(14) The differences
seemed to be normally distributed with a mean (SD) difference of 0.05
(0.41). The lower limit of agreement (two standard deviations from the
mean) was -0.77 units, the upper limit was 0.87 units, and 1286 of
1512 cases (85%) lay between these limits.
Statistical analysis
In the pilot study the mean (SD) percentage of cases in which
doctors ignored a cheaper, equally effective drug was 31% (14%) with
the alphabetic list of drugs, 20% (8%) with limited support, and 12%
(7%) with full support. To detect a difference of 10% (standardised
difference 0.71) in the mean percentage between the alphabetical list
and either support level (alpha=0.05, 1-beta=0.90), a sample size of 42
was required.
We used Friedman's two way analysis of variance to compare the
support levels. When these tests were significant, we used the
Wilcoxon's matched pairs signed ranking test to compare pairs of
groups. Results
Characteristics of subjects
The doctors who participated in the study were similar in age,
sex, and involvement in medical education to those in the area who did
not take part.
Types of drugs prescribed.
Table 2 shows the types of drugs prescribed in the simulated cases
compared with all drugs prescribed in the practice from which the case
reports were taken and with all drugs prescribed by general
practitioners in Oxfordshire. The test cases included fewer
prescriptions for cardiovascular drugs and more for gastrointestinal
and musculoskeletal conditions.
| Table 2 - No (%) of different types of drug prescribed
by the 42 doctors for the test cases compared with numbers of
prescriptions in the practice from which the cases were taken and with
regional average for general practitioners in Oxfordshire |
| Drug type* |
Test cases |
Source
general practice (1994 prescribing) |
Average Oxfordshire
general practice† |
| Cardiovascular
system | 1 (3) | 1,343
(17) | 1,381 (16) |
| Central nervous
system | 5 (14) | 1,240
(16) | 1,376 (16) |
| Respiratory system | 2
(6) | 633 (8) | 765 (9) |
| Gastrointestinal
system | 7 (19) | 558 (7) | 651
(8) |
| Endocrine system | 1 (3) | 384
(5) | 500 (6) |
| Musculoskeletal | 5
(14) | 295 (4) | 429 (5) |
| All
other | 15 (42) | 3,424 (43) | 3,279
(39)
|
| Total | 36 | 7,877 | 8,381 |
*According to classification in British National
Formulary.
†Estimate based on a notional practice of the same size and with
the same age and sex distribution as source general practice. Data from
Prescription Pricing Authority's PACT report Bury Knowle Health
Centre 1994 Sept-Dec. |
Prescribing outcomes
Table 3 shows the outcomes of prescribing for each level of
support. With computer support, the proportion of times that doctors
ignored a cheaper, equally effective drug fell, the prescribing score
rose, and other outcome measures improved. There were no significant
differences in outcomes between limited and full support: 17 (40%) of
the 42 doctors did not examine the computer's explanations when
provided with full support. The median (range) cost of drug treatment
for the control cases was £4.33 (£1.33-£10.5), £3.16 (£1.5-£9.38)
with limited support, and £4.83 (£1.67-£12.90) with full support
(P=0.14).
| Table 3 - Median (range) outcome measures for 42
doctors' prescribing with different levels of computer support
|
| Outcome measure | Level of
support | Difference (P value)
|
|---|
| Control* | Limited
support | Full support |
| Ignored cheaper,
equally effective substitute
(%) | 50 (25 to 75) | 36
(8 to 67) | 35 (0 to 67) | <0.001
|
| Prescribing score (units) | 6.0 (4.2 to
7.0) | 6.8 (5.8 to 7.7) | 6.7 (5.6 to
7.8) | <0.001 |
| Same prescription as experts
(%) | 25 (0 to 50) | 42 (17 to
75) | 42 (17 to 83) | <0.001 |
| Time taken
(seconds) | 66 (43 to 113) | 53 (18 to
104) | 56 (23 to 105) | <0.001 |
| Generic
drug prescribed (%) | 75 (27 to 100) | 100
(73 to 100) | 100 (63 to 100) | <0.001 |
| *Alphabetical listing of drugs. |
The order in which the three sets of 12 cases were seen did not affect
prescribing outcomes, although the time taken fell when sets were
presented second or third (table
4). | Table 4 - Median (range) outcome measures for 42
doctors' prescribing by order of administration of test cases |
| Outcome measure | Order of
administration of cases | Difference (P value)
|
|---|
| First set | Second set | Third
set |
| Ignored cheaper, equally effective substitute
(%) | 43 (8 to
70) | 40 (0 to 75) | 38 (8 to
75) | 0.41 |
| Prescribing score
(units) | 6.5 (5.6 to 7.5) | 6.5 (4.8 to
7.7) | 6.6 (4.2 to 7.8) | 0.57 |
| Same
prescription as experts (%) | 33 (0 to
67) | 33 (8 to 75) | 42 (0 to
83) | 0.75 |
| Time taken (seconds) | 64
(33 to 113) | 58 (23 to 91) | 57 (18 to
92) | 0.002 |
| Generic drug prescribed
(%) | 91 (27 to 100) | 90 (44 to
100) | 91 (36 to 100) | 0.55 |
Comparison of outcomes for case sets
An unintended fault in the design was that each set of cases was
not used equally often at each support level. With full support, the
doctors were given case sets 1, 2, and 3 in the ratio 4:1:1; with
limited support, the ratio was 1:2:3; and with the control support, the
ratio was 1:3:2. Because there was little difference in the cases used
to test the limited support system and the control support, it is
highly unlikely that the significant differences in outcomes between
these two levels could be due to the imbalance. After further detailed
analysis we concluded that the trial outcome was not biased.
Semistructured interview
Of 41 doctors, 24 (59%) said that they would find the
prescribing support useful if it was integrated with their
practice computer, 24 said they would be likely to use it, 36 (88%)
found the system easy to use, and 22 (54%) found the system helpful in
making therapeutic choices. Only five of the doctors (12%) found the
explanations given by the full support system helpful. Discussion
In this study of 42 general practitioners' prescribing practices
we have shown the influence that computer support might have. The
doctors tended to pick drugs from the computer's list of suggestions,
so their prescribing was very similar to the expert recommendations for
the same cases. This resulted in an increase in the number of times
that the cheapest, most effective drug was used, with increases in
generic prescribing. We saw an improvement in the quality of
prescribing, and the doctors prescribed more quickly with the
decision support system.
Comparison with other studies
Our results compare well with those of other studies of using
computers to implement prescribing guidelines. MacDonald found that
computer support increased compliance with therapeutic guidelines by
15% in one study(3) and by 29% in another.(5)
White et al showed that computer support for warfarin therapy reduced
the length of hospital stay by 35% and reduced the time taken to reach
a stable dose by 29%.(9)
PRODIGY is a computer support system for drug prescribing currently
being tested in the United Kingdom.(15) The system
is similar to CAPSULE but gives support based on a narrower range of
patient data (for example, it does not include information about
current drug treatment) so the advice is less well tailored to
individual patients. Preliminary results show a 0.3% rise in generic
prescribing in the practices using the system. Full results from the
trial are expected next year. Both CAPSULE and PRODIGY represent
attempts to bring the benefits of computer support, previously shown in
hospitals, into primary care.
Computer implementation of prescribing guidelines compares well
with other methods to improve prescribing. One study in Canadian family
practice showed that compliance with guidelines on managing cystitis
could be increased by 28%, and for vaginitis by 9%, if the
doctors had been involved in producing the
guidelines.(16) A financial incentive to reduce the use of
injectable antibiotics reduced their prescription by
60%.(17)
Implications for computer support in general practice Evidence to support computer choices
The simple explanations for the suggested prescriptions that our
system generated had no effect on prescribing. This is in keeping with
earlier work, which showed that offering bibliographic citations to
support computer suggestions made only minor improvements in
compliance.(6) However, few of the doctors in our study
looked at the computer's explanations. This was surprising since the
information could be easily obtained with one mouse movement or
keystroke. There is evidence that advice from a computer will be more
convincing if presented simultaneously with an explanation for that
advice.(18) Finding the most effective way of presenting the
explanation is an important goal for future studies of computer support
for prescribing.
Our study relied on expert consensus to produce prescribing guidelines.
Systems designed for routine use in general practice should be based on
more rigorous methods to produce evidence based
guidelines.(19) Logic engineering, using the CAPSULE model
for making decisions, could use these guidelines to make treatment
choices and to give explanations for those choices. Evidence based
explanations may be perceived as more authoritative and hence be more
effective in changing practice.
Recording of patient details
Using computers to help with routine treatment decisions may
substantially influence the details that doctors need to record about
their patients. Our software made use of information about current drug
treatment and illnesses. There may also be occasions when age and sex
are important. Computer support systems that use logic engineering can
easily make such decisions and can still produce useful results when
data are inaccurate or missing.(13) Most of the information
necessary for computer support (medical history, current medication,
allergies, age, sex, height, weight) is already available on general
practice computers. It is essential that computer support systems have
access to this information. We simulated this access in our experiment.
Systems that require a busy doctor to re-enter data are unlikely to be
successful.
However, we found it impossible to reproduce accurately the decisions
of the expert panel without taking into account patients' preferences
for a particular drug and details of how well it had worked in the
past. Information on past efficacy and patient choice are not currently
recorded on general practitioners' computer systems. Computer
suppliers may need to modify their systems to collect the information;
general practitioners will then have to decide whether the improved
quality of support is worth the time taken to record it. Entering the
diagnosis during the consultation together with the additional data may
increase the time needed with each patient. However, our study suggests
that the time taken in making the therapeutic decision and printing the
prescription is reduced with the CAPSULE program.
Prescription costs
Good prescribing is not necessarily cheap. In our test set of
cases the expert panel suggested several expensive but highly effective
treatments. The general practitioners tended to choose these treatments
when they were suggested by the computer, whereas in control situations
they used cheaper but less effective drugs. We did not, however, find
an overall increase in the cost of the drugs prescribed. This was
because, with computer support, the doctors tended to choose a cheaper,
equally effective drug whenever possible. This suggests that computer
support could improve the quality of prescribing without increasing the
cost.
Conclusion
We achieved an excellent response to our invitation to participate
in the study, suggesting that the promising results could be generally
applicable. However, our results should be interpreted with caution
because we used simulated cases and so the effects that we saw may not
be reproduced in everyday use. It is important that the effects of
decision support systems for prescribing are carefully evaluated both
during their development cycle and in real life.(20)
We thank Drs Karen Crawford and Justin Amery for scoring the
cases; Mr Paul Fergusson for managing software development; Drs Mike
Sheldon, Jeremy Wyatt, and Helen Doll for advice on study design; Drs
Andrew Herxheimer, Tom Jones, and John Reynolds and Mr Danesh Metha for
joining the expert panel; and the general practitioners for
participating in the study.
Funding: Anglia Oxford Regional Health Authority. RW is
supported by a Royal College of General Practitioners/BUPA research
training fellowship.
Conflict of interest: The ICRF is commercially developing
capsule technology.
(Accepted 28 May 1997)
ICRF General Practice Research Group,
Department of
Public Health and Primary Care,
Radcliffe Infirmary,
Oxford OX2
GHE
R T Walton, research
fellow
P Yudkin, senior research
statistician
H Mistry, research
assistant
M P Vessey, head of
department
ICRF Advanced Computation Laboratory,
PO
Box 123,
London WC2A 3PX
C Gierl,
software engineer
J
Fox, head
Correspondence to: Dr R T
Walton
Bury Knowle Health Centre,
Headington,
Oxford OX3
9JA
robert.walton@public-health.oxford.ac.uk
References
1 Antman E M, Lau J, Kupelnick B, Mosteller F, Chalmers T C. A
comparison of results of meta-analyses of randomised control trials and
recommendations of clinical experts. Treatments for myocardial
infarction. JAMA 1992;268:240-8.
2 Jones I, Griffiths S. A prescription for improvement.
Towards more rational prescribing in general practice. London:
Audit Commission, HMSO, 1994.
3 McDonald C J. Protocol-based computer reminders, the quality of
care and the non-perfectibility of man. N Engl J Med
1976;295:1351-5.
4 McDonald C J, Tierney W M, Martin D K, Overhage J M, Day Z. The
Regenstrief medical record: 1991 a campus-wide system. Proc Annu
Symp Comput Appl Med Care 1991;1991:925-8.
5 McDonald C J. Use of a computer to detect and respond to
clinical events: its effect on clinical behaviour. Ann Intern
Med 1976;84:162-7.
6 McDonald C J, Wilson G A, McCabe G J. Physician response to
computer reminders. JAMA 1980;244:1579-81.
7 McDonald C J, Hui S L, Smith D M, Tierney W M, Cohen S J,
Weinberger M, et al. Reminders to physicians from an introspective
computer medical record. A two-year randomised trial. Ann Intern
Med 1984;100:130-8.
8 McDonald C J, Hammond W E. Standard formats for electronic
transfer of clinical data. Ann Intern Med
1989;110:333-5.
9 White R H, Hong R, Venook A P, Daschbach M M, Murray W, Mungall
D R, et al. Initiation of warfarin therapy: comparison of physician
dosing with computer-assisted dosing. J Gen Intern Med
1987;2(3):141-8.
10 White K S, Lindsay A, Pryor T A, Brown W F, Walsh K. Application
of a computerised medical decision-making process to the problem of
digoxin intoxication. J Am Coll Cardiol 1984;4:571-6.
11 Gonzalez E R, Vanderheyden B A, Ornato J P, Comstock T G.
Computer-assisted optimization of aminophylline therapy in the
emergency department. Am J Emergency Med 1989;7:395-401.
12 Walton R. An evaluation of CAPSULE, a computer system giving
advice to general practitioners about prescribing drugs. J Inf
Primary Care 1996 March:2-7.
13 Fox J, Glowinski A, Hajnal S, O'Niel M. Logic engineering for
knowledge engineering: design and implementation of the Oxford system
of medicine. Artif Intell Med 1990;2:323-39.
14 Bland J M, Altman D. Statistical methods for assessing agreement
between two methods of clinical measurement. Lancet
1986;i:307-10.
15 Wise J. Computer prescribing scheme gets green light.
BMJ 1996;313:250.
16 Norton P, Dempsey L. Self audit: its effects on quality of
care. J Fam Pract 1985;21:289-91.
17 Brook R, Williams K. Effect of medical care review on the use
ofinjections: a study of the New Mexico Experimental Care Review
Organisation. Ann Intern Med 1976;85:509-15.
18 Sotos J. MYCIN and NEOMYCIN: two approaches to generating
explanations in rule based expert systems. Aviat Space Environ
Med 1990;61:950-4.
19 Grimshaw J, Russell I. Achieving health gain through clinical
guidelines. I: Developing scientifically valid guidelines. Qual
Health Care 1993;2:243-8.
20 Wyatt J, Spiegelhalter D. Evaluating medical expert systems:
what to test and how? Med Inf 1990;15:205-17.
Home | Current issue | Past issues |
Classified ads | Career Focus | Feedback
Collections |
About this site | About the BMJ | BMA | Medline
|
