Evaluation of computer support for prescribing (CAPSULE) using simulated casesBMJ 1997; 315 doi: https://doi.org/10.1136/bmj.315.7111.791 (Published 27 September 1997) Cite this as: BMJ 1997;315:791
- R T Walton (), research fellowa,
- C Gierl, software engineerb,
- P Yudkin, senior research statisticiana,
- H Mistry, research assistanta,
- M P Vessey, head of departmenta,
- J Fox, headb
- a ICRF General Practice Research Group, Department of Public Health and Primary Care, Radcliffe Infirmary, Oxford OX2 GHE
- b ICRF Advanced Computation Laboratory, PO Box 123, London WC2A 3PX
- Correspondence to: Dr R T Walton Bury Knowle Health Centre, Headington, Oxford OX3 9
- Accepted 28 May 1997
Objective: To evaluate the potential effect of computer support on general practitioners' prescribing, and to compare the effectiveness of three different support levels.
Design: Crossover experiment with balanced block design.
Subjects: Random sample of 50 general practitioners (42 agreed to participate) from 165 in a geographically defined area of Oxfordshire.
Interventions: Doctors prescribed for 36 simulated cases constructed from real consultations. Levels of computer support were control (alphabetical list of drugs), limited support (list of preferred drugs), and full support (the same list with explanations available for suggestions).
Main outcome measures: Percentage of cases where doctors ignored a cheaper, equally effective drug; prescribing score (a measure of how closely prescriptions matched expert recommendations); interview to elicit doctors' views of support system.
Results: Computer support significantly improved the quality of prescribing. Doctors ignored a cheaper, equally effective drug in a median 50% (range 25%-75%) of control cases, compared with 36% (8%-67%) with limited support and 35% (0-67%) with full support (P<0.001). The median prescribing score rose from 6.0 units (4.2-7.0) with control support to 6.8 (5.8 to 7.7) and 6.7 (5.6 to 7.8) with limited and full support (P<0.001). Of 41 doctors, 36 (88%) found the system easy to use and 24 (59%) said they would be likely to use it in practice.
Conclusions: Computer support improved compliance with prescribing guidelines, reducing the occasions when doctors ignored a cheaper, equally effective drug. The system was easy to operate, and most participating doctors would be likely to use it in practice.
Effective drug use is an essential part of medical practice, but a large and rapidly expanding body of information is needed to make a rational choice of treatment. Therapeutic decisions are not always based on best evidence.1 The economic importance of prescribing was highlighted by the Audit Commission, which suggested that, although costs were lower in the United Kingdom than in other European countries, savings of nearly £600m each year could be achieved.2
Computer support can improve the quality of general medical care.3 4 5 6 7 8 Benefits have also been seen in patients taking drugs with a narrow therapeutic range—warfarin,9 digoxin,10 and aminophylline.11 Previous research has been in hospitals, and there is no evidence that computers could improve the quality of prescribing in general practice in the United Kingdom. The aim of this study was to examine the potential effects of computer support in general practice and to determine the most effective way of presenting advice.
We used simulated cases to examine doctors' prescribing when aided by computer support. The cases, computer programs, and scoring system were developed and refined in a pilot project.12 Three support levels were compared: (a) an alphabetic list of drugs (control level), (b) a short “pick list” of drugs in recommended order (limited support), and (c) the same pick list with reasons for the recommendations available for inspection (full support).
Simulated patient cases
We constructed the simulated cases from the notes of 43 patients who presented consecutively to three doctors at a health centre during one morning in May 1994. Ninety two problems were identified, of which 67 (73%) were treated with drugs. We selected the first 40 of these 67 problems for the experiment.
The cases were shown to an expert panel comprising two general practitioners with an interest in drug treatment, two clinical pharmacologists, and a pharmacist. The experts met to agree the optimum treatment for the first 20 cases and achieved consensus on the treatment for the remaining cases by correspondence. The experts considered the benefit that might result from treatment, the likelihood of unwanted effects, the presence of contraindications, the patient's preference, and the drug's past effectiveness. Benefits other than the main therapeutic effect were considered (for example, a sedative antidepressant might help a patient with depression and disturbed sleep). Four or five treatments were suggested for each case, in rank order. Consensus was easily reached. In some cases the experts recommended that no drug treatment should be given.
The computer support system was constructed using logic engineering13 at the Imperial Cancer Research Fund Advanced Computation Laboratory, London. A knowledge base of 800 facts allowed the computer to mimic exactly the expert panel's decisions for the 40 cases.12
The top half of the computer screen showed a patient's medical history, social history, presenting problem, and current treatment. (These same data were used by the computer to generate suggestions for treatment.) The bottom half of the screen contained either an alphabetical drug list or the computer support system's suggestions for treatment. In the limited support mode the pick list of drugs was ordered with the most favoured drug at the top. With full support, the same list was given but the doctor had the option of inspecting the computer's reasons for suggesting the drug—for example: “This is a generic drug, recommended by the British National Formulary and in the practice formulary; however, it is relatively expensive.” The doctor could switch easily from support to the alphabetical list of drugs.
Using computer generated random numbers, we selected 50 general practitioners from the 165 practising in the Oxford district. A researcher asked them whether they would take part in an experiment on the effects of computers on prescribing, and they were reassured that their individual prescribing was not under scrutiny. Forty two doctors agreed to participate.
The system was set up on a laptop computer in each doctor's consulting room. The doctors had access to their usual sources of prescribing information, and copies of the British National Formulary and Monthly Index of Medical Specialities (MIMS) were provided. The doctors were familiarised with the computer system by means of four test cases.
Each doctor prescribed for 36 cases, 12 at each level of support. The doctors were also supplied with a summary of each case on paper; the same amount of information was used as is normally stored in general practice computer systems. The three support levels and the three sets of cases were allocated in random order according to a balanced block design (see table 1). The six different orders of allocation were replicated seven times among the 42 doctors. The study design ensured that comparisons between support levels would be independent of any learning effect. A semistructured interview was conducted to elicit the doctor's views about the support system and suggestions for improvements.
For each set of 12 cases, we recorded the percentage of cases in which each doctor ignored a cheaper, equally effective drug; the mean prescribing score; the percentage of cases for which the doctor achieved the maximum possible score; the mean time taken to prescribe; and the percentage of cases in which a generic drug was chosen.
The drugs prescribed were scored independently by two doctors not previously involved with the study. Scoring was blind to the prescribing general practitioner and used a system devised and refined in the pilot study. Scores (which could range from 0 to 8) were allocated according to how closely the treatment agreed with that of the experts (see box). For example, in treatment of skin infections flucloxacillin 500 mg four times daily for one week would score 8 points, while Ceflex 250 mg four times daily for two weeks would score 3 points. We calculated the mean score for each doctor for each set of 12 cases.
Scoring of drug prescriptions
Choice of drug 4—Top of expert panel's list
3—Second on expert panel's list
2—Third on expert panel's list
1—Fourth or lower or not on list but still effective and safe
0—Ineffective or unsafe
Dose and frequency of administration 2—Same as expert panel's
1—Not as good as above but still effective and safe
0—Ineffective or unsafe
Duration of treatment 2—Same as expert panel's
1—Not as good as above but still effective and safe
0—Ineffective or unsafe
Repeatability of scoring was checked by plotting the difference between the two scores against their mean.14 The differences seemed to be normally distributed with a mean (SD) difference of 0.05 (0.41). The lower limit of agreement (two standard deviations from the mean) was −0.77 units, the upper limit was 0.87 units, and 1286 of 1512 cases (85%) lay between these limits.
In the pilot study the mean (SD) percentage of cases in which doctors ignored a cheaper, equally effective drug was 31% (14%) with the alphabetic list of drugs, 20% (8%) with limited support, and 12% (7%) with full support. To detect a difference of 10% (standardised difference 0.71) in the mean percentage between the alphabetical list and either support level (α=0.05, 1-=0.90), a sample size of 42 was required.
We used Friedman's two way analysis of variance to compare the support levels. When these tests were significant, we used the Wilcoxon's matched pairs signed ranking test to compare pairs of groups.
Characteristics of subjects
The doctors who participated in the study were similar in age, sex, and involvement in medical education to those in the area who did not take part.
Types of drugs prescribed.
Table 2 shows the types of drugs prescribed in the simulated cases compared with all drugs prescribed in the practice from which the case reports were taken and with all drugs prescribed by general practitioners in Oxfordshire. The test cases included fewer prescriptions for cardiovascular drugs and more for gastrointestinal and musculoskeletal conditions.
Table 3 shows the outcomes of prescribing for each level of support. With computer support, the proportion of times that doctors ignored a cheaper, equally effective drug fell, the prescribing score rose, and other outcome measures improved. There were no significant differences in outcomes between limited and full support: 17 (40%) of the 42 doctors did not examine the computer's explanations when provided with full support. The median (range) cost of drug treatment for the control cases was £4.33 (£1.33-£10.5), £3.16 (£1.5-£9.38) with limited support, and £4.83 (£1.67-£12.90) with full support (P=0.14).
The order in which the three sets of 12 cases were seen did not affect prescribing outcomes, although the time taken fell when sets were presented second or third (table 4).
Comparison of outcomes for case sets
An unintended fault in the design was that each set of cases was not used equally often at each support level. With full support, the doctors were given case sets 1, 2, and 3 in the ratio 4:1:1; with limited support, the ratio was 1:2:3; and with the control support, the ratio was 1:3:2. Because there was little difference in the cases used to test the limited support system and the control support, it is highly unlikely that the significant differences in outcomes between these two levels could be due to the imbalance. After further detailed analysis we concluded that the trial outcome was not biased.
Of 41 doctors, 24 (59%) said that they would find the prescribing support useful if it was integrated with their practice computer, 24 said they would be likely to use it, 36 (88%) found the system easy to use, and 22 (54%) found the system helpful in making therapeutic choices. Only five of the doctors (12%) found the explanations given by the full support system helpful.
In this study of 42 general practitioners' prescribing practices we have shown the influence that computer support might have. The doctors tended to pick drugs from the computer's list of suggestions, so their prescribing was very similar to the expert recommendations for the same cases. This resulted in an increase in the number of times that the cheapest, most effective drug was used, with increases in generic prescribing. We saw an improvement in the quality of prescribing, and the doctors prescribed more quickly with the decision support system.
Comparison with other studies
Our results compare well with those of other studies of using computers to implement prescribing guidelines. MacDonald found that computer support increased compliance with therapeutic guidelines by 15% in one study3 and by 29% in another.5 White et al showed that computer support for warfarin therapy reduced the length of hospital stay by 35% and reduced the time taken to reach a stable dose by 29%.9
PRODIGY is a computer support system for drug prescribing currently being tested in the United Kingdom.15 The system is similar to CAPSULE but gives support based on a narrower range of patient data (for example, it does not include information about current drug treatment) so the advice is less well tailored to individual patients. Preliminary results show a 0.3% rise in generic prescribing in the practices using the system. Full results from the trial are expected next year. Both CAPSULE and PRODIGY represent attempts to bring the benefits of computer support, previously shown in hospitals, into primary care.
Computer implementation of prescribing guidelines compares well with other methods to improve prescribing. One study in Canadian family practice showed that compliance with guidelines on managing cystitis could be increased by 28%, and for vaginitis by 9%, if the doctors had been involved in producing the guidelines.16 A financial incentive to reduce the use of injectable antibiotics reduced their prescription by 60%.17
Implications for computer support in general practice
Evidence to support computer choices
The simple explanations for the suggested prescriptions that our system generated had no effect on prescribing. This is in keeping with earlier work, which showed that offering bibliographic citations to support computer suggestions made only minor improvements in compliance.6 However, few of the doctors in our study looked at the computer's explanations. This was surprising since the information could be easily obtained with one mouse movement or keystroke. There is evidence that advice from a computer will be more convincing if presented simultaneously with an explanation for that advice.18 Finding the most effective way of presenting the explanation is an important goal for future studies of computer support for prescribing.
Our study relied on expert consensus to produce prescribing guidelines. Systems designed for routine use in general practice should be based on more rigorous methods to produce evidence based guidelines.19 Logic engineering, using the CAPSULE model for making decisions, could use these guidelines to make treatment choices and to give explanations for those choices. Evidence based explanations may be perceived as more authoritative and hence be more effective in changing practice.
Recording of patient details
Using computers to help with routine treatment decisions may substantially influence the details that doctors need to record about their patients. Our software made use of information about current drug treatment and illnesses. There may also be occasions when age and sex are important. Computer support systems that use logic engineering can easily make such decisions and can still produce useful results when data are inaccurate or missing.13 Most of the information necessary for computer support (medical history, current medication, allergies, age, sex, height, weight) is already available on general practice computers. It is essential that computer support systems have access to this information. We simulated this access in our experiment. Systems that require a busy doctor to re-enter data are unlikely to be successful.
However, we found it impossible to reproduce accurately the decisions of the expert panel without taking into account patients' preferences for a particular drug and details of how well it had worked in the past. Information on past efficacy and patient choice are not currently recorded on general practitioners' computer systems. Computer suppliers may need to modify their systems to collect the information; general practitioners will then have to decide whether the improved quality of support is worth the time taken to record it. Entering the diagnosis during the consultation together with the additional data may increase the time needed with each patient. However, our study suggests that the time taken in making the therapeutic decision and printing the prescription is reduced with the CAPSULE program.
Good prescribing is not necessarily cheap. In our test set of cases the expert panel suggested several expensive but highly effective treatments. The general practitioners tended to choose these treatments when they were suggested by the computer, whereas in control situations they used cheaper but less effective drugs. We did not, however, find an overall increase in the cost of the drugs prescribed. This was because, with computer support, the doctors tended to choose a cheaper, equally effective drug whenever possible. This suggests that computer support could improve the quality of prescribing without increasing the cost.
We achieved an excellent response to our invitation to participate in the study, suggesting that the promising results could be generally applicable. However, our results should be interpreted with caution because we used simulated cases and so the effects that we saw may not be reproduced in everyday use. It is important that the effects of decision support systems for prescribing are carefully evaluated both during their development cycle and in real life.20
We thank Drs Karen Crawford and Justin Amery for scoring the cases; Mr Paul Fergusson for managing software development; Drs Mike Sheldon, Jeremy Wyatt, and Helen Doll for advice on study design; Drs Andrew Herxheimer, Tom Jones, and John Reynolds and Mr Danesh Metha for joining the expert panel; and the general practitioners for participating in the study.
Funding: Anglia Oxford Regional Health Authority. RW is supported by a Royal College of General Practitioners/BUPA research training fellowship.
Conflict of interest: The ICRF is commercially developing capsule technology.