Information in practice

Evaluation of computer support for prescribing (CAPSULE) using simulated cases

^a ICRF General Practice Research Group, Department of Public Health and Primary Care, Radcliffe Infirmary, Oxford OX2 GHE, ^b ICRF Advanced Computation Laboratory, PO Box 123, London WC2A 3PX

Correspondence to: Dr R T Walton Bury Knowle Health Centre, Headington, Oxford OX3 9 JArobert.walton@public-health.oxford.ac.uk

	Abstract

Top Abstract Introduction Methods Results Discussion References

Objective: To evaluate the potential effect of computer support on general practitioners' prescribing, and to compare the effectiveness of three different support levels.
Design: Crossover experiment with balanced block design.
Subjects: Random sample of 50 general practitioners (42 agreed to participate) from 165 in a geographically defined area of Oxfordshire.
Interventions: Doctors prescribed for 36 simulated cases constructed from real consultations. Levels of computer support were control (alphabetical list of drugs), limited support (list of preferred drugs), and full support (the same list with explanations available for suggestions).
Main outcome measures: Percentage of cases where doctors ignored a cheaper, equally effective drug; prescribing score (a measure of how closely prescriptions matched expert recommendations); interview to elicit doctors' views of support system.
Results: Computer support significantly improved the quality of prescribing. Doctors ignored a cheaper, equally effective drug in a median 50% (range 25%-75%) of control cases, compared with 36% (8%-67%) with limited support and 35% (0-67%) with full support (P<0.001). The median prescribing score rose from 6.0 units (4.2-7.0) with control support to 6.8 (5.8 to 7.7) and 6.7 (5.6 to 7.8) with limited and full support (P<0.001). Of 41 doctors, 36 (88%) found the system easy to use and 24 (59%) said they would be likely to use it in practice.
Conclusions: Computer support improved compliance with prescribing guidelines, reducing the occasions when doctors ignored a cheaper, equally effective drug. The system was easy to operate, and most participating doctors would be likely to use it in practice.

	Introduction

Top Abstract Introduction Methods Results Discussion References

Effective drug use is an essential part of medical practice, but a large and rapidly expanding body of information is needed to make a rational choice of treatment. Therapeutic decisions are not always based on best evidence.¹ The economic importance of prescribing was highlighted by the Audit Commission, which suggested that, although costs were lower in the United Kingdom than in other European countries, savings of nearly £600m each year could be achieved.²

Computer support can improve the quality of general medical care.^{3 4 5 6 7 8} Benefits have also been seen in patients taking drugs with a narrow therapeutic range—warfarin,⁹ digoxin,¹⁰ and aminophylline.¹¹ Previous research has been in hospitals, and there is no evidence that computers could improve the quality of prescribing in general practice in the United Kingdom. The aim of this study was to examine the potential effects of computer support in general practice and to determine the most effective way of presenting advice.

	Methods

Top Abstract Introduction Methods Results Discussion References

We used simulated cases to examine doctors' prescribing when aided by computer support. The cases, computer programs, and scoring system were developed and refined in a pilot project.¹² Three support levels were compared: (a) an alphabetic list of drugs (control level), (b) a short "pick list" of drugs in recommended order (limited support), and (c) the same pick list with reasons for the recommendations available for inspection (full support).

Simulated patient cases
We constructed the simulated cases from the notes of 43 patients who presented consecutively to three doctors at a health centre during one morning in May 1994. Ninety two problems were identified, of which 67 (73%) were treated with drugs. We selected the first 40 of these 67 problems for the experiment.

Recommended prescriptions
The cases were shown to an expert panel comprising two general practitioners with an interest in drug treatment, two clinical pharmacologists, and a pharmacist. The experts met to agree the optimum treatment for the first 20 cases and achieved consensus on the treatment for the remaining cases by correspondence. The experts considered the benefit that might result from treatment, the likelihood of unwanted effects, the presence of contraindications, the patient's preference, and the drug's past effectiveness. Benefits other than the main therapeutic effect were considered (for example, a sedative antidepressant might help a patient with depression and disturbed sleep). Four or five treatments were suggested for each case, in rank order. Consensus was easily reached. In some cases the experts recommended that no drug treatment should be given.

Prescribing support
The computer support system was constructed using logic engineering¹³ at the Imperial Cancer Research Fund Advanced Computation Laboratory, London. A knowledge base of 800 facts allowed the computer to mimic exactly the expert panel's decisions for the 40 cases.¹²

The top half of the computer screen showed a patient's medical history, social history, presenting problem, and current treatment. (These same data were used by the computer to generate suggestions for treatment.) The bottom half of the screen contained either an alphabetical drug list or the computer support system's suggestions for treatment. In the limited support mode the pick list of drugs was ordered with the most favoured drug at the top. With full support, the same list was given but the doctor had the option of inspecting the computer's reasons for suggesting the drug—for example: "This is a generic drug, recommended by the British National Formulary and in the practice formulary; however, it is relatively expensive." The doctor could switch easily from support to the alphabetical list of drugs.

Participants
Using computer generated random numbers, we selected 50 general practitioners from the 165 practising in the Oxford district. A researcher asked them whether they would take part in an experiment on the effects of computers on prescribing, and they were reassured that their individual prescribing was not under scrutiny. Forty two doctors agreed to participate.

Intervention
The system was set up on a laptop computer in each doctor's consulting room. The doctors had access to their usual sources of prescribing information, and copies of the British National Formulary and Monthly Index of Medical Specialities (MIMS) were provided. The doctors were familiarised with the computer system by means of four test cases.

Study design
Each doctor prescribed for 36 cases, 12 at each level of support. The doctors were also supplied with a summary of each case on paper; the same amount of information was used as is normally stored in general practice computer systems. The three support levels and the three sets of cases were allocated in random order according to a balanced block design (see table 1). The six different orders of allocation were replicated seven times among the 42 doctors. The study design ensured that comparisons between support levels would be independent of any learning effect. A semistructured interview was conducted to elicit the doctor's views about the support system and suggestions for improvements.

Outcome measures
For each set of 12 cases, we recorded the percentage of cases in which each doctor ignored a cheaper, equally effective drug; the mean prescribing score; the percentage of cases for which the doctor achieved the maximum possible score; the mean time taken to prescribe; and the percentage of cases in which a generic drug was chosen.

The drugs prescribed were scored independently by two doctors not previously involved with the study. Scoring was blind to the prescribing general practitioner and used a system devised and refined in the pilot study. Scores (which could range from 0 to 8) were allocated according to how closely the treatment agreed with that of the experts (see box). For example, in treatment of skin infections flucloxacillin 500 mg four times daily for one week would score 8 points, while Ceflex 250 mg four times daily for two weeks would score 3 points. We calculated the mean score for each doctor for each set of 12 cases.

Repeatability of scoring was checked by plotting the difference between the two scores against their mean.¹⁴ The differences seemed to be normally distributed with a mean (SD) difference of 0.05 (0.41). The lower limit of agreement (two standard deviations from the mean) was -0.77 units, the upper limit was 0.87 units, and 1286 of 1512 cases (85%) lay between these limits.

Statistical analysis
In the pilot study the mean (SD) percentage of cases in which doctors ignored a cheaper, equally effective drug was 31% (14%) with the alphabetic list of drugs, 20% (8%) with limited support, and 12% (7%) with full support. To detect a difference of 10% (standardised difference 0.71) in the mean percentage between the alphabetical list and either support level (=0.05, 1-ß=0.90), a sample size of 42 was required.

We used Friedman's two way analysis of variance to compare the support levels. When these tests were significant, we used the Wilcoxon's matched pairs signed ranking test to compare pairs of groups.

	Results

Top Abstract Introduction Methods Results Discussion References

Characteristics of subjects
The doctors who participated in the study were similar in age, sex, and involvement in medical education to those in the area who did not take part.

Types of drugs prescribed.
Table 2 shows the types of drugs prescribed in the simulated cases compared with all drugs prescribed in the practice from which the case reports were taken and with all drugs prescribed by general practitioners in Oxfordshire. The test cases included fewer prescriptions for cardiovascular drugs and more for gastrointestinal and musculoskeletal conditions.

Prescribing outcomes
Table 3 shows the outcomes of prescribing for each level of support. With computer support, the proportion of times that doctors ignored a cheaper, equally effective drug fell, the prescribing score rose, and other outcome measures improved. There were no significant differences in outcomes between limited and full support: 17 (40%) of the 42 doctors did not examine the computer's explanations when provided with full support. The median (range) cost of drug treatment for the control cases was £4.33 (£1.33-£10.5), £3.16 (£1.5-£9.38) with limited support, and £4.83 (£1.67-£12.90) with full support (P=0.14).

The order in which the three sets of 12 cases were seen did not affect prescribing outcomes, although the time taken fell when sets were presented second or third (table 4).

Comparison of outcomes for case sets
An unintended fault in the design was that each set of cases was not used equally often at each support level. With full support, the doctors were given case sets 1, 2, and 3 in the ratio 4:1:1; with limited support, the ratio was 1:2:3; and with the control support, the ratio was 1:3:2. Because there was little difference in the cases used to test the limited support system and the control support, it is highly unlikely that the significant differences in outcomes between these two levels could be due to the imbalance. After further detailed analysis we concluded that the trial outcome was not biased.

Semistructured interview
Of 41 doctors, 24 (59%) said that they would find the prescribing support useful if it was integrated with their practice computer, 24 said they would be likely to use it, 36 (88%) found the system easy to use, and 22 (54%) found the system helpful in making therapeutic choices. Only five of the doctors (12%) found the explanations given by the full support system helpful.

	Discussion

Top Abstract Introduction Methods Results Discussion References

In this study of 42 general practitioners' prescribing practices we have shown the influence that computer support might have. The doctors tended to pick drugs from the computer's list of suggestions, so their prescribing was very similar to the expert recommendations for the same cases. This resulted in an increase in the number of times that the cheapest, most effective drug was used, with increases in generic prescribing. We saw an improvement in the quality of prescribing, and the doctors prescribed more quickly with the decision support system.

Comparison with other studies
Our results compare well with those of other studies of using computers to implement prescribing guidelines. MacDonald found that computer support increased compliance with therapeutic guidelines by 15% in one study³ and by 29% in another.⁵ White et al showed that computer support for warfarin therapy reduced the length of hospital stay by 35% and reduced the time taken to reach a stable dose by 29%.⁹

PRODIGY is a computer support system for drug prescribing currently being tested in the United Kingdom.¹⁵ The system is similar to CAPSULE but gives support based on a narrower range of patient data (for example, it does not include information about current drug treatment) so the advice is less well tailored to individual patients. Preliminary results show a 0.3% rise in generic prescribing in the practices using the system. Full results from the trial are expected next year. Both CAPSULE and PRODIGY represent attempts to bring the benefits of computer support, previously shown in hospitals, into primary care.

Computer implementation of prescribing guidelines compares well with other methods to improve prescribing. One study in Canadian family practice showed that compliance with guidelines on managing cystitis could be increased by 28%, and for vaginitis by 9%, if the doctors had been involved in producing the guidelines.¹⁶ A financial incentive to reduce the use of injectable antibiotics reduced their prescription by 60%.¹⁷

Implications for computer support in general practice
Evidence to support computer choices
The simple explanations for the suggested prescriptions that our system generated had no effect on prescribing. This is in keeping with earlier work, which showed that offering bibliographic citations to support computer suggestions made only minor improvements in compliance.⁶ However, few of the doctors in our study looked at the computer's explanations. This was surprising since the information could be easily obtained with one mouse movement or keystroke. There is evidence that advice from a computer will be more convincing if presented simultaneously with an explanation for that advice.¹⁸ Finding the most effective way of presenting the explanation is an important goal for future studies of computer support for prescribing.

Our study relied on expert consensus to produce prescribing guidelines. Systems designed for routine use in general practice should be based on more rigorous methods to produce evidence based guidelines.¹⁹ Logic engineering, using the CAPSULE model for making decisions, could use these guidelines to make treatment choices and to give explanations for those choices. Evidence based explanations may be perceived as more authoritative and hence be more effective in changing practice.

Recording of patient details
Using computers to help with routine treatment decisions may substantially influence the details that doctors need to record about their patients. Our software made use of information about current drug treatment and illnesses. There may also be occasions when age and sex are important. Computer support systems that use logic engineering can easily make such decisions and can still produce useful results when data are inaccurate or missing.¹³ Most of the information necessary for computer support (medical history, current medication, allergies, age, sex, height, weight) is already available on general practice computers. It is essential that computer support systems have access to this information. We simulated this access in our experiment. Systems that require a busy doctor to re-enter data are unlikely to be successful.

However, we found it impossible to reproduce accurately the decisions of the expert panel without taking into account patients' preferences for a particular drug and details of how well it had worked in the past. Information on past efficacy and patient choice are not currently recorded on general practitioners' computer systems. Computer suppliers may need to modify their systems to collect the information; general practitioners will then have to decide whether the improved quality of support is worth the time taken to record it. Entering the diagnosis during the consultation together with the additional data may increase the time needed with each patient. However, our study suggests that the time taken in making the therapeutic decision and printing the prescription is reduced with the CAPSULE program.

Prescription costs
Good prescribing is not necessarily cheap. In our test set of cases the expert panel suggested several expensive but highly effective treatments. The general practitioners tended to choose these treatments when they were suggested by the computer, whereas in control situations they used cheaper but less effective drugs. We did not, however, find an overall increase in the cost of the drugs prescribed. This was because, with computer support, the doctors tended to choose a cheaper, equally effective drug whenever possible. This suggests that computer support could improve the quality of prescribing without increasing the cost.

Conclusion
We achieved an excellent response to our invitation to participate in the study, suggesting that the promising results could be generally applicable. However, our results should be interpreted with caution because we used simulated cases and so the effects that we saw may not be reproduced in everyday use. It is important that the effects of decision support systems for prescribing are carefully evaluated both during their development cycle and in real life.²⁰

	Acknowledgements

We thank Drs Karen Crawford and Justin Amery for scoring the cases; Mr Paul Fergusson for managing software development; Drs Mike Sheldon, Jeremy Wyatt, and Helen Doll for advice on study design; Drs Andrew Herxheimer, Tom Jones, and John Reynolds and Mr Danesh Metha for joining the expert panel; and the general practitioners for participating in the study.

Funding: Anglia Oxford Regional Health Authority. RW is supported by a Royal College of General Practitioners/BUPA research training fellowship.

Conflict of interest: The ICRF is commercially developing capsule technology.

	References

Top Abstract Introduction Methods Results Discussion References

1. Antman EM, Lau J, Kupelnick B, Mosteller F, Chalmers TC. A comparison of results of meta-analyses of randomised control trials and recommendations of clinical experts. Treatments for myocardial infarction. JAMA 1992;268:240-8. [Abstract/Free Full Text]
2. Jones I, Griffiths S. A prescription for improvement. Towards more rational prescribing in general practice. London: Audit Commission, HMSO, 1994.
3. McDonald CJ. Protocol-based computer reminders, the quality of care and the non-perfectibility of man. N Engl J Med 1976;295:1351-5.
4. McDonald CJ, Tierney WM, Martin DK, Overhage JM, Day Z. The Regenstrief medical record: 1991 a campus-wide system. Proc Annu Symp Comput Appl Med Care 1991;1991:925-8.
5. McDonald CJ. Use of a computer to detect and respond to clinical events: its effect on clinical behaviour. Ann Intern Med 1976;84:162-7.
6. McDonald CJ, Wilson GA, McCabe GJ. Physician response to computer reminders. JAMA 1980;244:1579-81.
7. McDonald CJ, Hui SL, Smith DM, Tierney WM, Cohen SJ, Weinberger M, et al. Reminders to physicians from an introspective computer medical record. A two-year randomised trial. Ann Intern Med 1984;100:130-8.
8. McDonald CJ, Hammond WE. Standard formats for electronic transfer of clinical data. Ann Intern Med 1989;110:333-5.
9. White RH, Hong R, Venook AP, Daschbach MM, Murray W, Mungall DR, et al. Initiation of warfarin therapy: comparison of physician dosing with computer-assisted dosing. J Gen Intern Med 1987;2(3):141-8.
10. White KS, Lindsay A, Pryor TA, Brown WF, Walsh K. Application of a computerised medical decision-making process to the problem of digoxin intoxication. J Am Coll Cardiol 1984;4:571-6.
11. Gonzalez ER, Vanderheyden BA, Ornato JP, Comstock TG. Computer-assisted optimization of aminophylline therapy in the emergency department. Am J Emergency Med 1989;7:395-401.
12. Walton R. An evaluation of CAPSULE, a computer system giving advice to general practitioners about prescribing drugs. J Inf Primary Care 1996 March:2-7.
13. Fox J, Glowinski A, Hajnal S, O'Niel M. Logic engineering for knowledge engineering: design and implementation of the Oxford system of medicine. Artif Intell Med 1990;2:323-39.
14. Bland JM, Altman D. Statistical methods for assessing agreement between two methods of clinical measurement. Lancet 1986;i:307-10.
15. Wise J. Computer prescribing scheme gets green light. BMJ 1996;313:250. [Free Full Text]
16. Norton P, Dempsey L. Self audit: its effects on quality of care. J Fam Pract 1985;21:289-91. [Medline]
17. Brook R, Williams K. Effect of medical care review on the use of injections: a study of the New Mexico Experimental Care Review Organisation. Ann Intern Med 1976;85:509-15.
18. Sotos J. MYCIN and NEOMYCIN: two approaches to generating explanations in rule based expert systems. Aviat Space Environ Med 1990;61:950-4.
19. Grimshaw J, Russell I. Achieving health gain through clinical guidelines. I: Developing scientifically valid guidelines. Qual Health Care 1993;2:243-8.
20. Wyatt J, Spiegelhalter D. Evaluating medical expert systems: what to test and how? Med Inf 1990;15:205-17.

CiteULike    Complore    Connotea    Del.icio.us    Digg    Reddit    StumbleUpon    Technorati    What's this?

Relevant Article

Testing of PRODIGY continues

Ian Purves, Gary Nestor, and Katherine Williams
BMJ 1998 316: 776. [Extract] [Full Text]

This article has been cited by other articles:

• Fox, J., Patkar, V., Chronakis, I., Begent, R. (2009). From practice guidelines to clinical decision support: closing the loop. JRSM 102: 464-473 [Full text]  
• Eslami, S., Abu-Hanna, A., de Keizer, N. F. (2007). Evaluation of Outpatient Computerized Physician Medication Order Entry Systems: A Systematic Review. J. Am. Med. Inform. Assoc. 14: 400-406 [Abstract] [Full text]  
• Haller, G., Myles, P. S., Stoelwinder, J., Langley, M., Anderson, H., McNeil, J. (2007). Integrating Incident Reporting into an Electronic Patient Record System. J. Am. Med. Inform. Assoc. 14: 175-181 [Abstract] [Full text]  
• Delpierre, C., Cuzin, L., Fillaux, J., Alvarez, M., Massip, P., Lang, T. (2004). A systematic review of computer-based patient record systems and quality of care: more randomized clinical trials or a broader approach?. Int J Qual Health Care 16: 407-416 [Abstract] [Full text]  
• Sutton, D. R., Fox, J. (2003). The Syntax and Semantics of the PROforma Guideline Modeling Language. J. Am. Med. Inform. Assoc. 10: 433-443 [Abstract] [Full text]  
• Fraser, H. S. F., Long, W. J., Naimi, S. (2003). Evaluation of a Cardiac Diagnostic Program in a Typical Clinical Setting. J. Am. Med. Inform. Assoc. 10: 373-381 [Abstract] [Full text]  
• Hippisley-Cox, J., Pringle, M., Cater, R., Wynn, A., Hammersley, V., Coupland, C., Hapgood, R., Horsfield, P., Teasdale, S., Johnson, C. (2003). The electronic patient record in primary care--regression or progression? A cross sectional study. BMJ 326: 1439-1443 [Abstract] [Full text]  
• Denig, P, Witteman, C L M, Schouten, H W (2002). Scope and nature of prescribing decisions made by general practitioners. Qual Saf Health Care 11: 137-143 [Abstract] [Full text]  
• Rose, P. W, Watson, E., Yudkin, P., Emery, J., Murphy, M., Fuller, A., Lucassen, A. (2001). Referral of patients with a family history of breast/ovarian cancer--GPs' knowledge and expectations. Fam Pract 18: 487-490 [Abstract] [Full text]  
• Tai, S. S., Donegan, C., Nazareth, I. (2000). Computers in general practice and the consultation: the health professionals' view. Health Informatics Journal 6: 27-31 [Abstract]  
• Emery, J., Walton, R., Coulson, A., Glasspool, D., Ziebland, S., Fox, J. (1999). Computer support for recording and interpreting family histories of breast and ovarian cancer in primary care (RAGs): qualitative evaluation with simulated patients. BMJ 319: 32-36 [Abstract] [Full text]  
• Roberts, C. (1998). Quality assurance in primary care with information management and technology. Health Informatics Journal 4: 101-105 [Abstract]  
• Schiff, G. D., Rucker, T. D. (1998). Computerized Prescribing: Building the Electronic Infrastructure for Better Medication Usage. JAMA 279: 1024-1029 [Abstract] [Full text]