Primary care

Effect of computerised evidence based guidelines on management of asthma and angina in adults in primary care: cluster randomised controlled trial

Martin Eccles, professor of clinical effectiveness ^a, Elaine McColl, national primary care career scientist ^a, Nick Steen, statistician ^a, Nikki Rousseau, research associate ^a, Jeremy Grimshaw, professor ^b, David Parkin, professor of health economics ^c, Ian Purves, professor ^d. 

^a Centre for Health Services Research, University of Newcastle, Newcastle upon Tyne NE2 4AA, ^b Health Services Research Unit, University of Aberdeen, Aberdeen AB25 2ZD, ^c Department of Economics, City University, London EC1V 0HB, ^d Sowerby Centre for Health Informatics, University of Newcastle, Newcastle upon Tyne NE2 4AB

Correspondence to: M Eccles martin.eccles{at}ncl.ac.uk

	Abstract

Top Abstract Introduction Methods Results Discussion References

Objective: To evaluate the use of a computerised support system for decision making for implementing evidence based clinical guidelines for the management of asthma and angina in adults in primary care.
Design: A before and after pragmatic cluster randomised controlled trial utilising a two by two incomplete block design.
Setting: 60 general practices in north east England.
Participants: General practitioners and practice nurses in the study practices and their patients aged 18 or over with angina or asthma.
Main outcome measures: Adherence to the guidelines, based on review of case notes and patient reported generic and condition specific outcome measures.
Results: The computerised decision support system had no significant effect on consultation rates, process of care measures (including prescribing), or any patient reported outcomes for either condition. Levels of use of the software were low.
Conclusions: No effect was found of computerised evidence based guidelines on the management of asthma or angina in adults in primary care. This was probably due to low levels of use of the software, despite the system being optimised as far as was technically possible. Even if the technical problems of producing a system that fully supports the management of chronic disease were solved, there remains the challenge of integrating the systems into clinical encounters where busy practitioners manage patients with complex, multiple conditions.

What is already known on this topic Computerised decision support systems produce improvements in patient care across a range of conditions and settings Previous evaluations have been undermined by flaws in study design Few studies have evaluated complex decision support systems for the management of chronic disease What this study adds No impact was found of a computerised decision support system delivering evidence based guidelines for chronic diseases on either the process or outcomes of care It is unclear whether there are benefits from integrating such systems into clinical encounters where busy practitioners manage patients with complex and multiple conditions

	Introduction

Top Abstract Introduction Methods Results Discussion References

Despite the current interest in clinical guidelines there remains uncertainty about how best to introduce them into routine practice. A systematic review suggested the use of patient specific prompts at the time of consultation, which can be achieved with a computerised support system for decision makingthat is, "a system that compares patient characteristics with a knowledge base and then guides a health provider by offering patient specific and situation specific advice." ^{1 2} A recent systematic review of 68 controlled trials examined the effectiveness of such systems.³ They were shown to be beneficial: nine of 15 trials of systems to improve drug dosing; one of five trials evaluating diagnostic aids; 14 of 19 trials evaluating systems to improve preventive care; and 19 of 26 trials evaluating "other" medical care such as the management of disease in hospital and ordering tests. Improvements were found in six of the 14 studies measuring patient outcomes. However, the authors reported that most of the studies had flaws in design or analysis so that the findings should be interpreted with caution. Moreover, no studies were identified in the management of chronic disease in primary care or in computerised decision support systems integrated into routine computer systems in primary care.

We undertook a pragmatic cluster randomised controlled trial of a computerised decision support system to implement clinical guidelines for the management of asthma and angina in adults in primary care.

	Methods

Top Abstract Introduction Methods Results Discussion References

Our study methods are reported in detail elsewhere. ^{4 5} We chose as chronic illnesses angina and asthma in adults; these diseases are predominantly cared for in primary care and are important because of their morbidity and mortality. We developed evidence based guidelines for the two conditions. ^{6 7} Our study was approved by the appropriate multicentre research ethics committee.

Study general practices
We chose the study practices because their computer systems were extensively used. General practices in north east England were eligible to participate if at least 50% of the doctors reported using one of two computer systems to view clinical data and to issue prescriptions during consultations. We excluded singlehanded practices as they had too few patients for the required sample size.

Study patients
Study patients were those registered with the participating practices, aged 18 or over with angina or asthma. They were identified from a computerised search for relevant codes for the conditions, their management, and drug treatment.

Study design and power
Our study was a before and after, cluster randomised controlled trial utilising a two by two incomplete block design. We allocated general practices to one of two groups by a computerised randomisation process, stratified by computer system and by the vocational training status of the general practices (as a proxy for being innovative).⁸ One group received computerised guidelines for the management of angina and provided intervention patients for the management of angina and control patients for the management of asthma. The other received computerised guidelines for the management of asthma and provided intervention patients for the management of asthma and control patients for the management of angina.

View larger version (27K): [in this window] [in a new window]   Fig 1.    Flow of practices through trial

We regarded our design as two embedded trials, and we determined the sample size for each separately. Each trial required 80% power to detect a 10% difference in adherence to guideline recommendations (for example, between 45% and 55%) with a significance level of 5%. Adherence to the guidelines was defined by measures of process as recorded in the patients' records. Because the intraclass correlation coefficients for measures of process were estimated to be around 0.05, we collected data from 40 patients with each condition in each of 60 practices.⁹ We assessed changes in patient outcome with summated Likert scales that could be considered as continuous variables with a normal distribution. Again, we assumed an intraclass correlation coefficient of 0.05.⁹ Application of standard methods indicated that if we collected data from 40 patients from each of 60 practices we would have 80% power to detect an effect size of 0.2 standard deviations with a significance of 5%.¹⁰

Data collection
We collected process data from patients' records at the end of the intervention period for the 24 months from 12 months before to 12 months after the introduction of the computerised decision support system. Prescribing data were abstracted electronically from computerised clinical records. Trained data collectors, blinded to practice allocation, manually abstracted non-prescribing data from the patients' written and computerised records.

We gathered patient reported outcomes and economic data from three rounds of postal questionnaires. We posted questionnaires 12 months before the intervention, immediately before the intervention, and 12 months after the intervention. We used a combination of generic (SF-36 and the EQ-5D) and condition specific measures of quality of life (the Seattle angina questionnaire, the Newcastle asthma symptoms questionnaire, and the asthma quality of life questionnaire).^11-16

We collected data on three overlapping non-identical samples of patients. To obtain 40 patients with each condition per practice after three rounds of questionnaires, we drew a random sample of 80 eligible patients with each condition per practice for the first postal outcome survey. The dataset for patient reported outcomes comprised all patients completing all three questionnaires. The dataset for prescribing comprised all patients who responded to the first outcome survey. We gathered process data based on clinical records for a random sample of 40 patients who responded to the first outcome survey. A usage log within the practice computers recorded when the guidelines were used and by whom.

Analysis
We analysed data from the three patient samples independently, and we analysed each condition separately. We analysed each dataset in Stata, with generalised linear modelling procedures appropriate for hierarchical data. We analysed binary variables, data in the form of counts, and continuous outcome variables with binary, Poisson, and normal error structures, respectively. We modelled as random effects variation between practices and variation between patients (nested within practices). We included as a fixed effect the effect of decision support software.

We analysed each variable for clinical record based data and prescribing data twice. We based a pragmatic intention to treat analysis on all patients, whether or not they had consulted. We restricted an explanatory analysis to those patients who consulted after implementation of the system and for whom there was thus an opportunity for the intervention to influence their management.

Intervention

Computerised decision support system
The computerised decision support system was based on pre-existing software that was currently available to support prescribing for acute conditions.¹⁷ The two suppliers of the practice computer systems integrated the software into their products. The system anticipated clinicians' requirements by using information contained within a patient's computerised record to trigger the guideline and present patient scenariosfor example, for asthma, review of stable patient or acute exacerbation. Based on the chosen scenario, the system offered suggestions for management (including prescribing) informed by the content of the patient's record and requested the entry of relevant information, which was then stored in the patient's record. The guideline was, however, a separate path within the clinical system, and it was not possible to access all other parts of the clinical system from within the guideline. If the guideline was exited, it was only possible to return to it at the beginning of the pathway. The system was triggered either automatically by entering the electronic record of a patient previously identified as eligible or by entering a relevant morbidity code. After 4 months (based on feedback from practices) the system was altered so that triggering occurred only in response to a relevant morbidity code.

View larger version (37K): [in this window] [in a new window]   Fig 2.   Data collected

Training
Immediately before the intervention period each practice was invited to send two members to a one day workshop on using the system (training materials were supplied). Each doctor or practice nurse in the study received a paper copy of the summary of both guidelines, and each practice received a paper copy of the full version of both guidelines.

	Results

Top Abstract Introduction Methods Results Discussion References

Figure 1 shows the flow of practices through the trial. Six eligible practices were selected for an embedded case study, which is reported elsewhere.⁵ One practice (randomised to the asthma guideline) upgraded their system software and was unable to provide data on prescribing.

Participating general practices and patient samples
Table 1 shows the characteristics of the participating practices. The randomised groups were of similar size and vocational training status, and there were no differences in any aspect of practice organisation or computer usage relevant to the care of patients with angina or asthma.

Figure 2 summarises the data collected. Across all three samples there were no significant differences in the mean number of patients per practice between the two study groups.

Clinical record data and prescribing data
Reliability coefficients of 0.73 to 1.00 were obtained across the three data collectors for collection of 15 variables of process of care. No significant effects were found of the computerised decision support system on consultation rates, any aspect of the process of care for patients with angina or asthma (tables 2 and 3), or prescription of any category of drugs (tables 4 and 5). For all the reported variables from the clinical records the guidelines suggested that the action should be performed, although at varying intervals ranging from yearly to once only. For angina the guidelines recommended the first line use of  blockers, with other drugs being used in the case of intolerance or inadequate control of symptoms. For asthma the guidelines recommended the appropriate use of all reported drugs. As data collection related to two 12 month periods the results will potentially underestimate less frequently performed actions. It is also possible that actions were performed but not recorded in the record. Intraclass correlation coefficients were calculated for each condition and are reported elsewhere.⁵

Patient reported outcomes
Overall, 4851 patients with angina and 4960 patients with asthma were identified for the first survey of patient outcomes, and 2241 (46%) patients with angina and 1760 (35%) patients with asthma completed questionnaires in all three rounds (fig 2). Response rates to the surveys were similar between the randomised groups. The computerised guidelines showed no effect on any patient reported outcome. For each condition intraclass correlation coefficients were calculated for each measure and each round and are reported elsewhere.⁵

Usage log files
Figure 3 shows the number of times the guidelines were triggered for each practice and the proportion of active interactions that involved going beyond the first screen. All data are standardised per whole time equivalent general practitioner. For both suppliers, the median number of active interactions was zero for much of the study.

View larger version (27K): [in this window] [in a new window]   Fig 3.   Box plot of number of times guidelines were triggered per whole time equivalent general practitioner, and proportion of active interactions that involved going beyond first screen, by month of study and supplier. Lower and upper edges of each box represent the 25th and 75th centile of observed data. The line partitioning box corresponds to median observation. In several instances median observation was zero and is coincident with lower edge of box. Whiskers give range of data

	Discussion

Top Abstract Introduction Methods Results Discussion References

We found no effect of a computerised decision support system as a vehicle for implementing evidence based guidelines for the management of two chronic diseases in primary care. We addressed the complexities of such management where clinicians provide ongoing care for patients with complex conditions and for extended periods. At the time of its deployment the system was embedded in two of the most widely used practice computer systems in the United Kingdom and was more sophisticated than any other commercially available to primary care facilities in the United Kingdom. We were, however, unable to show any incremental effect over the distribution of paper versions of the guidelines across a comprehensive range of measures, almost certainly due to the low levels of use.

A systematic review suggested caution in interpreting the results of identified studies of computerised decision support systems because of flaws in their design or analysis, a common situation in studies on changing professional behaviour. ^{3 18} However, we addressed all of the important issues for design, conduct, and analysis. ^{4 19}

Asthma and angina are common chronic diseases in primary care. Both are of low incidence so dimensions of care such as initial investigations would be infrequent and less likely to be affected by any method of guideline implementation. Much of both guidelines dealt with the ongoing management of established cases, and the low incidence of both conditions should not have affected this. None the less, there are areas of care that an interactive computerised decision support system may be less able to influence, such as the issuing of routine repeat prescriptions by administrative staff. However, good clinical practice suggests that most patients with angina or asthma should receive an annual review, and within the trial almost all the patients consulted sufficiently frequently for this to have been considered. Although the guidelines reflected much of current practiceand performance of some actions may have been close to optimal (for example, use of short acting ₂ agonists)they both made recommendations for management that were not routine at the time of the study. Therefore it is unlikely that the guidelines merely enshrined all of current practice, and the data on process of care before the intervention show that this was not the case.

Although the study practices were selected on the basis of the extensive use of their computer systems (and thus were most likely to use a computerised decision support system), the staff had limited training in the functioning and use of the system. Limiting the amount of training in our study to one day was a pragmatic decision based on resources but was not that far removed from what is routine computer training within primary care in the United Kingdom.

For most general practitioners in the study the computerised system functioned in the context of routine surgeries (as opposed to settings such as clinics dedicated to disease management). Patients could present with any clinical problem such as arthritis or depression and, despite having asthma or angina, might not wish to discuss this, even though the computerised system might suggest this was appropriate. Given the range and complexity of problems that patients present to general practitioners it is demanding for any system to function in an unobtrusive yet helpful manner. The negative findings and low levels of use in our trial are similar to those observed by Hetlevik et al, who evaluated a computerised decision support system for the management of patients with hypertension or diabetes in primary care. ^{20 21} They found that the guideline was used in the management of only 12% of patients with diabetes.²¹

Assuming that the technical challenges of producing a system that truly supports the management of complex disease can be overcome there remains the problem of how such systems function within clinical encounters where patients with complex conditions are managed. "To be widely accepted by practising clinicians, computerised support systems for decision making must be integrated into the clinical workflow. They must present the right information, in the right format, at the right time, without requiring special effort."²² It is at least possible that for some or all of the reasons discussed above the low levels of use are all that can be reasonably expected of a computerised decision support system for the management of chronic disease.²¹ Certainly, in terms of implementing evidence based care, computerisation seems unlikely ever to be the "magic bullet" that answers all questions, and the current system could not be recommended.²³ Although an increasing number of studies show that computerised decision support systems can function in a variety of circumstances, the challenge still remains to show how far this is possible, desirable, and efficient.³

	Acknowledgments

We thank the participants, David Stables (EMIS Computing), Jon Rogers (Torex Meditel), Nick Booth, Neil Jones, and Bob Sugden (Sowerby Centre for Health Informatics), and members of the trial steering group.

Contributors: ME was the lead investigator and author; he will act as guarantor for the paper. ME, JG, and IP conceived the study. ME, JG, DP, IP, and NS were responsible for the initial design and applied for funding. IP designed and delivered the computerised decision support system. ME, JG, EMcC, DP, IP, NR, and NS were responsible for ongoing study design and interpretation. EMcC managed the project. EMcC and NR managed the survey data and collected the process data. NS performed the statistical analysis. DP performed the economic evaluation. All authors contributed to writing the paper.

	Footnotes

Funding: The study was funded by the UK NHS research and development programme "Methods to promote the uptake of research findings" (£800 497; $1 257 661; 1 273 055) with additional funding from EMIS Computing (£45 500; $71 485; 72 360) and the Department of Health for England and Wales (£85 156; $133 789; 135 426). The Health Services Research Unit, University of Aberdeen, is funded by the Chief Scientist Office of the Scottish Executive Health Department. EMcC is funded by the UK NHS Primary Care Development Programme. The Centre for Health Services Research, University of Newcastle and the Health Services Research Unit, University of Aberdeen are part of the UK Medical Research Council Health Services Research Collaboration. The views expressed are those of the authors and not necessarily those of the funding bodies.

Competing interests: None declared.

	References

Top Abstract Introduction Methods Results Discussion References

1.	Grimshaw J, Freemantle N, Wallace S, Russell I, Hurwitz B, Watt I, et al. Developing and implementing clinical practice guidelines. QHC 1995; 4: 55-64[Free Full Text].
2.	Friedman CP, Wyatt JC. Evaluation methods in medical informatics. New York: Springer-Verlag, 1997.
3.	Hunt DL, Haynes RB, Hanna SE, Smith K. Effects of computer-based clinical decision support systems on physician performance and patient outcomes. JAMA 1998; 280: 1339-1346[Abstract/Free Full Text].
4.	Eccles M, Grimshaw J, Steen N, Parkin D, Purves I, McColl E, et al. The design and analysis of a randomised controlled trial to evaluate computerised decision support in primary care: the COGENT Study. Fam Pract 2000; 17: 180-186[Free Full Text].
5.	Eccles M, McColl E, Steen N, Rousseau N, Grimshaw J, Parkin D. An evaluation of computerised guidelines for the management of two chronic conditions. University of Newcastle: Centre for Health Services Research, 2002.
6.	Eccles M, Rousseau N, Adams P, Thomas L. Evidence-based guideline for the primary care management of stable angina. Fam Pract 2001; 18: 217-222[Free Full Text].
7.	Eccles M, Rousseau N, Higgins B, Thomas L. Evidence-based guideline on the primary care management of asthma. Fam Pract 2001; 18: 223-229[Free Full Text].
8.	Baker R. General practice in Gloucestershire, Avon and Somerset: explaining variations in standards. Br J Gen Pract 1992; 42: 415-418[Web of Science][Medline].
9.	Campbell M, Grimshaw J, Steen N. Sample size calculations for cluster randomised trials. J Health Serv Res Pol 2000; 5: 12-16[Medline].
10.	Donner A. Some aspects of the design and analysis of cluster randomization trials. Appl Stats 1998; 47: 95-113.
11.	Ware Jr JE, Sherbourne CD. The MOS 36-item short-form health survey (SF-36). I. Conceptual framework and item selection. Med Care 1992; 30: 473-483[Web of Science][Medline].
12.	Brooks R. Euroqolthe current state of play. Health Policy 1996; 37: 53-72[CrossRef][Web of Science][Medline].
13.	Spertus JA, Winder JA, Dewhurst TA, Deyo RA, Prodzinski J, McDonnell M, et al. Development and evaluation of the Seattle Angina Questionnaire: a new functional status measure for coronary artery disease. J Am Coll Cardiol 1999; 25: 333-341.
14.	Steen IN, Hutchinson A, McColl E, Eccles MP, Hewison J, Meadows KA, et al. Development of a symptom-based outcome measures for asthma. BMJ 1994; 309: 1065-1069[Free Full Text].
15.	Juniper EF, Guyatt GH, Ferrie PJ, Griffith LE. Measuring quality of life in asthma 832-838. Am Rev Respir Dis 1993; 147: 832-838[Web of Science][Medline].
16.	Juniper EF, Buist AS, Cox FM, Ferrie PJ, King DR. Validation of a standardized version of the Asthma Quality of Life Questionnaire. Chest 1999; 115: 1265-1270[Abstract/Free Full Text].
17.	Purves IN. PRODIGY: implementing clinical guidance using computers. Br J Gen Pract 1998; 48: 1552-1553[Web of Science][Medline].
18.	Bero L, Grilli R, Grimshaw JM, Harvey E, Oxman AD, Thomson MA. Closing the gap between research and practice: an overview of systematic reviews of interventions to promote implementation of research findings by health care professionals. BMJ 1998; 317: 465-468[Free Full Text].
19.	Campbell MK, Steen N, Grimshaw JM, Eccles M, Mollison J, Lombard C. Design and statistical issues in implementation research. In: Thorsen T, Makela M, eds. Changing professional practice: theory and practice of clinical guidelines implementation, pp 57-76. Copenhagen: DSI - Danish Institute for Health Services Research and Development, 1999.
20.	Hetlevik I, Holmen J, Krüger Ø, Kristensen P, Iversen H. Implementing clinical guidelines in the treatment of hypertension in general practice. Blood Pressure 1998; 7: 270-276[CrossRef][Medline].
21.	Hetlevik I, Holmen J, Krüger Ø, Kristensen P, Iversen H, Furuseth K. Implementing clinical guidelines in the treatment of diabetes mellitus in general practice. Evaluation of effort, process and patient outcome related to implementation of a computer-based decision support system. Int J Tech Assess Health Care 2000; 16: 210-227.
22.	James BC. Making it easy to do right. N Engl J Med 2001; 345: 991-992[Free Full Text].
23.	Foy R, Eccles M, Grimshaw J. Why does primary care need more implementation research? Fam Pract 2001; 18: 353-355[Free Full Text].

(Accepted 12 September 2002)