Primary care

Practice based, longitudinal, qualitative interview study of computerised evidence based guidelines in primary care

Nikki Rousseau, research associate ^a, Elaine McColl, national primary care career scientist ^a, John Newton, principal lecturer ^b, Jeremy Grimshaw, professor of health services research ^c, Martin Eccles, professor of clinical effectiveness ^a. 

^a Centre for Health Services Research, University of Newcastle upon Tyne, Newcastle upon Tyne NE2 4AA, ^b Department of Sociology, University of Northumbria, Newcastle upon Tyne, ^c Health Services Research Unit, University of Aberdeen, Aberdeen

Correspondence to: M Eccles martin.eccles{at}ncl.ac.uk

	Abstract

Top Abstract Introduction Methods Results Discussion References

Objective: To understand the factors influencing the adoption of a computerised clinical decision support system for two chronic diseases in general practice.
Design: Practice based, longitudinal, qualitative interview study.
Setting: Five general practices in north east England.
Participants: 13 respondents (two practice managers, three nurses, and eight general practitioners) gave a total of 19 semistructured interviews. 40 people in practices included in the randomised controlled trial (34 doctors, three nurses) and interview study (three doctors, one previously interviewed) gave feedback.
Results: Negative comments about the decision support system significantly outweighed the positive or neutral comments. Three main areas of concern among clinicians emerged: timing of the guideline trigger, ease of use of the system, and helpfulness of the content. Respondents did not feel that the system fitted well within the general practice context. Experience of "on-demand" information sources, which were generally more positively viewed, informed the comments about the system. Some general practitioners suggested that nurses might find the guideline content more clinically useful and might be more prepared to use a computerised decision support system, but lack of feedback from nurses who had experienced the system limited the ability to assess this.
Conclusions: Significant barriers exist to the use of complex clinical decision support systems for chronic disease by general practitioners. Key issues include the relevance and accuracy of messages and the flexibility to respond to other factors influencing decision making in primary care.

What is already known on this topic Randomised controlled trials of complex computerised decision support systems have found low rates of use and no effects on process and outcomes of care What this study adds Clinicians found a computerised decision support system for chronic disease in general practice to be difficult to use and unhelpful clinically It did not fit well into a general practice consultation and compared unfavourably with "on-demand" information "Active" decision support can make clinicians aware of gaps between their own practice and "best" practice, but computer prompts need to be relevant and accurate

	Introduction

Top Abstract Introduction Methods Results Discussion References

We conducted a randomised controlled trial of a computerised decision support system for the primary care management of two common chronic diseases, which is reported in detail elsewhere and summarised on bmj.com.^1-3 In this paper we report a qualitative interview study conducted in parallel in order to illuminate our findings that use of the computerised decision support system had no significant effects on consultation rates, process of care measures (including prescribing), or any patient reported outcomes for either condition and that levels of use of the system were low. ^{4 5}

	Methods

Top Abstract Introduction Methods Results Discussion References

Participating general practices
As the conduct of an interview study in practices participating in a randomised controlled trial could both constitute a co-intervention and increase the burden of participating in the study, we recruited practices to only one or other aspect of the study. From the 60 general practices eligible and willing to take part in the trial, we recruited five (from north east England) to the interview study. ^{4 5} We purposively selected practices on the following criteria: supplier of clinical computer system, vocational training status, number of general practitioners, reported use of guidelines for asthma and angina, and level of computerisation (table).

Interviews
We undertook initial interviews with the designated contact person in each practice. We undertook further interviews with a purposive sample of professionals to ensure representation of clinicians described by their colleagues as having a particular interest in asthma or angina, those who attended a training workshop on the use of the computerised decision support system, and those who had not shown any particular interest in the system. We conducted interviews before and at different times during the intervention period.

Topics covered in the interviews included use of the computer; use of guidelines, especially for asthma and angina; organisation of care for patients with asthma or angina; and experiences of using the computerised decision support system. NR conducted all the interviews, and all were taped and transcribed verbatim. Three researchers (EMcC, JN, and NR) identified emergent themes and then met to construct an agreed list and coding frame. The dominant themes presented in this paper emerged through an iterative process of coding, analysis of coded text, and discussion among the authors.

Other sources of data
Six months after installation of the computerised decision support system we sent forms to all clinicians in randomised controlled trial practices and interview study practices inviting feedback on the software, the content of the guidelines, the information they had received about the system, how the system fitted into their care for patients, and any other aspect of the system. We compared this feedback with themes from the interviews, looking in particular for conflicting views or new themes.

Intervention
The intervention, which was the same in interview practices and trial practices, is described elsewhere. ^{2 3} In summary, two suppliers of general practice clinical computer systems integrated evidence based guidelines for the primary care management of asthma in adults and angina into their products. ^{6 7} The computerised decision support system anticipated clinicians' requirements by using information contained in patients' computerised records to trigger the guideline and present patient scenarios (for example, for asthma: review of stable patient; acute exacerbation). On the basis of the scenario chosen, the system offered suggestions for management informed by the content of the patient's record and requested the entry of relevant information, which was subsequently stored in the patient's record. The system could be triggered in two wayseither automatically when the clinician entered the electronic record of a patient previously identified as eligible or when a relevant morbidity code was entered.

Immediately before the intervention period we invited each practice to send two members of the practice to a one day training workshop for demonstration of the system and supply of training materials (including an html version of the guidelines).

	Results

Top Abstract Introduction Methods Results Discussion References

We carried out 19 semistructured interviews with a total of 13 respondentstwo practice managers, three nurses, and eight general practitioners. We received feedback from 40 people in randomised controlled trial practices (34 doctors, three nurses) and qualitative interview study practices (three doctors, including one previously interviewed). We identified no new themes in the feedback; rather, the feedback further clarified and reinforced themes from interviews.

Negative comments about the computerised decision support system significantly outweighed the positive or neutral comments. We identified three main areas of concern: the timing of the guideline trigger, the ease of use of the system, and the helpfulness of the content.

Triggering of the system
Automatic triggering of the computerised decision support system on entry into the record of a patient with asthma or angina was designed to facilitate opportunistic chronic disease management. However, clinicians generally disliked this feature and said they would be unlikely to carry out a chronic disease review if a patient was consulting for another reason (box 1). Given the time it took the system to launch, clinicians operating from branch surgeries with a slower computer connection found it particularly intrusive, as did those authorising repeat prescriptions for multiple patients.

The timing of the trigger in relation to the consultation was also problematic. Many clinicians liked to glance through the computer record while waiting for the patient to enter the consulting room. This was not a good time for the guideline to trigger, as the clinician did not yet know why the patient was consulting. Equally, the entry of a morbidity code at the end of a consultation (a common pattern) activated the system, but too late to be used. It therefore became an automatic reaction to "escape" out of the guidelines whenever they triggered. Part way through the intervention period, in response to feedback from practices, we altered triggering to present the system only in response to the entry of a morbidity code.

Ease of use
Most clinicians who tried out the system found it difficult to navigate (box 2). Attendance at the training workshops did not seem to help clinicians to use the system. A delay between the training day and the guideline becoming operational in practices (increased in some cases because of factors external to the study) reduced the benefit of the day. An html version of the guideline available in the interim period did not adequately prepare clinicians for the full version. Additionally, clinicians had limited access to clinical information from within the system. In practice, this meant that clinicians had to exit the system to access the patient's medical record, and once they had exited it was unusual for them to re-enter.

Helpfulness
Among the clinicians who persisted with using the system a strong theme that it was not helpful emerged (box 3). Three main factors contributed to this. (1) The guideline had limited ability to present options individualised to a specific patient. (2) Clinicians believed that they were already familiar with the content of the guideline (box 4). (3) The system did not (with some exceptions) aid adherence to those aspects of the guidelines that general practitioners were able and willing to follow and overemphasised areas to which they had given low priority or to which there were other barriers.

Areas in which clinicians acknowledged that they did not follow or disagreed with the guidelines highlighted perceived shortcomings of evidence based medicine in relation to new treatments, issues of patient preferences, and perceived structural barriers in the healthcare system. Some suggestion emerged that the computerised decision support system encouraged clinicians to consider aspects of care that they regarded as more marginal, as did a stronger impression that the inclusion of these aspects contributed to the unpopularity of the system.

"On-demand" information
Clinicians judged helpfulness by comparing the computerised decision support system with "on-demand" information (box 5). As well as guidelines and traditional sources of information, such as the advice of colleagues, clinicians used other sources of evidence in both paper and computerised formats. They seemed to enjoy using these tools and had found sources that they trusted and that gave them information in a style and volume that they found helpful. Some people suggested that the computerised decision support system could be used in this way, particularly in the html version.

Positive comments
Clinicians made a handful of more positive comments about the computerised decision support system. Some people seemed to be interested in the potential of the computer to remind them to carry out activities or suggest a course of action; some liked the patient information leaflets available through the system.

Nurses
Nurses have an important role in chronic disease management, and general practitioners suggested that nurses might be able to make use of computerised decision support systems as part of increasing responsibilities in this area (box 6). In some practices lower levels of access to computers meant that nurses could not use the system. This, coupled with low levels of feedback from nurses, meant that we could not fully assess the relative value of the system for nurses compared with general practitioners.

	Discussion

Top Abstract Introduction Methods Results Discussion References

The results of the randomised controlled trial showed that a computerised decision support system was not effective in improving the process or outcome of care for patients with asthma or angina, and this was almost certainly owing to low levels of use of the system.¹ The results of this interview study illuminate the reasons for this low use. Some of the issues highlighted by clinicians could be tackled with more timely training, in-practice support, and versions of the software that allow ready access to other parts of the clinical system. However, this would not tackle the more substantive challenges of providing a system that "fits" into the general practice context.⁸

Both the timing of the guideline trigger and the content of interjections were problematic. A primary care consultation is a complex interaction on both a professional and an interpersonal level, so intervening in this setting is difficult. Berg suggests that one problem with guidelines is the implication that patient management is a series of formal rational decisions and that there is a single optimal solution to every medical problem.⁹ Computerising guidelines within a decision support system can be seen as an extreme form of this. With a written guideline a clinician can still decide what is relevant to a particular patient and what to prioritise. With a computerised guideline it is the computer that compares what is known about the patient with formalised knowledge and presents solutions, but without the clinician's ability to judge the quality of the data and the relevance to a particular patient at a particular time.¹⁰ Instead of simplifying the process, this gives the clinician a new taskto evaluate the computer's choices and decisions.

General practitioners seem to value on-demand information (or "passive" decision support¹¹), particularly when this is in an accessible form.¹² However, to use such tools clinicians need to recognise that they have a need for information. Although clinicians considered themselves familiar with the content of the guidelines, process data from the trial indicate that clinicians did not always practise in line with the recommendations of the guidelines.¹ Clinicians seemed least happy when prompted in areas that they would not usually tackle or could not tackle because of external barriers. Any strategy for change in behaviour that prompts in such areas is likely to generate feelings of dissonance.

Limitations of the study
Although our sample of practices reflected the practices participating in the randomised controlled trial, within practices we interviewed fewer general practitioners who were low users of computers. The interviews are therefore more representative of general practitioners who were more likely to trigger the computerised decision support system. The people in the feedback group were self selecting and likely to include disproportionately more of those with strong reactions to the system. The voice of the disappointed enthusiast comes across strongly, and we know less about the views of those people who chose not to try the system. However, although the level of criticism of the system varied between clinicians, the nature of the criticisms, in terms of where the problems lay with the system, was remarkably consistent.

Developing technologies pose particular challenges in evaluationit is difficult to identify a "right time" to conduct a summative evaluation, and the technology has often moved on by the time the results are known. This does not mean that evaluations should not be done. Although both qualitative and quantitative methods can assist in the development of technologies, eventually the question "does it work?" needs to be answered.¹³ In questions of effectiveness the randomised controlled trial is the most appropriate research design. When evaluating complex interventions, such as a computerised decision support system, a parallel qualitative study serves to "open the black box" and elucidate why an intervention does or does not work. A combination of qualitative and quantitative methods provided a more thorough evaluation of the intervention than either alone would have done.

Conclusion
Clinicians did not adopt the computerised decision support system because they found it difficult to use and did not perceive it to bring benefits for practice. Key issues included the relevance and accuracy of messages and the flexibility to respond to other factors influencing decision making in primary care. These are important even for simple prompting systems but are multiplied in the more complex systems needed for chronic disease management. Complex decision support systems for chronic disease, integrated into clinical computer systems, are, in their current state of development, unlikely to be widely taken up by general practitioners.

	Acknowledgments

We thank the general practitioners, practice nurses, and practice staff in the study practices, especially those who took part in interviews. We also thank David Stables (EMIS Computing); Jon Rogers (Torex Meditel); and Nick Booth, Neil Jones, and Bob Sugden (Sowerby Centre for Health Informatics). Monica Smith advised on the design of this study. Rachel Baker and Tim Rapley gave helpful comments on drafts of the paper. Sylvia Hudson provided secretarial support and transcribed interviews. David Parkin, Ian Purves, and Nick Steen were members of the research team for the wider study.

Contributors: See bmj.com

	Footnotes

Funding: NHS R&D programme "Methods to promote the uptake of research findings"; additional funding from EMIS Computing and the Department of Health for England and Wales. The Health Services Research Unit, University of Aberdeen, is funded by the Chief Scientist Office of the Scottish Executive Health Department. EMcC and NR are funded by the UK NHS primary care development programme. The Centre for Health Services Research, University of Newcastle upon Tyne and the Health Services Research Unit, University of Aberdeen are part of the UK MRC Health Services Research Collaboration. The views expressed are those of the authors and not necessarily those of the funding bodies.

Competing interests: None declared.

This is an abridged version; the full version is on bmj.com

	References

Top Abstract Introduction Methods Results Discussion References

1.	Eccles M, McColl E, Steen N, Rousseau N, Grimshaw J, Parkin D, et al. Effect of computerised evidence based guidelines on management of asthma and angina in adults in primary care: cluster randomised controlled trial. BMJ 2002; 325: 941-947[Abstract/Free Full Text].
2.	Eccles M, Grimshaw J, Steen N, Parkin D, Purves I, McColl E, et al. The design and analysis of a randomised controlled trial to evaluate computerised decision support in primary care: the COGENT study. Fam Pract 2000; 17: 180-186[Free Full Text].
3.	Eccles M, McColl E, Steen N, Rousseau N, Grimshaw J, Parkin D. An evaluation of computerised guidelines for the management of two chronic conditions. Newcastle: Centre for Health Services Research, 2002.
4.	Yin RK. Case study research: design and methods 2nd ed. Thousand Oaks, CA: Sage, 1994.
5.	Rousseau N, McColl E, Eccles M, Hall L. Qualitative methods in implementation research. In: Thorsen T, Makela M, eds. Changing professional practice: theory and practice of clinical guidelines implementation. Copenhagen: DSI, 1999:99-116.
6.	Eccles M, Rousseau N, Adams P, Thomas L. Evidence-based guideline for the primary care management of stable angina. Fam Pract 2001; 18: 217-222[Free Full Text].
7.	Eccles M, Rousseau N, Higgins B, Thomas L. Evidence-based guideline on the primary care management of asthma. Fam Pract 2001; 18: 223-229[Free Full Text].
8.	Kaplan B. Evaluating informatics applicationssome alternative approaches: theory, social interactionism and call for methodological pluralism. Int J Med Inf 2001; 64: 39-56[CrossRef][ISI][Medline].
9.	Berg M. Problems and promises of the protocol. Soc Sci Med 1997; 44: 1081-1088[CrossRef][ISI][Medline].
10.	Berg M, Goorman E. The contextual nature of medical information. Int J Med Inf 1999; 56: 51-60[CrossRef][ISI][Medline].
11.	Elson RB, Connelly DP. Computerized decision support systems in primary care. Med Decis Making 1995; 22: 365-384.
12.	McColl A, Smith H, White P, Field J. General practitioners' perceptions of the route to evidence based medicine: a questionnaire survey. BMJ 1998; 316: 361-365[Abstract/Free Full Text].
13.	Eccles M, McColl E, Steen N, Rousseau N, Grimshaw J, Parkin D. Authors' reply to electronic responses. Effect of computerised evidence based guidelines on management of asthma and angina in adults in primary care: cluster randomised controlled trial. BMJ 2002. bmj.com/cgi/eletters/325/7370/941#26964 (accessed 9 Jan 2003).

(Accepted 23 December 2002)