Primary care

Practice based, longitudinal, qualitative interview study of computerised evidence based guidelines in primary care

Nikki Rousseau, research associate ^a, Elaine McColl, national primary care career scientist ^a, John Newton, principal lecturer ^b, Jeremy Grimshaw, professor of health services research ^c, Martin Eccles, professor of clinical effectiveness ^a. 

^a Centre for Health Services Research, University of Newcastle upon Tyne, Newcastle upon Tyne NE2 4AA, ^b Department of Sociology, University of Northumbria, Newcastle upon Tyne, ^c Health Services Research Unit, University of Aberdeen, Aberdeen

Correspondence to: M Eccles martin.eccles{at}ncl.ac.uk

	Abstract

Top Abstract Introduction Methods Results Discussion References

Objective: To understand the factors influencing the adoption of a computerised clinical decision support system for two chronic diseases in general practice.
Design: Practice based, longitudinal, qualitative interview study.
Setting: Five general practices in north east England.
Participants: 13 respondents (two practice managers, three nurses, and eight general practitioners) gave a total of 19 semistructured interviews. 40 people in practices included in the randomised controlled trial (34 doctors, three nurses) and interview study (three doctors, one previously interviewed) gave feedback.
Results: Negative comments about the decision support system significantly outweighed the positive or neutral comments. Three main areas of concern among clinicians emerged: timing of the guideline trigger, ease of use of the system, and helpfulness of the content. Respondents did not feel that the system fitted well within the general practice context. Experience of "on-demand" information sources, which were generally more positively viewed, informed the comments about the system. Some general practitioners suggested that nurses might find the guideline content more clinically useful and might be more prepared to use a computerised decision support system, but lack of feedback from nurses who had experienced the system limited the ability to assess this.
Conclusions: Significant barriers exist to the use of complex clinical decision support systems for chronic disease by general practitioners. Key issues include the relevance and accuracy of messages and the flexibility to respond to other factors influencing decision making in primary care.

What is already known on this topic Randomised controlled trials of complex computerised decision support systems have found low rates of use and no effects on process and outcomes of care What this study adds Clinicians found a computerised decision support system for chronic disease in general practice to be difficult to use and unhelpful clinically It did not fit well into a general practice consultation and compared unfavourably with "on-demand" information "Active" decision support can make clinicians aware of gaps between their own practice and "best" practice, but computer prompts need to be relevant and accurate

	Introduction

Top Abstract Introduction Methods Results Discussion References

Systematic reviews have shown that computerised systems can be an effective means of implementing guidelines in clinical practice.^1-3 However, they identified no studies of sophisticated computerised decision support systems in chronic disease management or integrated into routine computer systems. Although one of the most recent reviews identified 68 controlled trials,² little use has been made of qualitative techniques in evaluating computerised decision support systems in health care, ^{3 4} leaving unanswered questions about why systems are or are not effective. Models of implementation of guidelines and other innovations emphasise the importance of pre-existing attitudes and the context of the intervention, as well as the nature of the intervention itself, in the successful adoption of an intervention.^5-7 We conducted a randomised controlled trial of a computerised decision support system for the primary care management of two common chronic diseases, which is reported in detail elsewhere and summarised in box 1.^8-10 In this paper we report a qualitative interview study conducted in parallel in order to illuminate trial findings. ^{11 12}

	Methods

Top Abstract Introduction Methods Results Discussion References

Design
We designed a practice based, longitudinal, qualitative interview study to enable us to examine attitudinal and contextual influences on the use of the computerised decision support system.^5-7 Interviews in clinicians' consultation rooms allowed a detailed discussion of their usual practice in relation to the index conditions and a demonstration of how the system interacted with these consultations. We considered observing clinicians interacting with the system but judged this to be impracticable because interactions were infrequent and unpredictable outside chronic disease clinics for the index conditions.

Participating general practices
As the conduct of an interview study in practices participating in a randomised controlled trial could both constitute a co-intervention and increase the burden of participating in the study, we recruited practices to only one or other aspect of the study. From those practices eligible and willing to take part in the trial (box 1), we recruited five (from north east England) to the interview study. ^{11 12} We purposively selected practices on the following criteria: supplier of clinical computer system, vocational training status, number of general practitioners, reported use of guidelines for asthma and angina, and level of computerisation (table).

Interviews
We undertook initial interviews with the designated contact person in each practice. We undertook further interviews with a purposive sample of professionals to ensure representation of clinicians described by their colleagues as having a particular interest in asthma or angina, those who attended a training workshop on the use of the computerised decision support system, and those who had not shown any particular interest in the system. We conducted interviews before and at different times during the intervention period. Interviews lasted approximately one hour, and we conducted most of them in a surgery consulting room with a computer available, enabling the interviewee to refer to the computerised guidelines.

Topics covered in the interviews included use of the computer; use of guidelines, especially for asthma and angina; organisation of care for patients with asthma or angina; and experiences of using the computerised decision support system. We asked respondents to discuss both their own experiences and those of their colleagues in the practice. NR conducted all the interviews, and all were taped and transcribed verbatim. NR and one other researcher (EMcC or JN) reviewed each transcript and made notes of topics to be followed up at subsequent interviews. This approach enabled later interviews to build on and explore further what was already known about a practice and to feed in ideas from other practices as appropriate. Three researchers (EMcC, JN, and NR) identified emergent themes and then met to construct an agreed list and coding frame. All three researchers applied this to two transcripts; comparison of coding decisions enabled some codes to be clarified, and others merged. We subsequently imported the transcripts into the NVivo qualitative data analysis package for detailed coding (version 1.3, QSR International, Melbourne). The dominant themes presented in this paper emerged through an iterative process of coding, analysis of coded text, and discussion among the authors.

Other sources of data
The importance of using different types of data in qualitative research has been highlighted.¹³ Six months after installation of the computerised decision support system we sent forms to all clinicians in randomised controlled trial practices and interview study practices inviting feedback on the software, the content of the guidelines, the information they had received about the system, how the system fitted into their care for patients, and any other aspect of the system. We compared this feedback with themes from the interviews, looking in particular for conflicting views or new themes.

Intervention
The intervention, which was the same in interview practices and trial practices, is described elsewhere. ^{9 10} In summary, two suppliers of general practice clinical computer systems integrated evidence based guidelines for the primary care management of asthma in adults and angina into their products. ^{14 15} The computerised decision support system anticipated clinicians' requirements by using information contained in patients' computerised records to trigger the guideline and present patient scenarios (for example, for asthma: review of stable patient; acute exacerbation). On the basis of the scenario chosen, the system offered suggestions for management informed by the content of the patient's record and requested the entry of relevant information, which was subsequently stored in the patient's record. The system could be triggered in two wayseither automatically when the clinician entered the electronic record of a patient previously identified as eligible or when a relevant morbidity code was entered.

Immediately before the intervention period we invited each practice to send two members of the practice to a one day training workshop for demonstration of the system and supply of training materials (including an html version of the guidelines). In addition, every clinician (doctor or practice nurse) received a paper copy of the summary version of both guidelines and each practice received one paper copy of the full version of both guidelines.

	Results

Top Abstract Introduction Methods Results Discussion References

We carried out 19 semistructured interviews with a total of 13 respondentstwo practice managers, three nurses, and eight general practitioners. We received feedback from 40 people in randomised controlled trial practices (34 doctors, three nurses) and qualitative interview study practices (three doctors, including one previously interviewed). We identified no new themes in the feedback; rather, the feedback further clarified and reinforced themes from interviews.

People interviewed were largely enthusiastic about the benefits of computing for general practice and were optimistic about the potential for computers to present guidelines in a manageable format. However, negative comments about the computerised decision support system significantly outweighed the positive or neutral comments. We identified three main areas of concern: the timing of the guideline trigger, the ease of use of the system, and the helpfulness of the content.

Triggering of the system
Automatic triggering of the computerised decision support system on entry into the record of a patient with asthma or angina was designed to facilitate opportunistic chronic disease management. It also made the system visible within practices, ensuring that all clinicians whose computers were able to operate the system (see comment on nurses below) and who used the computer in their clinical practice (most clinicians in the study practices), were aware that the system was available. However, clinicians generally disliked this feature and said they would be unlikely to carry out a chronic disease review if a patient was consulting for another reason (box 2). In addition, inconsistencies in morbidity coding meant that the guidelines were sometimes triggered for patients without the condition. Given the time it took the system to launch, clinicians operating from branch surgeries with a slower computer connection found it particularly intrusive, as did those authorising repeat prescriptions for multiple patients.

The timing of the trigger in relation to the consultation was also problematic. Many clinicians liked to glance through the computer record while waiting for the patient to enter the consulting room. This was not a good time for the guideline to trigger, as the clinician did not yet know why the patient was consulting. Equally, the entry of a morbidity code at the end of a consultation (a common pattern) activated the system, but too late to be used. It therefore became an automatic reaction to "escape" out of the guidelines whenever they triggered, even on occasions when it might have been appropriate to use them. Part way through the intervention period, in response to feedback from practices, we altered triggering to present the system only in response to the entry of a morbidity code.

Ease of use
Most clinicians who tried out the system found it difficult to navigate (box 3). They acknowledged that this would be less likely if they were more familiar with the system. However, this meant taking some time outside a consultation to explore the software; they were generally reluctant to experiment with the system during a patient consultation because of the risk of "getting lost."

Attendance at the training workshops did not seem to help clinicians to use the system. A delay between the training day and the guideline becoming operational in practices (increased in some cases because of factors external to the study) reduced the benefit of the day. An html version of the guideline available in the interim period did not adequately prepare clinicians for the full version. Although many clinicians seemed resigned to having to "just get in and fiddle" with new computer software, several people made suggestions for additional support.

Additionally, clinicians had limited access to clinical information from within the system. In practice, this meant that clinicians had to exit the system to access the patient's medical record, and once they had exited it was unusual for them to re-enter.

Helpfulness
Among the clinicians who persisted with using the system a strong theme that it was not helpful emerged (box 4). Three main factors contributed to this. (1) The guideline had limited ability to present options individualised to a specific patient. (2) Clinicians believed that they were already familiar with the content of the guideline (box 5). (3) The system did not (with some exceptions) aid adherence to those aspects of the guidelines that general practitioners were able and willing to follow and overemphasised areas to which they had given low priority or to which there were other barriers.

To reduce the number of decisions for the clinician the computerised decision support system presented options customised to a particular patient, by using information in the medical record. However, clinicians found that the system often presented too many or inappropriate options. In addition, clinicians expressed concerns about trusting a computer to make management decisions and about the prompting mechanisms within the existing clinical computer system.

Many people interviewed did critically engage with the content of the guidelines at the training workshop, in paper format, or on the html version of the computerised guideline. Relatively few people had read the guideline content within the computerised decision support system. Many people believed that they were already practising in line with the recommendations in the guidelines or that the guidelines did not contain much new information (box 5). Areas in which clinicians acknowledged that they did not follow or disagreed with the guidelines highlighted perceived shortcomings of evidence based medicine in relation to new treatments, issues of patient preferences, and perceived structural barriers in the healthcare system. Some areas of disagreement related to the computerised implementation of the guidelines, which went further than the paper guidelines in recommending particular brands and quantities of drugs.

All practices had recently been involved in initiatives tackling aspects of asthma or (more often) angina care that fell within the clinical area of the guidelines. Attempting change could be unrewarding or have negative effects on other areas of practice (box 6). Clinicians therefore seemed to weigh up the pros and cons of different activities and prioritised those for which they felt that stronger incentives existed; these included financial incentives, personal interest, and pressure from external bodies. However, with limited time available, general practitioners also prioritised the aspects that they felt were most likely to produce positive effects. Some suggestion emerged that the computerised decision support system encouraged clinicians to consider aspects of care that they regarded as more marginal, as did a stronger impression that the inclusion of these aspects contributed to the unpopularity of the system.

"On-demand" information
Clinicians judged helpfulness by comparing the computerised decision support system with "on-demand" information (box 7). As well as guidelines and traditional sources of information, such as the advice of colleagues, clinicians used other sources of evidence in both paper and computerised formats. They seemed to enjoy using these tools and had found sources that they trusted and that gave them information in a style and volume that they found helpful. Some people suggested that the computerised decision support system could be used in this way, particularly in the html version.

Positive comments
Clinicians made a handful of more positive comments about the computerised decision support system. Some people seemed to be interested in the potential of the computer to remind them to carry out activities or suggest a course of action; some liked the patient information leaflets available through the system. Although the general perception was that the system took a long time to use, one clinician did suggest that some activities could be done more quickly with the system than by using usual approaches (box 4).

Nurses
Nurses have an important role in chronic disease management, and general practitioners suggested that nurses might be able to make use of computerised decision support systems as part of increasing responsibilities in this area (box 8). Consideration of the existing chronic disease management in study practices showed that nurses were more likely to make use of systematic forms of data collection. Those nurses who did try the system were more positive about features such as the missing information prompts and data collection tools than were general practitioners. In some practices lower levels of access to computers meant that nurses could not use the system. This, coupled with low levels of feedback from nurses, meant that we could not fully assess the relative value of the system for nurses compared with general practitioners.

	Discussion

Top Abstract Introduction Methods Results Discussion References

The results of the randomised controlled trial showed that a computerised decision support system was not effective in improving the process or outcome of care for patients with asthma or angina, and this was almost certainly owing to low levels of use of the system.⁸ The results of this interview study illuminate the reasons for this low use. Some of the issues highlighted by clinicians could be tackled with more timely training, in-practice support, and versions of the software that allow ready access to other parts of the clinical system. However, this would not tackle the more substantive challenges of providing a system that "fits" into the general practice context.⁴

Both the timing of the guideline trigger and the content of interjections were problematic. A primary care consultation is a complex interaction on both a professional and an interpersonal level, so intervening in this setting is difficult. Berg suggests that one problem with guidelines is the implication that patient management is a series of formal rational decisions and that there is a single optimal solution to every medical problem.¹⁶ Computerising guidelines within a decision support system can be seen as an extreme form of this. With a written guideline a clinician can still decide what is relevant to a particular patient and what to prioritise. With a computerised guideline it is the computer that compares what is known about the patient with formalised knowledge and presents solutions, but without the clinician's ability to judge the quality of the data and the relevance to a particular patient at a particular time.¹⁷ Instead of simplifying the process, this gives the clinician a new taskto evaluate the computer's choices and decisions.

General practitioners seem to value on-demand information (or "passive" decision support¹⁸), particularly when this is in an accessible form.¹⁹ However, to use such tools clinicians need to recognise that they have a need for information. Although clinicians considered themselves familiar with the content of the guidelines, process data from the trial indicate that clinicians did not always practise in line with the recommendations of the guidelines.⁸ Clinicians in this study mentioned many of the issues highlighted in previous work on implementation of guidelines. ^{20 21} Clinicians seemed least happy when prompted in areas that they would not usually tackle or could not tackle because of external barriers. Any strategy for change in behaviour that prompts in such areas is likely to generate feelings of dissonance. Conversely, computerised decision support systems may be appreciated if they give clinicians tools, such as patient information leaflets, with which to overcome barriers to change.

Although on-demand information as a strategy for behavioural change requires that clinicians recognise a gap in their knowledge, our data suggest that more active decision support can be difficult to integrate into general practice. Unless the computer can be trusted to provide messages that are highly relevant and accurate, a strong tendency to ignore these interventions exists.²² Although systematic reviews have concluded that simple computer prompt systems can be effective, ^{2 18} in routine practice prompts need to be carefully targeted. Prompting systems rely on consistent coding of medical record data and might best be reserved for occasions not only when strong evidence exists for a course of action but also when the potential benefit to the patient is greatest.

Limitations of the study
Although our sample of practices reflected the practices participating in the randomised controlled trial, within practices we interviewed fewer general practitioners who were low users of computers. The interviews are therefore more representative of general practitioners who were more likely to trigger the computerised decision support system. The people in the feedback group were self selecting and likely to include disproportionately more of those with strong reactions to the system. The voice of the disappointed enthusiast comes across strongly, and we know less about the views of those people who chose not to try the system. However, although the level of criticism of the system varied between clinicians, the nature of the criticisms, in terms of where the problems lay with the system, was remarkably consistent.

Developing technologies pose particular challenges in evaluationit is difficult to identify a "right time" to conduct a summative evaluation, and the technology has often moved on by the time the results are known. This does not mean that evaluations should not be done. Although both qualitative and quantitative methods can assist in the development of technologies, eventually the question "does it work?" needs to be answered.²³ In questions of effectiveness the randomised controlled trial is the most appropriate research design. When evaluating complex interventions, such as a computerised decision support system, a parallel qualitative study serves to "open the black box" and elucidate why an intervention does or does not work. Here the qualitative interview study enabled us to follow the intervention over a period of time, from different perspectives, without needing to cover preliminary ground on each occasion, and to build on what we already knew about the practice. Thus a combination of qualitative and quantitative methods provided a more thorough evaluation of the intervention than either alone would have done.

Conclusion
Clinicians did not adopt the computerised decision support system because they found it difficult to use and did not perceive it to bring benefits for practice. Key issues included the relevance and accuracy of messages and the flexibility to respond to other factors influencing decision making in primary care. These are important even for simple prompting systems but are multiplied in the more complex systems needed for chronic disease management. Computers have brought benefits to primary care and clearly have an important role in promoting evidence based practice. However, complex decision support systems for chronic disease, integrated into clinical computer systems, are, in their current state of development, unlikely to be widely taken up by general practitioners.

	Acknowledgments

We thank the general practitioners, practice nurses, and practice staff in the study practices, especially those who took part in interviews. We also thank David Stables (EMIS Computing); Jon Rogers (Torex Meditel); and Nick Booth, Neil Jones, and Bob Sugden (Sowerby Centre for Health Informatics). Monica Smith advised on the design of this study. Rachel Baker and Tim Rapley gave helpful comments on drafts of the paper. Sylvia Hudson provided secretarial support and transcribed interviews. David Parkin, Ian Purves, and Nick Steen were members of the research team for the wider study.

Contributers: NR, EMcC, JN, ME, and JG designed the study. NR conducted the interviews. NR, EMcC, and JN were responsible for analysis and interpretation. NR is the lead author on this paper, and all authors contributed to writing it. ME is the guarantor.

	Footnotes

Funding: NHS R&D programme "Methods to promote the uptake of research findings"; additional funding from EMIS Computing and the Department of Health for England and Wales. The Health Services Research Unit, University of Aberdeen, is funded by the Chief Scientist Office of the Scottish Executive Health Department. EMcC and NR are funded by the UK NHS primary care development programme. The Centre for Health Services Research, University of Newcastle upon Tyne and the Health Services Research Unit, University of Aberdeen are part of the UK MRC Health Services Research Collaboration. The views expressed are those of the authors and not necessarily those of the funding bodies.

Competing interests: None declared.

	References

Top Abstract Introduction Methods Results Discussion References

1.	Grimshaw J, Freemantle N, Wallace S, Russell I, Hurwitz B, Watt I, et al. Developing and implementing clinical practice guidelines. Qual Health Care 1995; 4: 55-64[Free Full Text].
2.	Hunt DL, Haynes RB, Hanna SE, Smith K. Effects of computer-based clinical decision support systems on physician performance and patient outcomes. JAMA 1998; 280: 1339-1346[Abstract/Free Full Text].
3.	Kaplan B. Evaluating informatics applicationsclinical decision support systems literature review. Int J Med Inf 2001; 64: 15-37[CrossRef][Web of Science][Medline].
4.	Kaplan B. Evaluating informatics applicationssome alternative approaches: theory, social interactionism and call for methodological pluralism. Int J Med Inf 2001; 64: 39-56[CrossRef][Web of Science][Medline].
5.	Kitson A, Harvey G, McCormack B. Enabling the implementation of evidence based practice: a conceptual framework. Qual Health Care 1998; 7: 149-158[Abstract].
6.	Lomas J. Teaching old (and not so old) docs new tricks: effective ways to implement research findings. In: Dunn EV, Norton PG, Sterwart M, Tudiver F, Bass MJ, eds. Disseminating research/changing practice. London: Sage, 1994.
7.	Rogers EM. Diffusion of innovations. 4th ed. New York: The Free Press, 1995.
8.	Eccles M, McColl E, Steen N, Rousseau N, Grimshaw J, Parkin D, et al. Effect of computerised evidence based guidelines on management of asthma and angina in adults in primary care: cluster randomised controlled trial. BMJ 2002; 325: 941-947[Abstract/Free Full Text].
9.	Eccles M, Grimshaw J, Steen N, Parkin D, Purves I, McColl E, et al. The design and analysis of a randomised controlled trial to evaluate computerised decision support in primary care: the COGENT study. Fam Pract 2000; 17: 180-186[Free Full Text].
10.	Eccles M, McColl E, Steen N, Rousseau N, Grimshaw J, Parkin D. An evaluation of computerised guidelines for the management of two chronic conditions. Newcastle: Centre for Health Services Research, 2002.
11.	Yin RK. Case study research: design and methods. 2nd ed. Thousand Oaks, CA: Sage, 1994.
12.	Rousseau N, McColl E, Eccles M, Hall L. Qualitative methods in implementation research. In: Thorsen T, Makela M, eds. Changing professional practice: theory and practice of clinical guidelines implementation. Copenhagen: DSI, 1999:99-116.
13.	Seale C. The quality of qualitative research. London: Sage, 1999.
14.	Eccles M, Rousseau N, Adams P, Thomas L. Evidence-based guideline for the primary care management of stable angina. Fam Pract 2001; 18: 217-222[Free Full Text].
15.	Eccles M, Rousseau N, Higgins B, Thomas L. Evidence-based guideline on the primary care management of asthma. Fam Pract 2001; 18: 223-229[Free Full Text].
16.	Berg M. Problems and promises of the protocol. Soc Sci Med 1997; 44: 1081-1088[CrossRef][Web of Science][Medline].
17.	Berg M, Goorman E. The contextual nature of medical information. Int J Med Inf 1999; 56: 51-60[CrossRef][Web of Science][Medline].
18.	Elson RB, Connelly DP. Computerized decision support systems in primary care. Med Decis Making 1995; 22: 365-384.
19.	McColl A, Smith H, White P, Field J. General practitioners' perceptions of the route to evidence based medicine: a questionnaire survey. BMJ 1998; 316: 361-365[Abstract/Free Full Text].
20.	Cabana MD, Rand CS, Powe NR, Wu AW, Wilson MH, Abboud PA, et al. Why don't physicians follow clinical practice guidelines? A framework for improvement. JAMA 1999; 282: 1458-1465[Abstract/Free Full Text].
21.	Freeman AC, Sweeney K. Why general practitioners do not implement evidence: qualitative study. BMJ 2001; 323: 1-5[Abstract/Free Full Text].
22.	Litzelman DK, Tierney WM. Physicians' reasons for failing to comply with the computerized preventive care guidelines. J Gen Intern Med 1996; 11: 497-499[Web of Science][Medline].
23.	Eccles M, McColl E, Steen N, Rousseau N, Grimshaw J, Parkin D. Authors' reply to electronic responses. Effect of computerised evidence based guidelines on management of asthma and angina in adults in primary care: cluster randomised controlled trial. BMJ 2002. bmj.com/cgi/eletters/325/7370/941#26964 (accessed 9 Jan 2003).

(Accepted 23 December 2002)