Guidelines on anticoagulant treatment in atrial fibrillation in Great Britain: variation in content and implications for treatment
BMJ 1998; 316 doi: https://doi.org/10.1136/bmj.316.7130.509 (Published 14 February 1998) Cite this as: BMJ 1998;316:509- Richard Thomson, senior lecturer in public health medicinea,
- Helen McElroy, research associatea,
- Mark Sudlow, MRC special training fellow in health services researchb
- a Department of Epidemiology and Public Health, School of Health Sciences, Medical School, University of Newcastle upon Tyne, Newcastle upon Tyne NE2 4HH
- b Departments of Epidemiology and Public Health and of Medicine, Medical School, University of Newcastle upon Tyne
- Correspondence to: Dr Thomson
- Accepted 22 October 1997
Abstract
Objective: To describe the content of guidelines on the use of anticoagulant treatment in patients with atrial fibrillation and the impact of variations in guidelines on treatment.
Design: Postal survey of guidelines, semistructured interview with lead developers of guidelines, and application of guidelines to patient sample.
Subjects: 15 lead developers of the 20 guidelines identified in the postal survey were interviewed. 100 patients over 65 with atrial fibrillation to whom the guidelines were applied.
Main outcome measures: Evaluation of guidelines and the methods of dissemination, implementation, review, and evaluation; proportion of patients recommended for anticoagulant treatment by each guideline; and level of agreement between guidelines.
Results: There was considerable variation in whether anticoagulant treatment was recommended for subjects (range 13% to 100%, κ=0.12). Guidelines varied greatly in advice on treatment by age, the use of echocardiography, and the target value or range of the international normalised ratio (8 of the 20 guidelines included values unlikely to be effective). Development was unsystematic; evidence based approaches were rarely used. 9 of the 15 lead developers had developed the guidelines themselves, and the 6 guidelines developed by groups relied on informal consensus. Methods to support effective dissemination, implementation, and evaluation were limited.
Conclusion: The widespread non-systematic production of guidelines has led to considerable variation with implications for the quality of care and clinical decision making. There is a need for a central, well funded programme of guideline development to ensure that valid guidelines are produced and disseminated.
Key messages
Clinical practice guidelines improve the process and outcomes of health care, but these improvements are dependent on the validity of the advice the guidelines contain
A survey of guidelines in Great Britain for the use of anticoagulant treatment to prevent stroke in patients with atrial fibrillation showed variation in their content and in the processes of development, dissemination, implementation, and evaluation
Applying these guidelines produced variation in treatment
This variation is most likely to be caused by non-systematic methods of development, which implies that some guidelines advocate management strategies that are less than optimal
Such variation could have profound effects on the incidence of stroke and bleeding complications and would lead to substantial differences in the use of resources, particularly anticoagulation services
Introduction
Clinical guidelines are an effective method for improving both process and outcome in health care.1 2 3 4 5 6 7 They have been promoted as an important tool in evidence based practice and may reduce inappropriate variations in treatment. However, the results of some surveys have created concerns about the quality of guidelines.8 9 10 Unless guidelines are produced using appropriate methods they may replace normal clinician variation with consistently inappropriate practice.
The use of anticoagulant treatment to prevent stroke in patients with atrial fibrillation is supported by randomised controlled trials and pooled analysis of their results.11 12 None the less treatment varies,13 14 15 16 17 18 19 20 thus providing conditions where valid clinical guidelines may be useful.
We performed a survey of guidelines in Great Britain to explore variation in content; we interviewed the lead developers of the guidelines to assess the reasons for variation. We then applied these guidelines to a community sample of patients with atrial fibrillation to determine whether the advice given in the guidelines would support consistent clinical decision making.
Subjects and methods
Clinical guidelines on the use of anticoagulant treatment in England, Wales, and Scotland were identified. Organisations that produced guidelines were contacted by telephone, and 440 people were sent a questionnaire seeking information on and copies of their guidelines and asking for details of other organisations that might also use guidelines. The sample comprised district directors of public health and public health representatives of the regional and national NHS Executive; chairpeople of the Medical Audit Advisory Group; national professional and charitable organisations with an interest in clinical guidelines, audit, cardiovascular and cerebrovascular disease; and members of the mailing list of the national medical audit conference. Non-respondents were sent an additional questionnaire. Further participants were sought through announcements at conferences on the quality of health care, personal contacts, and queries on relevant email discussion lists. An additional 94 contacts were identified this way and sent the questionnaire. Participants were chosen to represent purchasers and providers of health care who were likely to be aware of appropriate guidelines and also to represent relevant national organisations.
Guidelines were assessed independently for inclusion in the study by two investigators (RT and MS). A guideline was defined as a document produced to help clinicians decide which patients should be given anticoagulant drugs. Draft documents, documents designed for use only in a single specialised unit, and documents designed to provide guidance on the use of warfarin only once treatment had been decided on were excluded.
Between July and September 1996 the lead developer for each guideline was invited to participate in a semistructured telephone interview, which was recorded and transcribed. All interviews were done by a single interviewer (HM). Questions were asked about the development of the guideline—because of the recognised importance of development procedures in ensuring guideline validity1 5—and the format, purpose, dissemination, implementation, and review of the guideline.
Interviews were analysed qualitatively using Glaser and Strauss's work to identify emergent themes.21 Initial analysis of the transcripts was undertaken independently by two observers (RT and HM). Themes and concepts identified by each observer were confirmed or modified following discussion between the researchers and re-examination of transcripts.
All guidelines selected for inclusion were then applied to 100 consecutive patients with atrial fibrillation aged 65 years or older who were identified in a community survey. Details of risk factors for stroke and of contraindications to treatment with anticoagulants were obtained. A single observer (MS) determined whether each guideline recommended anticoagulation or another course of action (no treatment, aspirin, further investigation, or referral) for every patient. Intraobserver (test-retest) reliability was assessed by repeating this method 3 months later on a random sample of 20 of the patients using five guidelines. Interobserver reliability was assessed by comparing the decisions derived from the guidelines with the decisions of another observer (RT). The degree of agreement was quantified using κ.
We also compared the guidelines on three selected topics (recommendations on the use of echocardiography, implication of the patient's age on treatment, and the recommended target value or range of the international normalised ratio).
Results
The survey response rate was 350/534 (66%) and yielded 48 documents, of which 20 fulfilled the study definition for clinical guidelines. Guidelines were produced in a variety of formats—from single page algorithms to a 29 page report with an algorithm—were primarily for general practitioners, and were intended to be used in populations ranging from 12 000 to 500 000. The production of only two guidelines had been funded (by a pharmaceutical company and a health authority).
Interviews
Altogether 15 of the 20 people who developed the guidelines agreed to be interviewed (1). Six of them were general practitioners, seven were consultant physicians, one was a director of public health, and one was an independent medical adviser to a health authority.
Only 6 of the 15 guidelines were developed by groups. Each development group included at least one general practitioner and one consultant physician; half included a public health doctor. The roles required within development groups (subject expert, facilitator, literature searcher, evidence evaluator, chairperson, and guideline methodologist5) were commonly recognised by the developer, but often one person performed several roles. Six of the nine lead developers who had sole responsibility for guideline development had sought views from another health professional (five from cardiologists), while 7 had sought an external review before finalising the guidelines. Five of the 15 guidelines were developed by the end users (internal development), 3 were developed externally by a local organisation (usually a hospital consultant producing guidelines for local general practitioners), and 7 were produced by a combination of methods (intermediate development).
The main reasons given for the development of guidelines were the need to clarify the management of atrial fibrillation and to reduce the incidence of stroke. Other reasons for developing guidelines included the perception that atrial fibrillation was a common and important condition and the lack of satisfactory alternative guidelines. According to those interviewed, the main objectives in developing guidelines were to improve clinical management and to produce uniform practice (7/15), to reduce the incidence of stroke (5/15), and to improve clinical knowledge (2/15). Only one of the guidelines included written objectives.
Literature searches were used in the development of all guidelines, but the extent and detail of the searches and the appraisal of the literature varied considerably. Only 5 of the 15 lead developers had undertaken a literature search; six had used only literature of which they were already aware. Review articles were used by 11 and articles on original trials by five. Altogether five lead developers had reviewed other guidelines, but four had tried and failed to identify any. Some form of grading of the evidence was used by four of the lead developers.
Those who had developed guidelines within a group believed that the guidelines were evidence based. However, most groups had not undertaken systematic literature searches, had not involved anyone with specific expertise, had no formal method of evaluating the evidence, and had not attempted to link the recommendations specifically to the evidence. On the basis of Woolf's classification of guideline development,22 only one guideline developed by a group could be categorised as evidence based. However, none of the guidelines developed by individuals were evidence based using this definition, and it is difficult to understand how such a group dependent process could be undertaken by an individual.
The method and extent of distribution of the guidelines also varied considerably. Seven of the 15 guidelines were promoted further in local educational meetings, such as seminars and workshops for medical staff.
Of those interviewed, 10 said that they had reviewed or would be reviewing their guidelines. Only four had a specified date or mechanism for review. Only three intended to evaluate or were evaluating the impact of their guideline. Despite a lack of formal monitoring, 7 of them thought that changes in the use of anticoagulant treatment had occurred.
Those who were interviewed recognised deficiencies in the approaches that they had used to devise their guidelines. Several thought that a wider range of health professionals could have participated in the development. There was a common belief that the development process was time consuming and expensive and that those developing the guidelines had limited time and expertise; there were suggestions that external organisations or professional organisations should have had a wider role in development (box).
Comments from interviews with developers of guidelines
Reasons for development “… [T]o clarify the management of atrial fibrillation because we are unclear exactly how to deal with the problem.” (general practitioner)
“Because the supporting evidence has become increasingly compelling and we were aware from all that we did that we were failing to anticoagulate as many patients, or give them aspirin, as we should be according to the evidence.” (consultant geriatrician)
“… [G]eneral practitioners were keen to have some advice on … the management of atrial fibrillation.” (cardiologist)
Literature search and appraisal “I went to … known … publications first and found what I could. I could have looked much wider I suppose, but I think that would have just confused me even more because it was simply introducing more uncertainties. I just specifically picked those ones which I felt were useful.” (consultant geriatrician)
“[A paper] was graded in terms of suitability of the population studied and the way the trials were constructed, and on the basis of this I accepted some and rejected others.” (cardiologist)
“We had a preference for certain parts of the literature, I think in particular randomised controlled trials. So yes, there was an appraisal of the literature and how good the evidence was.” (director of public health)
“There was some evidence to which we gave preference and other that we did not, but no conscious weighting scale was drawn up.” (consultant geriatrician)
“We tried to grade the evidence by looking to see what type of studies they were, whether they were randomised controlled trials or whether they were unblinded or not controlled, whether they were consensus documents, where they were published, what conflicts of interest … standard evidence grading methods.” (general practitioner)
Limitations “I would have preferred to go through the process of doing a proper literature search looking at other people's reviews of guidelines and reviews of literature, and then to have involved not just the practice but other local experts. But it just wasn't possible in the time available … something had to be knocked together in between doing everything else.” (general practitioner)
“I've been banging my head on the wall of the Royal College of General Practitioners for the last two years to say that I think they ought to take up guideline production … They should produce national guideline skeletons which are then used by local groups … to produce locally relevant guidelines.” (principal general practitioner)
Content and impact on clinical decisions
Between 13 and 100 of the 100 patients with atrial fibrillation would have had anticoagulation treatment depending on which guideline was used (κ for interguideline agreement=0.12) (1). Every patient would have had anticoagulant treatment recommended by at least two guidelines; only one patient had anticoagulant treatment recommended by all. There was substantial intraobserver and interobserver agreement (κ=0.91 and 0.78 respectively).
Seventeen of the 20 guidelines contained advice on the use of echocardiography. However, this varied between advising echocardiography for all patients to advising it only for those with specific features—for example, recent onset atrial fibrillation or murmurs. Age specific advice was included in 13 of the 20 guidelines. The potentially smaller benefit of warfarin in younger patients was mentioned in 11 guidelines; five suggested that patients younger than 65 without other risk factors did not need warfarin, and four suggested this for patients younger than 60. Half of the guidelines mentioned the potentially higher risks of anticoagulant treatment in elderly patients, eight giving upper age limits. Four of these strongly advised against treatment for patients over 80 and two suggested that treatment should be considered in those over 75 only in special circumstances. One guideline suggested that patients over 75 should be treated only if lesions were present on echocardiography, and one that the evidence remained unclear for those aged over 80. Two of the guidelines stated that age was not a contraindication to anticoagulant treatment. A target value or range of the international normalised ratio was included in 15 guidelines (varying between 1.2-1.5 and 2.5-3.0); eight included values <2.
Discussion
This study has shown that there are large variations in the advice included in guidelines on the use of anticoagulants in patients with atrial fibrillation. We found that these differences would affect treatment decisions by applying the guidelines to a sample of patients with atrial fibrillation in the community. The good interobserver and intraobserver agreement suggests that variation in treatment is largely due to variations in the advice offered by guidelines. These differences could have a profound effect on the process and outcome of care and on the incidence of stroke and bleeding complications; they might also lead to substantial differences in the use of resources, particularly anticoagulation services. Our results support a similar finding on the use of different guidelines to treat hypertension in general practice.23
The variation found in the guidelines is likely to be caused by their non-systematic development. Only one guideline could be classified as evidence based—that is, incorporating explicit links between recommendations and the quality of supporting evidence.22 In most cases literature review and appraisal were non-systematic. Though expert opinions and external reviews were sought, the methods of incorporating evidence and opinion into the guidelines were unstructured. Even in cases in which guidelines had been developed as a group process, access to the range of necessary skills was limited and group members often took on several roles.
Those developing guidelines on anticoagulant treatment were apparently unaware of the literature on development and validity; several participants thought that their guidelines were evidence based and that they had graded the evidence, although after the interview and on review of the guidelines this was not the case.
Although the interviews with the lead developers of the guidelines concentrated on the development process, shortcomings in dissemination, implementation, and review of the guidelines were also identified. Almost half of those who developed guidelines used educational initiatives, such as seminars for medical staff, to support dissemination, but strategies for implementation were absent. Planned review and evaluation were rare.
Though we undertook an extensive survey, we will have failed to identify some guidelines. However, those who responded voluntarily are likely to have provided higher quality guidelines than those who declined to provide guidelines. Thus, our conclusions are unlikely to be weakened by the inclusion of guidelines that we initially failed to identify. This is equally true of any potential non-response bias from the interviews.
It is not our intention to criticise those who produced these guidelines. They have contributed considerable time and effort with the explicit intention of improving patient care. However, the resources required to develop valid guidelines are substantial.5 Production of most of these guidelines was inadequately funded. Many of the guidelines suggest courses of action that are not ideal, presumably as a result of errors in interpretation or presentation of the evidence. Many of the respondents identified the shortcomings of their approaches but, in the absence of alternatives, recognised a need to produce guidelines for local use.
These results confirm that guidelines developed non-systematically are likely to be variable in content and impact. Hence, those considering developing guidelines should first explore whether evidence based guidelines already exist and can be adapted for local use. If they do not exist advice should be sought from experts on guideline development and literature appraisal. We suggest that a centralised organisation could develop clinical guidelines in a more cost effective manner than the current system and would be more likely to produce reliable and valid guidelines. The organisation would need to apply rigorous and transparent methods to the development of guidelines, and the guidelines produced would need to be readily accessible. This organisational role could be taken by the NHS Research and Development Organisation or professional organisations such as the royal colleges or specialist associations. Critical appraisal of available guidelines,24 such as that commissioned by the Clinical Outcomes Group of the Department of Health, would lend further support to this approach. Such centrally produced, validated, evidence based guidelines could then be adapted to local circumstances.
Acknowledgments
We thank all those who responded to our postal survey, and Phil Adams, Malcolm Aylett, Francoise Cluzeau, Martin Eccles, Denise Howel, and Rose Anne Kenny for their help and advice.
Funding: MS is supported by an MRC Special Training Fellowship in Health Services Research.
Conflict of interest: MS and RT are now involved in developing evidence based guidelines on the use of anticoagulant treatment in atrial fibrillation using decision analysis.
Notes
Contributors: RT initiated and coordinated the formulation of the study idea and its design; contributed to the assessment of the guidelines, including their application to patients; contributed and advised on the qualitative analysis of the interviews; participated in the interpretation of the results; and led the writing of the paper. MS contributed to the design of the study and to the assessment of the guidelines, participated in data collection, led the analysis of the quantitative application of the guidelines to the series of community based patients, participated in the interpretation of results, and contributed to the writing of the paper. HM contributed to the design of the study, undertook the interviews, led the qualitative assessment of the guidelines, participated in the interpretation of the results, and contributed to the writing of the paper. Carol Fraser helped with management of the postal survey. RT and MS are guarantors for this paper.