Use of systematic reviews in clinical practice guidelines: case study of smoking cessationBMJ 2001; 323 doi: https://doi.org/10.1136/bmj.323.7317.833 (Published 13 October 2001) Cite this as: BMJ 2001;323:833
- a Monash Institute of Health Services Research, Monash Medical Centre, Clayton, Victoria 3168, Australia
- b Cochrane Tobacco Addiction Group, Imperial Cancer Research Fund General Practice Research Group, Department of Primary Health Care, Institute of Health Sciences, Oxford OX3 7LF
- c Department of Primary Health Care, Institute of Health Sciences
- Correspondence to: C Silagy
- Accepted 14 May 2001
Objective: To examine the extent to which recommendations in the national guidelines for the cessation of smoking are based on evidence from systematic reviews of controlled trials.
Design: Retrospective analysis of recommendations for the national guidelines for the cessation of smoking.
Materials: National guidelines in clinical practice on smoking cessation published in English.
Main outcome measures: The type of evidence (systematic review of controlled trials, individual trials, other studies, expert opinion) used to support each recommendation. We also assessed whether a Cochrane systematic review was available and could have been used in formulating the recommendation.
Results: Four national smoking cessation guidelines (from Canada, New Zealand, the United Kingdom, and the United States) covering 105 recommendations were identified. An explicit evidence base for 100%, 89%, 68%, and 98% of recommendations, respectively, was detected, of which 60%, 56%, 59%, and 47% were based on systematic reviews of controlled studies. Cochrane systematic reviews could have been used to develop between 39% and 73% of recommendations but were actually used in 0% to 36% of recommendations. The UK guidelines had the highest proportion of recommendations based on Cochrane systematic reviews.
Conclusions: Use of systematic reviews in guidelines is a measure of the “payback” on investment in research synthesis. Systematic reviews commonly underpinned recommendations in guidelines on smoking cessation. The extent to which they were used varied by country and there was evidence of duplication of effort in some areas. Greater international collaboration in developing and maintaining an evidence base of systematic reviews can improve the efficiency of use of research resources.
Good guidelines in clinical practice may help translate research findings into health gain.1 Guidelines should be based on a valid summary of the available evidence, and it has been shown that Dutch general practitioners were more likely to adopt recommendations that were evidence based than those that were not.2
Systematic reviews are an important source of valid evidence.3 However, undertaking and maintaining systematic reviews of high quality is resource intensive. The Cochrane Collaboration represents an international attempt to organise this type of activity across most fields of health care.4
Use of Cochrane reviews in guidelines for clinical practice is a possible measure of the “payback” on investment in the collaboration, but we do not know the extent to which they are used for this purpose. Grant and colleagues evaluated payback on biomedical research based on papers cited in clinical guidelines.5 They found that only 3% of cited references were systematic reviews.6 However, their approach does not necessarily reflect the extent to which systematic reviews were used to provide the evidence base for formulating specific recommendations within a guidelines document.7
We used smoking cessation as a case study to investigate the extent to which developers of national guidelines make use of systematic reviews. We also aimed to determine the source of systematic reviews used and the proportion of recommendations that could have been developed using an existing Cochrane review. We chose smoking because a large number of systematic reviews on this subject have been available for some time.8
We searched the internet (using the free text terms “smoking cessation” and “guideline”) and the US National Guidelines Clearinghouse (http://www.guideline.gov/) to identify national guidelines on smoking cessation for clinical practice fully published in English.
For each set of guidelines we identified the commissioning agency, the date that the guidelines were initially released, the date of the last search for evidence to be included in the guideline, the date of subsequent review of the guideline, the process for generating the guidelines, and individual recommendations. When there was more than one document associated with the guidelines, we focused on the recommendations contained in the document that targeted practitioners primarily.
We sought to identify the type of evidence used to support each recommendation, categorising evidence as systematic review, individual study(s), consensus of panel members, or not stated. We noted whether systematic reviews cited were Cochrane reviews, other published systematic reviews, or reviews undertaken as part of the development process for guidelines. When there was more than one document and the evidence supporting a recommendation was not identifiable in the primary document targeting practitioners, we examined other available documents.
For each recommendation we assessed whether a Cochrane review could have been used (at the time the guidelines were being prepared) to formulate the recommendation either directly or by extrapolation from the data in the review.
We identified four national guidelines published in English without adaptation from an existing set of guidelines.9–12 Table 1 shows the country and commissioning agency for the four guidelines, together with the dates of release, last search for included evidence, and scheduled review (where recorded). The guidelines for the United States, United Kingdom, and New Zealand were all developed within the past two years. An updated amendment to the UK guidelines was released in late December 2000 (after this study was completed) and was, therefore, not included.13 A review of the New Zealand guidelines was due in June 2001. The Canadian guidelines were developed in 1996 and reviewed during 1998, when the recommendations were still deemed relevant.
All four guidelines had clear documentation about the development process (table 2). In a number of cases this involved review and adaptation of existing material. Both the UK and New Zealand guidelines stated that they drew explicitly on Cochrane systematic reviews. All three non-US guidelines drew on reviews undertaken as part of an earlier US smoking cessation guideline.14 In addition, the guidelines for New Zealand explicitly drew on the UK guidelines. All assessed strength of evidence, although in the New Zealand guidelines this was provided in a separate document from the guideline document, which was targeted at practitioners.15
The number of clearly identifiable recommendations contained in the different guidelines varied from five (Canadian) to 60 (US) (table 3). Of these, we identified an explicit evidence base supporting the recommendations in most cases—that is, 68%, 89%, 98%, and 100%—for the UK, New Zealand, US, and Canadian guidelines, respectively.
A Cochrane systematic review (that is, one that was available at the time of the last search undertaken in assembling the guidelines) could have been used, either directly or indirectly, to formulate 39%-73% of recommendations (table 4). Despite the potential to use Cochrane systematic reviews in this way, the proportion of recommendations specifically supported by a Cochrane review was much lower (0%-36%). The two North American guidelines in particular made little use of Cochrane reviews, with the US guidelines undertaking 26 new meta-analyses as part of the guideline development process. Of these, approximately two thirds covered topics similar to those covered by Cochrane systematic reviews. The UK and Canadian guidelines also relied heavily on published systematic reviews, particularly those from the earlier version of the US guidelines.14
This study of smoking cessation shows that guideline developers provide detailed information about the steps involved in the development process. They were careful to ensure that most recommendations had an explicit evidence base. However, systematic reviews were used to support only about half of the total number of recommendations (and they supported a slightly higher percentage of those recommendations that mentioned an explicit evidence base). Of these, the proportion of recommendations based on a Cochrane review compared with a non-Cochrane systematic review varied depending on the country of origin of the guideline.
Generally, the North American guidelines drew on Cochrane reviews to a lesser extent than UK guidelines did. With the New Zealand guidelines the proportion of recommendations based on evidence from Cochrane systematic reviews varied depending on which of the two documents was considered. The document on literature review and background information makes extensive use of Cochrane reviews.15 The reason that the practitioner document makes less use of Cochrane reviews probably reflects the fact that several of the recommendations related to an area where there are currently no Cochrane reviews—for example, the establishment of systems of care (including prompting) to encourage practitioners to ask their patients about their smoking habits and to advise in a systematic way.
The reason for the relative lack of Cochrane reviews in the two guidelines developed in North America is unclear. One explanation could be the availability of Cochrane reviews in relation to the timing of developing the different guidelines. The initial release of the Canadian guidelines preceded the availability of any Cochrane reviews, but the review of these guidelines in 1998 could have drawn on a number of Cochrane reviews. Similarly, the US Agency for Health Care Policy and Research's 1996 smoking cessation guidelines (http://www.ahcpr.gov/) were prepared before any Cochrane reviews were available. The US 2000 guidelines specifically mention consideration of other meta-analyses (and reference the entire Cochrane Library) in the description of methods.9 However, the evidence base draws mainly on updates of meta-analyses previously undertaken for the 1996 guidelines. This suggests that there is scope for improving international cooperation to reduce duplication of effort (and hence cost) in research synthesis.
Although the present study is based on a small sample, the guidelines mentioned have been widely adopted and used around the world by other national organisations.16 The extent to which the findings generalise to other topics is an area for further research.
We do not suggest that all clinical practice guidelines must use Cochrane systematic reviews as their primary source of evidence, or that Cochrane reviews are necessarily better quality than other systematic reviews. However, smoking cessation is an area in which a sizeable body of evidence is organised and updated systematically by the Cochrane Collaboration. In contrast with Grant et al 5 6 we found, in the case of smoking cessation, that the recommendations of UK guidelines made extensive use of Cochrane systematic reviews. This is one source of evidence of the “payback” for the investment of funding organisations in supporting the systematic review process.17
Finally, we recognise that not all guideline recommendations can be underpinned by systematic reviews. Guideline developers can and should make recommendations that are specific to local contexts. However, reliable, up to date summaries of the evidence available internationally can allow guideline developers to reduce the costs associated with local guideline development.
Dr Mike Clarke and Dr Iain Chalmers provided helpful comments and feedback on drafts of the manuscript.
The Cochrane tobacco addiction review group is supported by the NHS Research and Development Programme and the Imperial Cancer Research Fund.
Contributors: CS and TL conceived the study and developed the protocol. CS identified the guidelines and initially extracted the data. LS checked the data. All authors contributed to writing the manuscript. CS is the guarantor for the paper. CS extracted and classified the data, resolving uncertainties by discussion with LS and TL.
Funding None received for this specific project. TL and LS receive funds from the NHS research and development programme and the general practice research group of the Imperial Cancer Research Fund to support the infrastructure of the Cochrane tobacco addictions group.
Competing interests CS and TL are editors of the Cochrane tobacco addiction group and LS is the review group coordinator of the same group. None of the authors stand to benefit directly or indirectly, financially, from increased use of the Cochrane Library. They have all been invited to provide comments and feedback on the US and UK guidelines as part of the process of the guidelines' development.