What's the evidence that NICE guidance has been implemented? Results from a national evaluation using time series analysis, audit of patients' notes, and interviews

BMJ 2004; 329 doi: http://dx.doi.org/10.1136/bmj.329.7473.999 (Published 28 October 2004)
Cite this as: BMJ 2004;329:999
  1. Trevor A Sheldon, professor (tas5{at}york.ac.uk)1,
  2. Nicky Cullum, professor1,
  3. Diane Dawson, senior research fellow2,
  4. Annette Lankshear, senior research fellow1,
  5. Karin Lowson, deputy director3,
  6. Ian Watt, professor1,
  7. Peter West, director3,
  8. Dianne Wright, research fellow3,
  9. John Wright, operations medical director4
  1. 1 Department of Health Sciences, University of York, York YO10 5DD
  2. 2 Centre for Health Economics, University of York
  3. 3 York Health Economics Consortium, University of York
  4. 4 Bradford Royal Infirmary, Bradford, BD9 6RJ.
  1. Correspondence to: T A Sheldon

    Abstract

    Objectives To assess the extent and pattern of implementation of guidance issued by the National Institute for Clinical Excellence (NICE).

    Design Interrupted time series analysis, review of case notes, survey, and interviews.

    Setting Acute and primary care trusts in England and Wales.

    Participants All primary care prescribing, hospital pharmacies; a random sample of 20 acute trusts, 17 mental health trusts, and 21 primary care trusts; and senior clinicians and managers from five acute trusts.

    Main outcome measures Rates of prescribing and use of procedures and medical devices relative to evidence based guidance.

    Results 6308 usable patient audit forms were returned. Implementation of NICE guidance varied by trust and by topic. Prescribing of some taxanes for cancer (P < 0.002) and orlistat for obesity (P < 0.001) significantly increased in line with guidance. Prescribing of drugs for Alzheimer's disease and prophylactic extraction of wisdom teeth showed trends consistent with, but not obviously a consequence of, the guidance. Prescribing practice often did not accord with the details of the guidance. No change was apparent in the use of hearing aids, hip prostheses, implantable cardioverter defibrillators, laparoscopic hernia repair, and laparoscopic colorectal cancer surgery after NICE guidance had been issued.

    Conclusions Implementation of NICE guidance has been variable. Guidance seems more likely to be adopted when there is strong professional support, a stable and convincing evidence base, and no increased or unfunded costs, in organisations that have established good systems for tracking guidance implementation and where the professionals involved are not isolated. Guidance needs to be clear and reflect the clinical context.

    Introduction

    The National Institute for Clinical Excellence (NICE), established in 1999 by the Department of Health, aims to improve standards of care for patients and reduce inequalities in access to innovative treatments.1 NICE's technology appraisals result in guidance on the use of individual health technologies; around 50 for implementation by the NHS in England and Wales since 2000.2 Hitherto, guidance on treatments was uncoordinated. This nationally coordinated programme of guidance is probably unique and represents a policy embodiment of evidence based medicine that, it is hoped, will lead to the rapid and systematic uptake of evidence based medicine into routine practice. This paper presents the results of a national evaluation examining the pattern of implementation of NICE guidance by healthcare organisations. More details on NICE and a copy of the full report can be accessed via www.nice.org.uk/).

    Methods

    We assessed the response of the NHS to 12 pieces of “tracer” NICE guidance. We selected guidance for study if at least six months had elapsed since its release. By the time the research began in October 2001, 22 sets of guidance were eligible for inclusion (table 1): four procedures (four selected), five devices (three selected), one diagnostic test, and 11 pharmaceutical drugs (five selected). We audited 11 (50%) of the eligible sets of guidance, chosen to reflect a range of drugs, devices, and procedures; different care settings; and cost consequences. Some included clear stopping messages (wisdom teeth, laparoscopic surgery for colorectal cancer); some fairly clear messages to use a technology (implantable cardioverter defibrillators, hearing aids); and other complex messages regarding appropriate use (hip prostheses, taxanes for breast cancer, orlistat).

    View this table:
    Table 1

    NICE guidance sampling for audit

    The research consisted of three phases, each one using a different method of data collection to answer different but complementary questions.

    Phase I

    We analysed routine national or regional data and national surveys (centrally collected data; box 1) to assess the extent to which practice changed after publication of the tracer guidance. We used data from the NHS Prescription Pricing Authority, which covers all of England, to explore trends in the primary care prescribing of relevant drugs. We used hospital episode statistics data (covering England and Wales) to assess the trends in use of invasive procedures. We accessed or directly collected a range of other sources of national and regional data where these two sources did not have relevant or adequate data (box 1). Table 2 shows data sources used for each of the 12 sets of guidance.

    View this table:
    Table 2

    Overview of the 12 sets of guidance selected as “tracer” guidance, research questions, and data sources

    We used interrupted time series analysis to assess if the pattern of practice had changed after NICE guidance3 and so infer whether the intervention had an impact. We used an autoregressive integrated moving average (ARIMA) model with dummy variables to examine the impact of publication on the growth rate and the average rate of use (constant parameter) of a technology once any growth rate had been removed.4

    Phase II

    Most NICE guidance provides criteria for appropriate use rather than a simple recommendation to use the technology or not. To assess whether the guidance was being implemented properly requires scrutiny of patient records by using methods developed for appropriateness studies.5 6 We selected a random sample of 20 (out of 221) acute trusts and their associated mental health hospitals and 21 (out of 303) primary care trusts (within which we selected a stratified random sample of five practices), from which we reviewed 50 relevant case notes for each of eight guidance topics (table 2). The resulting trusts reflected a good cross selection of geographical spread and size.

    Box 1: Other sources of centrally collected data

    • Dental Practice Board and the Scottish Practitioner Services data for national information on extractions of wisdom teeth in the community

    • Trent Arthroplasty Audit Group and Welsh Arthroplasty Audit Group Database (TWAAG) for information on prostheses used in hip replacement. This register of knee and hip replacements covers the former NHS Trent Region and the North Wales region contained the records of 7898 patients who had received hip replacements since January 1998 in 22 hospitals (accounting for 40% of the hip replacements undertaken in these hospitals)

    • British Pacing and Electro-physiology Group ICD Register for information on implantable cardioverter defibrillators for arrhythmias. The data, submitted to the register by hospital clinicians, covered the period from the first quarter of 1995 (33 centres) to the last quarter of 2001 (58 centres) and represents about 95% of activity

    • A new survey of hearing aid provision sent to all 228 audiology departments in England and Wales (50% response rate)

    • A new survey of all 331 hospital pharmacies in England and Wales (68% response rate and 60% with usable data)

    Local audit staff agreed to extract data from patients' records by using audit proforma. Overall usable audit forms for 6308 patients were returned (table 3). We calculated the proportion of cases conforming to the NICE guidance for each healthcare organisation in the sample at two periods of time and estimated the overall average.

    View this table:
    Table 3

    Response from audit of patient case notes with reasons for failure to complete

    Phase III

    We surveyed the chief executives, leads of clinical governance, and leads of clinical specialties of the 20 acute trusts that participated in the audit of patients' notes (65% response rate for chief executive officers (13 out of 20) and 57% overall (68 out of 120)) to assess their handling of NICE guidance. We also conducted semistructured interviews to access professional and managerial perspectives on our quantitative findings and, in particular, to explore the response to the NICE guidance.

    We purposively selected five acute trusts7 that returned positive consent forms, to represent differing degrees of implementation of guidance (see phase II). In each trust we approached the chief executive, medical director, and lead clinicians for the guidance topic. We supplemented a common interview schedule (see appendix on bmj.com) by specific interview questions based on findings from the audit. We offered all interviewees face to face interviews but finally conducted three interviews by telephone. Where possible we recorded and transcribed interviews. Where recording proved impossible we returned notes to the interviewee for checking.

    Data analysis was concurrent, and clear thematic categories and subcategories emerged.8 9 Two researchers working independently developed these categories and compared and reconciled differences by discussion. We used analytic matrices to examine differences between trusts and across sets of guidance.10

    Results

    Wisdom teeth extraction

    National data indicate a sharp decline in the number of extractions between 1995 and 2001 (fig 1). Although these fell in the year in which the guidance had been published (in March 2000), we found no evidence of a change in the downward trend in extractions (−8.9 extractions per month, 95% confidence interval −57.5 to 39.6). Case note review showed that more than 90% of extractions were compliant with the guidance. Survey respondents indicated that compliance was high because the costs of implementation were low, involved a single specialty service, and had professional support and a strong evidence base.

    Fig 1

    Wisdom teeth extraction activity, 1992-2001

    Hip prostheses

    The guidance recommended the use of prostheses with a demonstrable replacement rate of 10% or less, at 10 years, or a minimum of three years, provided that the performance of the prostheses is consistent with the 10 year benchmark. The guidance did not specify which prostheses met the benchmark and the NHS Purchasing and Supply Agency was slow to issue this information (which was subsequently withdrawn).

    Use of more than 50 different prostheses was documented in the Trent and Wales register over the period 1998-2002. Of the single prostheses, 69% (3671) met the 10 year benchmark and 81% (4327) met the three year benchmark, proportions that had been declining over recent years. There is some evidence that the decline in 10 year benchmarked prostheses may have stabilised slightly after guidance (1.7 monthly increase after guidance (95% confidence interval 0.4 to 3.1) compared with a long term monthly decrease of −1.8 prostheses per month (−0.75 to −2.80)). We observed similar results from the audit of patients' notes: 66% (325), 74% (727), and 75% (734) of heads, cups, and stems, respectively, meeting the benchmarks. Interviews and the survey indicated that surgeons thought that the guidance did not acknowledge the complexity of hip surgery, and some surgeons interviewed believed that cementless prostheses would ultimately prove to offer longer service.

    Taxanes for breast and ovarian cancer

    We obtained usable data on taxanes from 24 hospital pharmacies (including nine cancer centres). We found a significant increase in the use of docetaxel and paclitaxel of 1112 (95% confidence interval 530 to 2222, P < 0.001) patient months and 3.7 (1.1 to 7.8, P < 0.002) patient months, respectively (fig 2), but no evidence of a change in the growth rate in the use of gemcitabine (0.5% per month, −36.2% to 37.2%) or vinorelbine (−1.0 patient months, −3.3 to1.3). This is unlikely to be due to lack of statistical power.

    Fig 2

    Total number of implantable cardioverter defibrillators implanted by quarter, 1995-2001

    Case note review showed that, of the 707 patients identified as receiving taxanes for breast cancer, all but one were receiving this appropriately (for more advanced forms of the disease or in the context of randomised controlled trials). Interviewees acknowledged that the NICE guidance had made funding easier to obtain. The picture for ovarian cancer, however, was more varied. Of the 520 women with ovarian cancer for whom we have data, only 33% (166) were recorded as having been prescribed paclitaxel. However, oncologists interviewed believed that the guidance had overstated the effectiveness of taxanes in ovarian cancer. NICE subsequently amended its guidance.

    Implantable cardioverter defibrillators

    Although the number of implantable cardioverter defibrillators implanted has risen, we found no evidence of a significant change after NICE guidance had been published (fig 3). Given the small data set, the power is low to detect a change as significant, but visual inspection does not indicate any structural break. This may reflect the high costs of implantable cardioverter defibrillators, at around £20 000 ($36 000; 29) per device, competition for resources with other interventional procedures in cardiology, and scarcity of skills in electrophysiology.11

    Fig 3

    Use of orlistat in the community

    Hearing aids

    The NICE guidance seems to have been received enthusiastically by audiology departments; all of those surveyed had undertaken an immediate audit of their service against the guidance requirements. However, review of case notes indicates that the range of analogue hearing aids offered does not seem to have been extended. Funding was described in the interviews as a major impediment to implementation. The guidance was issued at the same time as the Department of Health implemented a series of pilots of digital hearing aids, which cut across the guidance on analogue aids, which was subsequently withdrawn.

    Laparoscopic surgery for primary inguinal hernia repair and colorectal cancer

    Only 4% of primary inguinal hernia repairs in England and Wales were undertaken laparoscopically (contrary to NICE guidance), and this did not change after the guidance (0.3 monthly increase in hernia repairs, 95% confidence interval −5.48 to 6.08).

    Although hospital episode statistics data for the 19 trusts that returned audit forms also showed 96% compliance, our audit of 545 repairs of primary unilateral hernias indicated only 65% compliance, indicating that coding of hospital episode statistics data may be unreliable. However, both national and audit data agreed that most laparoscopic procedures were concentrated in a few trusts, and that did not change over time.

    Interviews showed that some local expert surgeons had the support of managers and commissioners to continue the use of laparoscopic surgery for primary repair. It was also claimed that patients often requested laparoscopic procedures.

    The percentage of cases of colorectal cancer treated with laparoscopic surgery remained unchanged, at around 0.1% from 1998 to 2001.

    Zanamivir for influenza

    National prescribing data show little inappropriate prescribing of zanamivir in the absence of high levels of flu; prescriptions remained very low, at 499 in 2001, 190 in 2002, and 124 in 2003.

    Orlistat for obesity

    We found a significant increase in the average monthly prescribing of orlistat after the guidance had been published 22 per month (0.43, 95% confidence interval 15.9 to 27.8, P < 0.001; fig 4). Health authorities increased their use of orlistat, standardised by age, in the year after NICE guidance had been issued by about eight patient months per 1000 people aged 18-75 years. The variation in use, as measured by the coefficient of variation, fell from 0.4 in 2000-1 to 0.3 in 2001-2.

    Fig 4

    Use of orlistat in the community

    In the 689 primary care patients for whom we have a record we found evidence that the drug was not being prescribed in accordance with guidance. Only in 12% of cases (n = 83) were there data showing compliance in the three key areas of age, body mass index, and weight loss. However, data recording was poor, and data were missing in one or more fields in 80% (551) of returns, particularly in respect of weight and weight loss. In those 308 cases where patients' weight loss prior to the visit had been recorded fully, the weight loss criterion was not met in 40% (123) of cases. Of patients with recorded weight loss data, only 41% (127) and 25% (40) still being prescribed orlistat at 3 months and 6 months, respectively, had reduced their weight as advised in the NICE guidance. The rise in prescribing of orlistat therefore does not necessarily imply a rise in appropriate prescribing.

    Drugs for Alzheimer's disease

    Total use grew logarithmically since February 1999 (fig 5). Once this had been taken into account, the growth rate at the time of publication of guidance (−1.4% per month; 95% confidence interval −4% to 1.2%) did not increase significantly. However, the data are also compatible with a view that an increase in trend occurred shortly before the guidance was formally issued. Variability between health authorities decreased; the coefficient of variation fell from 1.1 to 0.97.

    Fig 5

    Total use of Alzheimer's drugs in the community

    Based on the audit data from 583 usable forms, compliance with the five recommendations in the guidance at first prescription varies between 52% and 85% for mental health organisations and 21% and 46% in primary care. Compliance with the recommendations at follow up is low, mainly because information regarding mini-mental state examination scores, follow ups, or the presence of relatives at assessment was not routinely recorded (a requirement of the guidance).

    Discussion

    Principal findings

    The evidence that NICE guidance has made a difference either to the quality of care or to variations in practice is mixed. Some NICE guidance has been associated in time with changes in prescribing. Use of orlistat and taxanes grew rapidly after NICE guidance had been published, and uptake of drugs to treat Alzheimer's disease also increased, although this slightly preceded the release of that guidance. The guidance for wisdom teeth was published during a long downward trend in the extraction rate and did not have a discernable additional effect. The guidance for hips in general showed no effect. We found no evidence of a change in the number of implantable cardioverter defibrillators used, the numbers of inguinal hernias or colorectal cancers treated laparoscopically, or expansion in the range of hearing aids made available. In some cases audit showed that clinical practice was highly compliant with the indications for treatment laid out in the guidance (for example, wisdom teeth and the use of taxanes for breast cancer), but compliance was low for some (such as orlistat) and more variable for others. Some trusts seemed to exhibit more consistent compliance than others across a range of guidance, and box 2 shows their characteristics, although all identified funding as a major issue, especially where infrastructure costs were high.

    Strengths and weaknesses of the study

    In a retrospective observational study we cannot fully assess the impact of NICE guidance because of the absence of the counter-factual and because NICE guidance is just one, although potentially important, factor influencing professional practice. We can only observe whether clinical practice is consistent with the guidance. This research was based on a large data gathering exercise, and although data taken from national routine hospital and primary care are complete, those obtained from hospital pharmacies were less so. Where data were recorded, the audits allowed us to assess more reliably whether the details of practice corresponded to the indications of appropriateness in the guidance compared with subjective surveys12 and analyses based simply on routine data.13 It was, however, sometimes difficult to design audits where the original wording of the NICE guidance had been complex or ambiguous. This is also likely to affect the ease with which the guidance is implemented. Our survey also offered a view of the management context into which the guidance was received, and the interviews were useful in highlighting the managerial, financial, and clinical perspectives on implementation.

    Meaning of the study

    The establishment of NICE as a mechanism for institutionalising evidence based health care was a unique initiative. This institutional response by itself is, however, not sufficient for the rapid and universal implementation of evidence based health care. This is unsurprising; NICE guidance is being issued at a time of great change in the NHS, with new structures, competing high level priorities, funding deficits, and staff shortages. In addition, healthcare organisations are complex, containing strong professional bureaucracies.14 Change is therefore heavily dependent on the actions of groups of professionals and individual clinicians.15 The ability to manage change in an organisation is further complicated by the increasing importance of networks that span several organisations.

    The diffusion of innovations literature indicates that the adoption of guidance would depend on several other factors (box 3; after Rogers).16 17 These also emerged in the thematic analysis of the interview data (table 4). In areas where practitioners and managers see advantages to adoption, where the value is hardly disputed (clear evidence), and where there is professional endorsement (taxanes for breast cancer, wisdom teeth extraction, and orlistat for obesity) practice has changed relatively fast. The marketing activity of the pharmaceutical industry should not be overlooked as a possible explanation for the apparent increased uptake of drugs over devices and changes in surgical behaviour.

    View this table:
    Table 4

    Analysis of interviews: factors influencing likelihood of implementation

    Box 2: Features of trusts consistent with high compliance

    • Commitment to managing process of implementing guidance

    • Identification of lead clinician at point of NICE announcement of topic for review

    • Proactive assessment of local costs and implications of implementation

    • Responsibility for funding and implementation vested in locality-wide group

    • Strong clinical governance function appropriately resourced

    • Culture of consensus

    • Recognition of legitimacy of NICE

    • Involvement of clinicians in guideline process

    • Financial stability

    • Expectation that compliance is mandatory, subject to identification of funding

    • Targeted audit of areas of non-compliance

    Box 3: Factors influencing adoption of innovation (after Rogers16)

    • Perceived attributes and consequences of adoption (for example, relative advantage, complexity, observability)

    • Type of innovation decision (optional, collective authority)

    • Communication channels (for example, media, interpersonal, professional)

    • Nature of the social system (norms, degree of interconnectedness of networks, concentration of opinion leaders)

    • Extent of promotion efforts by agents of change

    If the evidence has been disputed, or the costs not covered by increased income, adoption is more variable. Where practice is complex and also depends on an interaction with a practitioner's skills (such as with hip replacements or laparoscopic surgery), more formative research to understand the clinical context of practice may have helped to produce more influential guidance. Adoption may also be influenced by the degree to which decisions rest with an individual or requires team or organisational agreement. Individual surgeons are likely to have a high degree of professional autonomy, and this may partly explain the variability in techniques for hernia repair and hip replacement.

    The extent to which trusts are prepared for NICE guidance and have put in place structures and processes to manage their implementation was variable.

    The degree of active promotion by NICE is likely to have some impact on adoption, although probably not directly proportional to the effort invested. The greatest effect is likely when opinion leaders including the professional bodies and associations adopt and promote the guidance.

    What is already known about this topic

    Research on the implementation of guideline implementation has been summarised, but its relevance to this unique national initiative was unknown

    The implementation of NICE guidance has not been evaluated overall. Previous work has been limited to single health technologies

    What this study adds

    Some clinical practice has changed in line with NICE guidance, in particular around prescribing (for example taxanes and orlistat)

    Other technologies have been adopted in line with NICE guidance but continued pre-existing practice patterns

    There is evidence that NICE guidance has been less influential in surgical procedures and use of medical devices

    Routine data are not sufficient to assess compliance with guidance; this needs review of case notes

    NICE guidance seems to have had an uneven impact on the uptake of evidence based medicine. This impact is likely to be greater if more effort is devoted to clarity of the guidance and its relevance to practice; adequate funding provision; getting professional support; and encouraging healthcare organisations to set up formal mechanisms for handling guidance

    Unanswered questions and future research

    Our research covers the early period of NICE guidance; it would be interesting to see if the response to subsequent guidance is different, particularly given the recent attention that NICE is giving to implementation. More research would also be useful to understand the professional and organisational responses to evidence based guidance better and to evaluate the relative contributions of various implementation strategies to practice patterns.17

    Conclusions

    NICE guidance has been associated with uptake of some technologies, although this has been variable. Implementation is likely to be improved if the guidance is clear and based on an understanding of clinical practice, if the evidence is strong and relatively stable, if adequate funding is available, and if the guidance is supported and disseminated by professional bodies. Trusts should institute strong supportive internal systems for handling guidance and gathering data on implementation.

    Acknowledgments

    We acknowledge the support and cooperation of the NHS trusts that provided data, the audit staff and clinicians who carried out the case note audits, and the clinicians and managers who responded to the survey and participated in interviews. We thank the British Pacing and Electrophysiology Group (BPEG) for anonymised access to their register of Implantable Cardioverter Defibrillators and to the Trent and Welsh Arthroplasty Audit Groups for access to their anonymised database (TWAAG). We thank David Gibbons for advice on aspects of wisdom tooth removal and related issues and staff from the MRC Institute for Hearing Research who provided advice on the survey of audiology departments.

    Contributors: NC contributed to study concept and design, study supervision, analysis and interpretation of the data, and manuscript preparation. DD contributed to the study concept and design, study supervision, analysis and interpretation of the data, and manuscript preparation. AL contributed to the design, conduct, and analysis of case note review; surveys and interviews; and manuscript preparation. KL contributed to the study concept and design, data collection, analysis and interpretation of the data, and manuscript preparation. James Mahon contributed to the statistical analysis of the data. Pauline Raynor contributed to the design and conduct of the primary care case note review. TAS contributed to study concept and design, study supervision, analysis and interpretation of the data, and manuscript preparation; he is guarantor. IW contributed to study concept and design, study supervision, analysis and interpretation of the data, and manuscript preparation. PW contributed to study concept and design, study supervision, analysis and interpretation of the data and manuscript preparation. DW contributed to the data collection and analysis. John Wilson contributed to the analysis of primary care and hospital pharmaceutical data. JW contributed to study concept and design, study supervision, analysis and interpretation of the data, and manuscript preparation.

    Footnotes

    • Embedded Image The interview schedule for clinicians is on bmj.com

    • Funding NHS R&D National Co-ordinating Centre for Research Methodology (NCCRM).

    • Competing interests NC was a member of the NICE Appraisals Committee between 1999 and 2002. KL, PW, DW, and JM work for York Health Economics Consortium, which undertakes work for a range of pharmaceutical companies, the Department of Health, and the NHS and has undertaken a cost-effectiveness study for Guidant, which manufactures implantable cardioverter defibrillators. This study was submitted to NICE as part of the assessment process.

    • Ethical approval North West Multicentre Research Ethics Committee.

    References

    1. 1.
    2. 2.
    3. 3.
    4. 4.
    5. 5.
    6. 6.
    7. 7.
    8. 8.
    9. 9.
    10. 10.
    11. 11.
    12. 12.
    13. 13.
    14. 14.
    15. 15.
    16. 16.
    17. 17.

    THIS WEEK'S POLL