BMJ  2004;329:999 (30 October), doi:10.1136/bmj.329.7473.999

Paper

What's the evidence that NICE guidance has been implemented? Results from a national evaluation using time series analysis, audit of patients' notes, and interviews

¹ Department of Health Sciences, University of York, York YO10 5DD, ² York Health Economics Consortium, University of York, ³ Centre for Health Economics, University of York, ⁴ Bradford Royal Infirmary, Bradford, BD9 6RJ

Abstract

Objectives To assess the extent and pattern of implementation of guidance issued by the National Institute for Clinical Excellence (NICE).

Design Interrupted time series analysis, review of case notes, survey, and interviews.

Setting Acute and primary care trusts in England and Wales.

Participants All primary care prescribing, hospital pharmacies; a random sample of 20 acute trusts, 17 mental health trusts, and 21 primary care trusts; and senior clinicians and managers from five acute trusts.

Main outcome measures Rates of prescribing and use of procedures and medical devices relative to evidence based guidance.

Results 6308 usable patient audit forms were returned. Implementation of NICE guidance varied by trust and by topic. Prescribing of some taxanes for cancer (P < 0.002) and orlistat for obesity (P < 0.001) significantly increased in line with guidance. Prescribing of drugs for Alzheimer's disease and prophylactic extraction of wisdom teeth showed trends consistent with, but not obviously a consequence of, the guidance. Prescribing practice often did not accord with the details of the guidance. No change was apparent in the use of hearing aids, hip prostheses, implantable cardioverter defibrillators, laparoscopic hernia repair, and laparoscopic colorectal cancer surgery after NICE guidance had been issued.

Conclusions Implementation of NICE guidance has been variable. Guidance seems more likely to be adopted when there is strong professional support, a stable and convincing evidence base, and no increased or unfunded costs, in organisations that have established good systems for tracking guidance implementation and where the professionals involved are not isolated. Guidance needs to be clear and reflect the clinical context.

Introduction

The National Institute for Clinical Excellence (NICE), established in 1999 by the Department of Health, aims to improve standards of care for patients and reduce inequalities in access to innovative treatments.¹ NICE's technology appraisals result in guidance on the use of individual health technologies; around 50 for implementation by the NHS in England and Wales since 2000.² Hitherto, guidance on treatments was uncoordinated. This nationally coordinated programme of guidance is probably unique and represents a policy embodiment of evidence based medicine that, it is hoped, will lead to the rapid and systematic uptake of evidence based medicine into routine practice. This paper presents the results of a national evaluation examining the pattern of implementation of NICE guidance by healthcare organisations. More details on NICE and a copy of the full report can be accessed via www.nice.org.uk.

Methods

We assessed the response of the NHS to 12 pieces of "tracer" NICE guidance. We selected guidance for study if at least six months had elapsed since its release. By the time the research began in October 2001, 22 sets of guidance were eligible for inclusion (table 1): four procedures (four selected), five devices (three selected), one diagnostic test, and 11 pharmaceutical drugs (five selected). We audited 11 (50%) of the eligible sets of guidance, chosen to reflect a range of drugs, devices, and procedures; different care settings; and cost consequences. Some included clear stopping messages (wisdom teeth, laparoscopic surgery for colorectal cancer); some fairly clear messages to use a technology (implantable cardioverter defibrillators, hearing aids); and other complex messages regarding appropriate use (hip prostheses, taxanes for breast cancer, orlistat).

View this table: [in this window] [in a new window]   Table 1 NICE guidance sampling for audit

 

The research consisted of three phases, each one using a different method of data collection to answer different but complementary questions.

Phase I
We analysed routine national or regional data and national surveys (centrally collected data; box 1) to assess the extent to which practice changed after publication of the tracer guidance. We used data from the NHS Prescription Pricing Authority, which covers all of England, to explore trends in the primary care prescribing of relevant drugs. We used hospital episode statistics data (covering England and Wales) to assess the trends in use of invasive procedures. We accessed or directly collected a range of other sources of national and regional data where these two sources did not have relevant or adequate data (box 1). Table 2 shows data sources used for each of the 12 sets of guidance.

View this table: [in this window] [in a new window]   Table 2 Overview of the 12 sets of guidance selected as "tracer" guidance, research questions, and data sources

 

We used interrupted time series analysis to assess if the pattern of practice had changed after NICE guidance³ and so infer whether the intervention had an impact. We used an autoregressive integrated moving average (ARIMA) model with dummy variables to examine the impact of publication on the growth rate and the average rate of use (constant parameter) of a technology once any growth rate had been removed.⁴

Phase II
Most NICE guidance provides criteria for appropriate use rather than a simple recommendation to use the technology or not. To assess whether the guidance was being implemented properly requires scrutiny of patient records by using methods developed for appropriateness studies.^{5 6} We selected a random sample of 20 (out of 221) acute trusts and their associated mental health hospitals and 21 (out of 303) primary care trusts (within which we selected a stratified random sample of five practices), from which we reviewed 50 relevant case notes for each of eight guidance topics (table 2). The resulting trusts reflected a good cross selection of geographical spread and size.

Box 1: Other sources of centrally collected data Dental Practice Board and the Scottish Practitioner Services data for national information on extractions of wisdom teeth in the community Trent Arthroplasty Audit Group and Welsh Arthroplasty Audit Group Database (TWAAG) for information on prostheses used in hip replacement. This register of knee and hip replacements covers the former NHS Trent Region and the North Wales region contained the records of 7898 patients who had received hip replacements since January 1998 in 22 hospitals (accounting for 40% of the hip replacements undertaken in these hospitals) British Pacing and Electro-physiology Group ICD Register for information on implantable cardioverter defibrillators for arrhythmias. The data, submitted to the register by hospital clinicians, covered the period from the first quarter of 1995 (33 centres) to the last quarter of 2001 (58 centres) and represents about 95% of activity A new survey of hearing aid provision sent to all 228 audiology departments in England and Wales (50% response rate) A new survey of all 331 hospital pharmacies in England and Wales (68% response rate and 60% with usable data)

Local audit staff agreed to extract data from patients' records by using audit proforma. Overall usable audit forms for 6308 patients were returned (table 3). We calculated the proportion of cases conforming to the NICE guidance for each healthcare organisation in the sample at two periods of time and estimated the overall average.

View this table: [in this window] [in a new window]   Table 3 Response from audit of patient case notes with reasons for failure to complete

 

Phase III
We surveyed the chief executives, leads of clinical governance, and leads of clinical specialties of the 20 acute trusts that participated in the audit of patients' notes (65% response rate for chief executive officers (13 out of 20) and 57% overall (68 out of 120)) to assess their handling of NICE guidance. We also conducted semistructured interviews to access professional and managerial perspectives on our quantitative findings and, in particular, to explore the response to the NICE guidance.

We purposively selected five acute trusts⁷ that returned positive consent forms, to represent differing degrees of implementation of guidance (see phase II). In each trust we approached the chief executive, medical director, and lead clinicians for the guidance topic. We supplemented a common interview schedule (see appendix on bmj.com) by specific interview questions based on findings from the audit. We offered all interviewees face to face interviews but finally conducted three interviews by telephone. Where possible we recorded and transcribed interviews. Where recording proved impossible we returned notes to the interviewee for checking.

Data analysis was concurrent, and clear thematic categories and subcategories emerged.^{8 9} Two researchers working independently developed these categories and compared and reconciled differences by discussion. We used analytic matrices to examine differences between trusts and across sets of guidance.¹⁰

Results

Wisdom teeth extraction
National data indicate a sharp decline in the number of extractions between 1995 and 2001 (fig 1). Although these fell in the year in which the guidance had been published (in March 2000), we found no evidence of a change in the downward trend in extractions (-8.9 extractions per month, 95% confidence interval -57.5 to 39.6). Case note review showed that more than 90% of extractions were compliant with the guidance. Survey respondents indicated that compliance was high because the costs of implementation were low, involved a single specialty service, and had professional support and a strong evidence base.

View larger version (43K): [in this window] [in a new window]   Fig 1 Wisdom teeth extraction activity, 1992-2001

 

Hip prostheses
The guidance recommended the use of prostheses with a demonstrable replacement rate of 10% or less, at 10 years, or a minimum of three years, provided that the performance of the prostheses is consistent with the 10 year benchmark. The guidance did not specify which prostheses met the benchmark and the NHS Purchasing and Supply Agency was slow to issue this information (which was subsequently withdrawn).

Use of more than 50 different prostheses was documented in the Trent and Wales register over the period 1998-2002. Of the single prostheses, 69% (3671) met the 10 year benchmark and 81% (4327) met the three year benchmark, proportions that had been declining over recent years. There is some evidence that the decline in 10 year benchmarked prostheses may have stabilised slightly after guidance (1.7 monthly increase after guidance (95% confidence interval 0.4 to 3.1) compared with a long term monthly decrease of -1.8 prostheses per month (-0.75 to -2.80)). We observed similar results from the audit of patients' notes: 66% (325), 74% (727), and 75% (734) of heads, cups, and stems, respectively, meeting the benchmarks. Interviews and the survey indicated that surgeons thought that the guidance did not acknowledge the complexity of hip surgery, and some surgeons interviewed believed that cementless prostheses would ultimately prove to offer longer service.

Taxanes for breast and ovarian cancer
We obtained usable data on taxanes from 24 hospital pharmacies (including nine cancer centres). We found a significant increase in the use of docetaxel and paclitaxel of 1112 (95% confidence interval 530 to 2222, P < 0.001) patient months and 3.7 (1.1 to 7.8, P < 0.002) patient months, respectively (fig 2), but no evidence of a change in the growth rate in the use of gemcitabine (0.5% per month, -36.2% to 37.2%) or vinorelbine (-1.0 patient months, -3.3 to1.3). This is unlikely to be due to lack of statistical power.

View larger version (39K): [in this window] [in a new window]   Fig 2 Hospital use of paclitaxel (top) and docetaxel (bottom)

 

Case note review showed that, of the 707 patients identified as receiving taxanes for breast cancer, all but one were receiving this appropriately (for more advanced forms of the disease or in the context of randomised controlled trials). Interviewees acknowledged that the NICE guidance had made funding easier to obtain. The picture for ovarian cancer, however, was more varied. Of the 520 women with ovarian cancer for whom we have data, only 33% (166) were recorded as having been prescribed paclitaxel. However, oncologists interviewed believed that the guidance had overstated the effectiveness of taxanes in ovarian cancer. NICE subsequently amended its guidance.

Implantable cardioverter defibrillators
Although the number of implantable cardioverter defibrillators implanted has risen, we found no evidence of a significant change after NICE guidance had been published (fig 3). Given the small data set, the power is low to detect a change as significant, but visual inspection does not indicate any structural break. This may reflect the high costs of implantable cardioverter defibrillators, at around £20 000 ($36 000; 29) per device, competition for resources with other interventional procedures in cardiology, and scarcity of skills in electrophysiology.¹¹

View larger version (30K): [in this window] [in a new window]   Fig 3 Total number of implantable cardioverter defibrillators implanted by quarter, 1995-2001

 

Hearing aids
The NICE guidance seems to have been received enthusiastically by audiology departments; all of those surveyed had undertaken an immediate audit of their service against the guidance requirements. However, review of case notes indicates that the range of analogue hearing aids offered does not seem to have been extended. Funding was described in the interviews as a major impediment to implementation. The guidance was issued at the same time as the Department of Health implemented a series of pilots of digital hearing aids, which cut across the guidance on analogue aids, which was subsequently withdrawn.

Laparoscopic surgery for primary inguinal hernia repair and colorectal cancer
Only 4% of primary inguinal hernia repairs in England and Wales were undertaken laparoscopically (contrary to NICE guidance), and this did not change after the guidance (0.3 monthly increase in hernia repairs, 95% confidence interval -5.48 to 6.08).

Although hospital episode statistics data for the 19 trusts that returned audit forms also showed 96% compliance, our audit of 545 repairs of primary unilateral hernias indicated only 65% compliance, indicating that coding of hospital episode statistics data may be unreliable. However, both national and audit data agreed that most laparoscopic procedures were concentrated in a few trusts, and that did not change over time.

Interviews showed that some local expert surgeons had the support of managers and commissioners to continue the use of laparoscopic surgery for primary repair. It was also claimed that patients often requested laparoscopic procedures.

The percentage of cases of colorectal cancer treated with laparoscopic surgery remained unchanged, at around 0.1% from 1998 to 2001.

Zanamivir for influenza
National prescribing data show little inappropriate prescribing of zanamivir in the absence of high levels of flu; prescriptions remained very low, at 499 in 2001, 190 in 2002, and 124 in 2003.

Orlistat for obesity
We found a significant increase in the average monthly prescribing of orlistat after the guidance had been published 22 per month (0.43, 95% confidence interval 15.9 to 27.8, P < 0.001; fig 4). Health authorities increased their use of orlistat, standardised by age, in the year after NICE guidance had been issued by about eight patient months per 1000 people aged 18-75 years. The variation in use, as measured by the coefficient of variation, fell from 0.4 in 2000-1 to 0.3 in 2001-2.

View larger version (31K): [in this window] [in a new window]   Fig 4 Use of orlistat in the community

 

In the 689 primary care patients for whom we have a record we found evidence that the drug was not being prescribed in accordance with guidance. Only in 12% of cases (n = 83) were there data showing compliance in the three key areas of age, body mass index, and weight loss. However, data recording was poor, and data were missing in one or more fields in 80% (551) of returns, particularly in respect of weight and weight loss. In those 308 cases where patients' weight loss prior to the visit had been recorded fully, the weight loss criterion was not met in 40% (123) of cases. Of patients with recorded weight loss data, only 41% (127) and 25% (40) still being prescribed orlistat at 3 months and 6 months, respectively, had reduced their weight as advised in the NICE guidance. The rise in prescribing of orlistat therefore does not necessarily imply a rise in appropriate prescribing.

Drugs for Alzheimer's disease
Total use grew logarithmically since February 1999 (fig 5). Once this had been taken into account, the growth rate at the time of publication of guidance (-1.4% per month; 95% confidence interval -4% to 1.2%) did not increase significantly. However, the data are also compatible with a view that an increase in trend occurred shortly before the guidance was formally issued. Variability between health authorities decreased; the coefficient of variation fell from 1.1 to 0.97.

View larger version (24K): [in this window] [in a new window]   Fig 5 Total use of Alzheimer's drugs in the community

 

Based on the audit data from 583 usable forms, compliance with the five recommendations in the guidance at first prescription varies between 52% and 85% for mental health organisations and 21% and 46% in primary care. Compliance with the recommendations at follow up is low, mainly because information regarding mini-mental state examination scores, follow ups, or the presence of relatives at assessment was not routinely recorded (a requirement of the guidance).

Discussion

Principal findings
The evidence that NICE guidance has made a difference either to the quality of care or to variations in practice is mixed. Some NICE guidance has been associated in time with changes in prescribing. Use of orlistat and taxanes grew rapidly after NICE guidance had been published, and uptake of drugs to treat Alzheimer's disease also increased, although this slightly preceded the release of that guidance. The guidance for wisdom teeth was published during a long downward trend in the extraction rate and did not have a discernable additional effect. The guidance for hips in general showed no effect. We found no evidence of a change in the number of implantable cardioverter defibrillators used, the numbers of inguinal hernias or colorectal cancers treated laparoscopically, or expansion in the range of hearing aids made available. In some cases audit showed that clinical practice was highly compliant with the indications for treatment laid out in the guidance (for example, wisdom teeth and the use of taxanes for breast cancer), but compliance was low for some (such as orlistat) and more variable for others. Some trusts seemed to exhibit more consistent compliance than others across a range of guidance, and box 2 shows their characteristics, although all identified funding as a major issue, especially where infrastructure costs were high.

Strengths and weaknesses of the study
In a retrospective observational study we cannot fully assess the impact of NICE guidance because of the absence of the counter-factual and because NICE guidance is just one, although potentially important, factor influencing professional practice. We can only observe whether clinical practice is consistent with the guidance. This research was based on a large data gathering exercise, and although data taken from national routine hospital and primary care are complete, those obtained from hospital pharmacies were less so. Where data were recorded, the audits allowed us to assess more reliably whether the details of practice corresponded to the indications of appropriateness in the guidance compared with subjective surveys¹² and analyses based simply on routine data.¹³ It was, however, sometimes difficult to design audits where the original wording of the NICE guidance had been complex or ambiguous. This is also likely to affect the ease with which the guidance is implemented. Our survey also offered a view of the management context into which the guidance was received, and the interviews were useful in highlighting the managerial, financial, and clinical perspectives on implementation.

Meaning of the study
The establishment of NICE as a mechanism for institutionalising evidence based health care was a unique initiative. This institutional response by itself is, however, not sufficient for the rapid and universal implementation of evidence based health care. This is unsurprising; NICE guidance is being issued at a time of great change in the NHS, with new structures, competing high level priorities, funding deficits, and staff shortages. In addition, healthcare organisations are complex, containing strong professional bureaucracies.¹⁴ Change is therefore heavily dependent on the actions of groups of professionals and individual clinicians.¹⁵ The ability to manage change in an organisation is further complicated by the increasing importance of networks that span several organisations.

The diffusion of innovations literature indicates that the adoption of guidance would depend on several other factors (box 3; after Rogers).^{16 17} These also emerged in the thematic analysis of the interview data (table 4). In areas where practitioners and managers see advantages to adoption, where the value is hardly disputed (clear evidence), and where there is professional endorsement (taxanes for breast cancer, wisdom teeth extraction, and orlistat for obesity) practice has changed relatively fast. The marketing activity of the pharmaceutical industry should not be overlooked as a possible explanation for the apparent increased uptake of drugs over devices and changes in surgical behaviour.

View this table: [in this window] [in a new window]   Table 4 Analysis of interviews: factors influencing likelihood of implementation

 

Box 2: Features of trusts consistent with high compliance Commitment to managing process of implementing guidance Identification of lead clinician at point of NICE announcement of topic for review Proactive assessment of local costs and implications of implementation Responsibility for funding and implementation vested in locality-wide group Strong clinical governance function appropriately resourced Culture of consensus Recognition of legitimacy of NICE Involvement of clinicians in guideline process Financial stability Expectation that compliance is mandatory, subject to identification of funding Targeted audit of areas of non-compliance

Box 3: Factors influencing adoption of innovation (after Rogers16) Perceived attributes and consequences of adoption (for example, relative advantage, complexity, observability) Type of innovation decision (optional, collective authority) Communication channels (for example, media, interpersonal, professional) Nature of the social system (norms, degree of interconnectedness of networks, concentration of opinion leaders) Extent of promotion efforts by agents of change

If the evidence has been disputed, or the costs not covered by increased income, adoption is more variable. Where practice is complex and also depends on an interaction with a practitioner's skills (such as with hip replacements or laparoscopic surgery), more formative research to understand the clinical context of practice may have helped to produce more influential guidance. Adoption may also be influenced by the degree to which decisions rest with an individual or requires team or organisational agreement. Individual surgeons are likely to have a high degree of professional autonomy, and this may partly explain the variability in techniques for hernia repair and hip replacement.

The extent to which trusts are prepared for NICE guidance and have put in place structures and processes to manage their implementation was variable.

The degree of active promotion by NICE is likely to have some impact on adoption, although probably not directly proportional to the effort invested. The greatest effect is likely when opinion leaders including the professional bodies and associations adopt and promote the guidance.

What is already known about this topic Research on the implementation of guideline implementation has been summarised, but its relevance to this unique national initiative was unknown The implementation of NICE guidance has not been evaluated overall. Previous work has been limited to single health technologies What this study adds Some clinical practice has changed in line with NICE guidance, in particular around prescribing (for example taxanes and orlistat) Other technologies have been adopted in line with NICE guidance but continued pre-existing practice patterns There is evidence that NICE guidance has been less influential in surgical procedures and use of medical devices Routine data are not sufficient to assess compliance with guidance; this needs review of case notes NICE guidance seems to have had an uneven impact on the uptake of evidence based medicine. This impact is likely to be greater if more effort is devoted to clarity of the guidance and its relevance to practice; adequate funding provision; getting professional support; and encouraging healthcare organisations to set up formal mechanisms for handling guidance

Unanswered questions and future research
Our research covers the early period of NICE guidance; it would be interesting to see if the response to subsequent guidance is different, particularly given the recent attention that NICE is giving to implementation. More research would also be useful to understand the professional and organisational responses to evidence based guidance better and to evaluate the relative contributions of various implementation strategies to practice patterns.¹⁷

Conclusions
NICE guidance has been associated with uptake of some technologies, although this has been variable. Implementation is likely to be improved if the guidance is clear and based on an understanding of clinical practice, if the evidence is strong and relatively stable, if adequate funding is available, and if the guidance is supported and disseminated by professional bodies. Trusts should institute strong supportive internal systems for handling guidance and gathering data on implementation.

---

The interview schedule for clinicians is on bmj.com

We acknowledge the support and cooperation of the NHS trusts that provided data, the audit staff and clinicians who carried out the case note audits, and the clinicians and managers who responded to the survey and participated in interviews. We thank the British Pacing and Electrophysiology Group (BPEG) for anonymised access to their register of Implantable Cardioverter Defibrillators and to the Trent and Welsh Arthroplasty Audit Groups for access to their anonymised database (TWAAG). We thank David Gibbons for advice on aspects of wisdom tooth removal and related issues and staff from the MRC Institute for Hearing Research who provided advice on the survey of audiology departments.

Contributors: NC contributed to study concept and design, study supervision, analysis and interpretation of the data, and manuscript preparation. DD contributed to the study concept and design, study supervision, analysis and interpretation of the data, and manuscript preparation. AL contributed to the design, conduct, and analysis of case note review; surveys and interviews; and manuscript preparation. KL contributed to the study concept and design, data collection, analysis and interpretation of the data, and manuscript preparation. James Mahon contributed to the statistical analysis of the data. Pauline Raynor contributed to the design and conduct of the primary care case note review. TAS contributed to study concept and design, study supervision, analysis and interpretation of the data, and manuscript preparation; he is guarantor. IW contributed to study concept and design, study supervision, analysis and interpretation of the data, and manuscript preparation. PW contributed to study concept and design, study supervision, analysis and interpretation of the data and manuscript preparation. DW contributed to the data collection and analysis. John Wilson contributed to the analysis of primary care and hospital pharmaceutical data. JW contributed to study concept and design, study supervision, analysis and interpretation of the data, and manuscript preparation.

Funding: NHS R&D National Co-ordinating Centre for Research Methodology (NCCRM).

Competing interests: NC was a member of the NICE Appraisals Committee between 1999 and 2002. KL, PW, DW, and JM work for York Health Economics Consortium, which undertakes work for a range of pharmaceutical companies, the Department of Health, and the NHS and has undertaken a cost-effectiveness study for Guidant, which manufactures implantable cardioverter defibrillators. This study was submitted to NICE as part of the assessment process.

Ethical approval: North West Multicentre Research Ethics Committee.

References

1. Department of Health. Faster access to modern treatment. London: DoH, London, 1999.
2. Dillon A, Gibbs TG, Riley T, Sheldon TA. The National Institute for Clinical Excellence and coverage of Relenza by the NHS. In: Fox DM, Oxman AD, eds. Informing judgment: case studies of health policy and research in six countries. New York: Milbank Memorial Fund, 2001.
3. McDowall D, McCleary R, Meidinger EE, Hay RA. Interrupted time series analysis. Quantitative applications in the social sciences, volume 21. Thousand Oaks, CA: Sage, 1980.
4. Box G, Jenkins G. Time series analysis: forecasting and control. 2nd ed. San Francisco: Holden Day, 1984.
5. Park RE, Fink A, Brook RH, Chassin MR, Kahn KL, Merrick NJ, et al. Physician ratings of appropriate indications for three procedures: theoretical indications vs indications used in practice. Am J Public Health 1989;79: 445-7.[Abstract/Free Full Text]
6. Shekelle PG, Kahan JP, Bernstein SJ, Leape LL, Kamberg CJ, Park RE. The reproducibility of a method to identify the overuse and underuse of procedures. N Engl J Med 1998;338: 1888-95.[Abstract/Free Full Text]
7. Mays N, Pope C. Qualitative research in health care. London: BMJ Publications, 1999.
8. Silverman D. Doing qualitative research: a practical handbook. London: Sage, 2000.
9. Strauss A, Corbin J. Basics of qualitative research. 2nd ed. Thousand Oaks, CA: Sage, 1998.
10. Miles M, Huberman A. Qualitative data analysis. London: Sage, 1994.
11. Plummer CJ, McComb JM. An audit of the implications of implementing NICE guidance on the use of implantable cardioverter-defibrillators. Heart 2003;89: 787-8.[Free Full Text]
12. Mace S, Taylor D. Adherence to NICE guidance for the use of anticholinesterases for Alzheimer's disease. Pharmaceut J 2002;269: 680-1.
13. Bloor K, Freemantle N, Khadjesari Z, Maynard A. Impact of NICE guidance on laparoscopic surgery for inguinal hernias: analysis of interrupted time series. BMJ 2003;326: 578.[Free Full Text]
14. Brock D, Powell M, Hinings CR. Restructuring the professional organization: accounting, health care and law. London: Routledge, 1999.
15. Fitzgerald L, Ferlie E. Interlocking interactions, the diffusion of innovations in health care. Hum Relat 2002;55: 1429-49.
16. Rogers EM. Diffusion of innovations. 5th ed. New York: Free Press, 2003.
17. Grimshaw JM, Thomas RE, MacLennan G, Fraser C, Ramsay CR, Vale L, et al. Effectiveness and efficiency of guideline dissemination and implementation strategies. Health Technol Assess 2004;8(6).

CiteULike    Complore    Connotea    Del.icio.us    Digg    Reddit    StumbleUpon    Technorati    What's this?

Relevant Articles

Why promote the findings of single research studies?

Paul Wilson, Mark Petticrew on behalf of the Medical Research Council’s Population Health Sciences Research Network knowledge transfer project team
BMJ 2008 336: 722. [Extract] [Full Text] [PDF]

Transparency in health technology assessments

Alan Maynard
BMJ 2007 334: 594-595. [Extract] [Full Text] [PDF]

How NICE may be outflanked

R E Ferner and Sarah E McDowell
BMJ 2006 332: 1268-1271. [Full Text] [PDF]

Seven years of feast, seven years of famine: boom to bust in the NHS?

Alan Maynard and Andrew Street
BMJ 2006 332: 906-908. [Full Text] [PDF]

What's the evidence that NICE guidance has been implemented?: Analysis is subject to confounding

Colin Tilley, Fay Crawford, Jan Clarkson, Nigel Pitts, and Mary McCann
BMJ 2005 330: 1084-1085. [Extract] [Full Text]

What's the evidence that NICE guidance has been implemented?: Maybe NICE needs to do more to ensure implementation of guidelines

Rishi Mannan and Melvyn Jones
BMJ 2005 330: 1085. [Extract] [Full Text]

What's the evidence that NICE guidance has been implemented?: More recent data on NICE implementation show different picture

Michael Rawlins and Andrew Dillon
BMJ 2005 330: 1086. [Extract] [Full Text]

What's the evidence that NICE guidance has been implemented?: Author's reply

Nick Freemantle
BMJ 2005 330: 1086. [Extract] [Full Text]

NICE influences prescribing but has more to do
BMJ 2004 329: 0. [Full Text] [PDF]

From optimism to hubris

Jane Smith
BMJ 2004 329: 0. [Extract] [Full Text] [PDF]

Is NICE delivering the goods?

Nick Freemantle
BMJ 2004 329: 1003-1004. [Extract] [Full Text] [PDF]

What has evidence based medicine done for us?

Sharon E Straus and Giselle Jones
BMJ 2004 329: 987-988. [Extract] [Full Text] [PDF]

This article has been cited by other articles:

• Ward, D., Murch, N., Agarwal, G., Bell, D. (2009). A multi-centre survey of inpatient pharmacological management strategies for alcohol withdrawal. QJM 102: 773-780 [Abstract] [Full text]  
• Maskrey, N., Greenhalgh, T. (2009). Getting a better grip on research: the fate of those who ignore history. InnovAiT 2: 619-625 [Abstract] [Full text]  
• Oliver, A. (2009). The Single-Payer Option: A Reconsideration. Journal of Health Politics, Policy and Law 34: 509-530 [Abstract]  
• Green, C. J., Maxwell, R., Verne, J., Martin, R. M., Blazeby, J. M. (2009). The influence of NICE guidance on the uptake of laparoscopic surgery for colorectal cancer. J Public Health (Oxf) 0: fdp027v1-fdp027 [Abstract] [Full text]  
• Carey, M., Buchan, H., Sanson-Fisher, R. (2009). The cycle of change: implementing best-evidence clinical practice. Int J Qual Health Care 21: 37-43 [Abstract] [Full text]  
• Ham, C. (2009). Chronic Care In The English National Health Service: Progress And Challenges. Health Aff (Millwood) 28: 190-201 [Abstract] [Full text]  
• Mears, A., Kendall, T., Strathdee, G., Sinfield, R., Aldridge, I. (2008). Progress on NICE guideline implementation in mental health trusts: meta-analyses. Psychiatr. Bull. 32: 383-387 [Abstract] [Full text]  
• Audit, N. P. T. (2008). Impact of NICE guidance on rates of haemorrhage after tonsillectomy: an evaluation of guidance issued during an ongoing national tonsillectomy audit. Qual Saf Health Care 17: 264-268 [Abstract] [Full text]  
• Schlander, M (2008). The use of cost-effectiveness by the National Institute for Health and Clinical Excellence (NICE): no(t yet an) exemplar of a deliberative process. J. Med. Ethics 34: 534-539 [Abstract] [Full text]  
• Wilson, P., Petticrew, M., on behalf of the Medical Research Council's Popula, (2008). Why promote the findings of single research studies?. BMJ 336: 722-722 [Full text]  
• Fenwick, E., Claxton, K., Sculpher, M. (2008). The Value of Implementation and the Value of Information: Combined and Uneven Development. Med Decis Making 28: 21-32 [Abstract]  
• Chidgey, J., Leng, G., Lacey, T. (2007). Implementing NICE guidance. JRSM 100: 448-452 [Full text]  
• Livesey, E. A., Noon, J M. (2007). Implementing guidelines: what works. EDUCATION AND PRACTICE 92: ep129-ep134 [Full text]  
• Armstrong, R., Waters, E., Crockett, B., Keleher, H. (2007). The nature of evidence resources and knowledge translation for health promotion practitioners. HEALTH PROMOT INT 22: 254-260 [Abstract] [Full text]  
• Nicholson, P J (2007). How to undertake a systematic review in an occupational setting. Occup. Environ. Med. 64: 353-358 [Full text]  
• Maynard, A. (2007). Transparency in health technology assessments. BMJ 334: 594-595 [Full text]  
• Grunfeld, E. (2007). Canadian breast cancer guidelines are as effective as possible under the circumstances. CMAJ 176: 785-786 [Full text]  
• Wright, J., Bibby, J., Eastham, J., Harrison, S., McGeorge, M., Patterson, C., Price, N., Russell, D., Russell, I., Small, N., Walsh, M., Young, J. (2007). Multifaceted implementation of stroke prevention guidelines in primary care: cluster-randomised evaluation of clinical and cost effectiveness. Qual Saf Health Care 16: 51-59 [Abstract] [Full text]  
• Drummond, M. F., Mason, A. R. (2007). European Perspective on the Costs and Cost-Effectiveness of Cancer Therapies. JCO 25: 191-195 [Abstract] [Full text]  
• Ferner, R E, McDowell, S. E (2006). How NICE may be outflanked.. BMJ 332: 1268-1271 [Full text]  
• Maynard, A., Street, A. (2006). Seven years of feast, seven years of famine: boom to bust in the NHS?. BMJ 332: 906-908 [Full text]  
• O'Loughlin, C., Darley, J. (2006). Has the referral of older adults with dementia changed since the availability of acetylcholinesterase inhibitors and the NICE guidelines?. Psychiatr. Bull. 30: 131-134 [Abstract] [Full text]  
• Rycroft-Malone, J. (2006). The politics of the evidencebased practice movements: Legacies and current challenges. Journal of Research in Nursing 11: 95-108 [Abstract]  
• Wright, J, Harrison, S, McGeorge, M, Patterson, C, Russell, I, Russell, D, Small, N, Taylor, M, Walsh, M, Warren, E, Young, J (2006). Improving the management and referral of patients with transient ischaemic attacks: a change strategy for a health community. Qual Saf Health Care 15: 9-12 [Abstract] [Full text]  
• Kapur, N. (2005). Management of self-harm in adults: which way now?. Br. J. Psychiatry 187: 497-499 [Abstract] [Full text]  
• Pearson, S. D., Rawlins, M. D. (2005). Quality, Innovation, and Value for Money: NICE and the British National Health Service. JAMA 294: 2618-2622 [Abstract] [Full text]  
• Henry, D. A., Hill, S. R., Harris, A. (2005). Drug Prices and Value for Money: The Australian Pharmaceutical Benefits Scheme. JAMA 294: 2630-2632 [Full text]  
• Lankshear, A J, Sheldon, T A, Lowson, K V, Watt, I S, Wright, J (2005). Evaluation of the implementation of the alert issued by the UK National Patient Safety Agency on the storage and handling of potassium chloride concentrate solution. Qual Saf Health Care 14: 196-201 [Abstract] [Full text]  
• Mannan, R., Jones, M. (2005). What's the evidence that NICE guidance has been implemented?: Maybe NICE needs to do more to ensure implementation of guidelines. BMJ 330: 1085-1085 [Full text]  
• Rawlins, M., Dillon, A. (2005). What's the evidence that NICE guidance has been implemented?: More recent data on NICE implementation show different picture. BMJ 330: 1086-1086 [Full text]  
• Tilley, C., Crawford, F., Clarkson, J., Pitts, N., McCann, M. (2005). What's the evidence that NICE guidance has been implemented?: Analysis is subject to confounding. BMJ 330: 1084-1085 [Full text]  
• Freemantle, N. (2005). What's the evidence that NICE guidance has been implemented?: Author's reply. BMJ 330: 1086-1086 [Full text]  
• Pennington, L., Roddam, H., Burton, C., Russell, I., Russell, D. (2005). Promoting research use in speech and language therapy: a cluster randomized controlled trial to compare the clinical effectiveness and costs of two training strategies. Clin Rehabil 19: 387-397 [Abstract]  
• Speller, V., Wimbush, E., Morgan, A. (2005). Evidence-based health promotion practice: how to make it work. Global Health Promotion 12: 15-20  
• Straus, S. E, Jones, G. (2004). What has evidence based medicine done for us?. BMJ 329: 987-988 [Full text]  
• Freemantle, N. (2004). Is NICE delivering the goods?. BMJ 329: 1003-1004 [Full text]  

Rapid Responses:

Read all Rapid Responses

NICE, but too late

Paul WX Foley
bmj.com, 1 Nov 2004 [Full text]

NICE Guidance: Sometimes Inappropriate in a DGH

Christopher F. Loughran
bmj.com, 2 Nov 2004 [Full text]

NICE Guidelines that recommend complex interventions may overlook systemic barriers and resource implications

Philip B Wilkinson
bmj.com, 4 Nov 2004 [Full text]

NICE and Electronic Foetal Monitoring

S Prabha, et al.
bmj.com, 12 Nov 2004 [Full text]

NICE guidelines

Neil R Poulter, et al.
bmj.com, 15 Nov 2004 [Full text]

Being NICE has advantages

Christine E Hine
bmj.com, 18 Nov 2004 [Full text]

Nice Implementation

Rishi Mannan, et al.
bmj.com, 18 Nov 2004 [Full text]

Confounding in the analysis of NICE guideline implementation

Colin J. Tilley, et al.
bmj.com, 2 Dec 2004 [Full text]

Understanding guideline implementation: the contribution of psychology

Susan F Michie, et al.
bmj.com, 31 May 2005 [Full text]

Online poll

Find out more on doc2doc >>