Education and debate

Evaluating healthcare policies: the case of clinical audit

^a Health Care Evaluation Unit, Department of Public Health Sciences, St George's Hospital Medical School, London SW17 ORE

	Introduction

Top Introduction The classic model of... Alternative models of evaluation Conclusions References

Clinicians are under increasing pressure to show that their services are effective and efficient. Some have recently suggested that policymakers should be subject to the same discipline.¹ Before the introduction of radical changes in the NHS in 1991 the government's decision not to pilot the proposals or evaluate them was widely criticised. One component of the reforms was a national programme to promote clinical audit by doctors, later extended to nurses and therapy professionals. This was generally welcomed, though dissenting voices questioned the underlying political motivations² and pointed to a lack of evidence on the value of clinical audit.^{3 4 5}

Since then there have been repeated calls for the evaluation of audit,⁶ and the public accounts committee has expressed concern at the failure to assess the overall cost effectiveness of the programme.⁷ In fact, the NHS Executive has commissioned several evaluations of audit,^{8 9} as well as monitoring progress through local and regional annual reports. However, it has not been possible to use scientifically rigorous methods to quantify the overall costs or benefits of national or local programmes of audit.¹⁰ This paper describes the various approaches that have been tried (see box box) and outlines the merits and disadvantages of each approach.

	The classic model of evaluation

Top Introduction The classic model of... Alternative models of evaluation Conclusions References

Evaluative research is essentially normative: it seeks to assess not just what is but also what ought to be. The classic rationalist model of evaluation consists of five basic steps: (a) identify the goals of the programme under evaluation; (b) translate the goals into measurable indicators; (c) measure indicators for the study group who have been exposed to the programme; (d) measure indicators for an equivalent group that has not been exposed to the programme; (e) compare the results for the experimental and control groups.¹⁴

Economic evaluation may be viewed as a special case of this means-ends analysis, where the prime goal is the maximisation of benefit from a given investment of resources. The method that most clearly exemplifies this model is the randomised controlled trial. The problems with trials are well known, but they remain the only way to guarantee freedom from bias and provide definitive answers to questions of effectiveness and cost effectiveness.

Randomised controlled trials of audit interventions
It is no longer possible to conduct a controlled trial to evaluate whole programmes of clinical audit in Britain: no provider organisations could serve as controls and any clinicians not participating in audit are highly selected. However, controlled trials of specific audit interventions have been conducted—for example, the Lomas study on the implementation of practice guidelines for caesarean section¹¹ and the north of England study of standard setting in general practice.¹⁵

The randomised controlled trial was designed to test individual clinical interventions and processes of care. Its extension to organisational technologies, such as clinical audit, is problematic¹⁴ since it is difficult to define exactly what we mean by audit, and it is difficult to agree the goals of audit and to measure its impact.

Defining and controlling the audit intervention
As one might expect in an organisation as large and diverse as the NHS, approaches to audit, and quality in general, vary tremendously. There are many parallel initiatives, coming from a range of traditions, with varying philosophies and methods. In addition to this plurality of methods, audit is highly context dependent: it is contingent on personalities, relationships, professional and organisational structures, and processes.¹⁶ Therefore trials of selected audit interventions in selected circumstances are not easily extrapolated to audit in general. For example, the "audit with feedback" intervention in the Lomas study is different from most clinical audit as routinely practised in Britain. Evaluation also requires some standardisation of the intervention under review, which is likely to change its nature. Even where investigators take care not to impose too tight a structure on the intervention, as in the north of England study, some artificiality is introduced.

Defining and measuring the impact of audit
Defining and operationalising the goals of clinical audit are surprisingly difficult. The ultimate aim should clearly be to improve the quality of patient care, but what do we mean by quality? Certainly good quality care must be clinically effective, but other factors, such as equity and respect for patients' autonomy, are also important. Furthermore, clinical audit seeks to improve patient care in various ways; as well as direct changes in clinical practice, there are potential indirect effects through professional education and team development. Turning abstract goals into measurable outcomes is also difficult. For individual projects it is often possible to define suitable outcome or process indicators—for example, the Lomas study used the proportion of women who had previously had a caesarean section who underwent a trial of labour. However, summarising such information for a range of projects is not straightforward and generic measures are insensitive.

Before-after studies of audit interventions
Several before-after studies of audit interventions have been conducted.¹² These may be less artificial than randomised controlled trials, but it is never really possible to isolate the effects due to the audit without a truly equivalent control group.

The methodological difficulties outlined above mean that classic methods of evaluation cannot tell us whether the requirement for routine clinical audit throughout the NHS or the creation of audit structures and processes has improved the quality of services overall, or whether similar or greater improvements could have been achieved by using resources in other ways. Systematic reviews of controlled trials and before-after studies of audit and other behaviourial change mechanisms show mixed results,^{10 17 18} and results are not easily synthesised through meta-analyses because of heterogeneity of intervention and outcome measures. The Cochrane Collaboration on Effective Professional Practice is continuing work on this. However, tentative conclusions may be drawn about the types of intervention that are most effective—for example, "active" feedback, involving clinicians, appears to be more effective than "passive" feedback.

	Alternative models of evaluation

Top Introduction The classic model of... Alternative models of evaluation Conclusions References

The extension of classic models of decision making and evaluation into the realm of public policy has been criticised as both unrealistic and undesirable.¹⁹ When high levels of complexity, uncertainty, and conflict exist it is often impossible to agree clear goals or to identify the best mechanisms for achieving these goals.

Several other approaches have been used to evaluate audit programmes, including various quantitative and qualitative observational studies. Qualitative approaches to evaluation differ fundamentally in the questions they seek to address—not just whether a programme is meeting its objectives but also why it is successful or unsuccessful. Qualitative, or "illuminative," approaches involve "intelligently using available situations, data, and methodology to produce best approximations to the otherwise unknowable relationships between cause and effect or between input and output."¹⁴ They share four key characteristics (see box box).

Quantitative observational studies of audit programmes

View larger version (69K): [in this window] [in a new window]   MARK HUDSON

Given the difficulties in direct quantitative evaluation of clinical audit, a range of less direct methods have been used. Firstly, the level of audit activity has been assessed in terms of the numbers of clinicians participating, time spent, or numbers of projects.²⁰ Secondly, the quality of audit projects has been appraised against process or quasi-outcome criteria, such as completion of the audit cycle.^{13 21} Thirdly, attitudes towards audit and perceptions of the impact of audit have been assessed through surveys of provider staff.^{22 23} These approaches are useful as they provide comparative information that can be used to improve audit. However, it is not necessarily the case that "more and better audits imply improved quality of care."¹³ The philosophy of quality improvement rests on the assumption that staff participation and a sense of "ownership" are necessary to bring about change, which implies that clinicians must believe in clinical audit if it is to be effective, but the reverse is not necessarily true. In fact clinicians have been shown to have positive views about audit despite failing to complete the audit cycle.²¹

Qualitative evaluations of audit programmes
In addition to these pragmatic studies, many researchers have taken a qualitative approach.^{8 9 16 24 25} For example, CASPE Research was commissioned by the Department of Health to evaluate the national programmes of audit.⁸ They used a range of methods, including national surveys of purchasers and providers and detailed case studies. The CASPE evaluation team observed the functioning of audit at 29 selected providers, studying documents and conducting interviews. Progress was seen to vary considerably, with "very few" doing very well and a "substantial minority" very poorly. As Walshe points out, whether this observed range of performance represents success or failure is a matter of opinion.⁸ From these observations CASPE proposed seven "critical success factors" for clinical audit programmes.

Just like classic quantitative methods of evaluation, qualitative approaches cannot answer our basic questions: Does audit work? Is it a good use of resources? They can, however, provide a rich picture of the perceived successes and failures of audit in practice and suggest some of the reasons for these successes and failures.

Systems approaches
Finally, one approach to evaluation that has been little used in health services research is based on systems theory. The goal of systems engineering is to design systems to meet defined objectives while adapting to their environment.²⁶ Evaluation in this context entails comparing organisational structures and processes with some model of the "ideal adaptive organisation." As part of our study of clinical audit in South West Thames,²⁷ we used an approach based on one version of the systems approach called soft systems methodology.¹⁹ Soft systems models may be both descriptive, a simplified representation of real world systems, and prescriptive, representing how systems should be. There are many possible versions of each type of model, so there are no right or wrong models.

We collected information through regional workshops, semistructured interviews, documents, and published reports. Alternative perspectives of how clinical audit does, could, or should work were articulated through conceptual models (see fig 2). Soft systems methods share the characteristics of other qualitative evaluations and have the same advantages and disadvantages.

View larger version (38K): [in this window] [in a new window]   Fig 1 Example of a soft systems conceptual model. This split model illustrates one view of how a local programme of audit might be organised. It is designed to balance the conflicting needs for clinical ownership of audit and a supportive environment for professional development with the need to ensure that the concerns of others (including managers, purchasers, and patients) are addressed.

	Conclusions

Top Introduction The classic model of... Alternative models of evaluation Conclusions References

"Research is a form of currency as varying interests negotiate a political resolution, but research is almost never definitive enough to resolve major issues on which strong political interests disagree."²⁸

It is unlikely with the current techniques available that evaluative research will ever convince the sceptics of the value of clinical audit or disillusion the enthusiasts. A substantial body of research evaluating clinical audit now exists, but the debate continues.²⁹ Only the strongest level of evidence, the randomised controlled trial, could settle the controversy. But it is too late to conduct a trial of the introduction of local audit programmes, and trials of selected audit interventions cannot be extrapolated to the generality of routine audit. What evidence there is from experimental or quasi-experimental studies is equivocal.

Responding to questioning by the public accounts committee, the chief executive of the NHS in Scotland stated, "Since we're investing specific sums in a specific programme it does behove us to ask specific questions about what benefits are flowing and whether we can quantify these."³⁰ It also behoves us to be honest about what is achievable in terms of evaluating policy initiatives. We will never really know whether the national policy on clinical audit had a positive effect overall, or whether the money could have been better spent. Audit will always be an act of faith: a product of personal values, experience, professional loyalties, and anecdotal evidence.

This is not to say that evaluative research on organisational or policy issues is a waste of time; it is still useful to describe the impact of policy and explore the reasons for differing experiences. Both qualitative and quantitative approaches to evaluation can lead to practical prescriptions for improvement. There is already a wealth of information on clinical audit, and guidelines for effective audit have been developed.¹⁸ These should reduce the number of unproductive, wasteful, and demoralising experiences of audit and increase the number of rewarding ones.

	References

Top Introduction The classic model of... Alternative models of evaluation Conclusions References

1. Ham C, Hunter DJ, Robinson R. Evidence based policymaking. BMJ 1995;310:71-2. [Free Full Text]
2. Sherwood T. Exitus auditus—no fun. Lancet 1992;340:37-8. [Medline]
3. Maynard A. Case for auditing audit. Health Services Journal 1991;18 Jul:26.
4. Mooney G, Ryan M. Rethinking medical audit: the goal is efficiency. J Epidemiol Community Health 1992;46:180-3.
5. Buxton MJ. Achievements of audit in the NHS. Quality in Health Care 1994;3:S31-4.
6. Barton AG, Thomson RG, Bhopal RS. Clinical audit: more research is required. J Epidemiol Community Health 1995;49:445-7.
7. Committee of Public Accounts. National health service executive clinical audit in England. 31st report. London: HMSO,1996.
8. Walshe K, ed. Evaluating clinical audit: past lessons, future directions. London: Royal Society of Medicine Press, 1995.
9. Humphrey C, Berrow D. Medical audit in primary care: a collation of evaluative projects 1991-93. London: Department of Health, 1993.
10. Robinson MB. Evaluation of medical audit. J Epidemiol Community Health 1994;48:435-40. [Abstract]
11. Lomas J, Enkin M, Anderson GM, Hannah WJ, Vayda E, Singer J. Opinion leaders vs audit and feedback to implement practice guidelines: delivery after previous cesarean section. JAMA 1991;265:2202-7.
12. Gruer R, Gordon DS, Gunn AA, Ruckley CV. Audit of surgical audit. Lancet 1986;i:23-5.
13. Lawrence M, Griew K, Derry J, Anderson J, Humphreys J. Auditing audits: use and development of the Oxfordshire Medical Audit Advisory Group rating system. BMJ 1994;309:513-6. [Abstract/Free Full Text]
14. Illsley R. Professional or public health? Sociology in health and medicine. The Rock Carling Fellowship monograph. London: Nuffield Provincial Hospitals Trust, 1980.
15. North of England Study of Standards and Performance in General Practice. Medical audit in general practice. II. Effect on health of patients with common childhood conditions. BMJ 1992;304:484-8.
16. Kerrison S, Packwood T, Buxton MJ. Monitoring medical audit. In: Robinson R, Le Grand J, eds. Evaluating the NHS reforms. London: King's Fund Institute, 1993;155-77.
17. Oxman AD, Thomson MA, Haynes RB, Davis DA. No magic bullets: a systematic review of 102 trials of interventions to improve professional practice. Can Med Assoc J 1995;153:1423-31. [Abstract]
18. Dixon N. Good practice in clinical audit—a summary of selected literature to support criteria for clinical audit. London: National Centre for Clinical Audit, 1996.
19. Rosenhead J. Rational analysis in a problematic world. London: Wiley, 1989.
20. Webb SJ, Dowell AC, Heywood P. Survey of general practice audit in Leeds. BMJ 1991;302:390-2.
21. Eccles M, Deverill M, McColl E, Richardson H. A national survey of audit activity across the primary-secondary care interface. Quality in Health Care 1996;5:193-200. [Abstract]
22. Firth-Cozens J, Storer D. Registrars' and senior registrars' perceptions of their audit activities. Quality in Health Care 1992;1:161-4.
23. Lord J, Littlejohns P. Impact of hospital and community provider based clinical audit programmes: perceptions of doctors, nurses and other health professionals. Int J Quality in Health Care 1996;8:527-35.
24. Black N, Thompson E. Obstacles to medical audit: British doctors speak. Soc Sci Med 1993;36:849-56.
25. Thomson RG, Elcoat C, Pugh E. Clinical audit and the purchaser-provider interaction: different attitudes and expectations in the United Kingdom. Quality in Health Care 1996;5:97-103. [Abstract]
26. Gregory AJ, Jackson MC. Evaluation methodologies: a system for use. J Operational Res Soc 1992;43:19-28.
27. Lord J, Littlejohns P. Links between clinical audit and contracting systems. Int J Health Care Quality Assurance 1995;8:15-24.
28. Mechanic D. Social research in health and the American sociopolitical context: the changing fortunes of medical sociology. Soc Sci Med 1993;36:95-102.
29. Sellu D. Time to audit audit. BMJ 1996;312:128-9. [Free Full Text]
30. Committee of Public Accounts. Auditing clinical care in Scotland. London: HMSO, 1994.

CiteULike    Complore    Connotea    Del.icio.us    Digg    Reddit    Technorati    What's this?

This article has been cited by other articles:

• Neale, G., Vincent, C., Darzi, S. A. (2007). The problem of engaging hospital doctors in promoting safety and quality in clinical care. The Journal of the Royal Society for the Promotion of Health 127: 87-94 [Abstract]  
• Price, C. P. (2000). Evidence-based Laboratory Medicine: Supporting Decision-Making. Clin. Chem. 46: 1041-1050 [Abstract] [Full text]