Where have all the pennies gone? The work of Manchester medical audit advisory groupBMJ 1994; 309 doi: https://doi.org/10.1136/bmj.309.6947.98 (Published 09 July 1994) Cite this as: BMJ 1994;309:98
- Roger Johnson, chairman, Manchester Medical Audit Advisory Groupa
- Accepted 27 April 1994
Medical audit has its critics, who point to the large sums of NHS cash that seem to be disappearing down a medical plughole. These criticisms are recognised by medical audit advisory groups but there are many reasons why the work of these groups has not yet resulted in many publications in journals or bumped up health indicators. After discussing the criticism this article describes the work of the medical audit advisory group in Manchester. Real changes in cooperative working with general practice teams and between practices are taking place, and improved relationships between general practice and the hospitals are being helped by joint audit work. The Manchester group is also working to help in setting standards and to cooperate with purchasing. The work of the group is changing as it develops.
In my role as chairman of the medical audit advisory group in Manchester I am accountable to my family health services authority on behalf of my group. I am also accountable to the general practices and to their patients, and to taxpayers, for the money that has been spent on audit of primary care in Manchester. In this article I first mention recent criticism of medical audit, then discuss the reasons for the lack of indicators of the success of audit at this stage. The work of the Manchester group argues that continuing support for medical audit advisory groups is justified but that they should welcome continuing evaluation.
Medical audit in the NHS has come under criticism. The critics are important and influential voices and deserve a response. From the health economists’ perspective Professor Alan Maynard, writing in the Health Service Journal,1 and Professor Martin Buxton, speaking at a 1993 London conference on raising quality in the NHS, have both identified the large sums of money spent on medical audit and questioned the benefit produced.
When clinicians and their teams look at the work that they do, their professional motivation will be to do those tasks better. To prove benefit from improvements in quality requires measures of health gain. Health economists are struggling to produce valid measures of health gain; forests have been sacrificed in the hunt for the quality adjusted life year (QALY). Drummond admits that work on the valuation of health states is experimental, and that the methodologies involved provide “no magical answer.”2
If economists see the outcome of clinical audit in short-term managerial savings they will not find the value for money they seek. Medical and clinical audit may well never save any money at all.
So we have a no win situation if asked to evaluate our performance in cost-benefit terms. Baker's review of medical audit confirms that many published audits of general practice have failed to complete the audit cycle,3 but the outlook is not uniformly gloomy. Evidence from the Netherlands shows that setting standards locally can play an important part in the success of audit.4 Hughes and Humphrey state that the criteria for publication in academic journals are likely to include sound research techniques and innovative methods or new findings which will have wide value when published. Audit may well not meet these criteria, and so publications would be lacking.5 This means that it is not likely that our current activity could be fairly represented by the body of publication to date, although Manchester medical audit advisory group has attempted to inform the debate.*RF 6-8*
Humphrey and Berrow point out that major differences exist in the approach taken by medical audit advisory groups and the roles that they fulfil. This will make evaluation of their effectiveness a complex task.9
The confidentiality of audit is guaranteed to doctors and patients so there can be no compulsion for the results of audit to be made available outside of the practice. One critic has pointed out that no salesman could get away with keeping the sales figures confidential, and yet that is what we must do. In fact, the profession has developed great confidence through the programme and we now normally find that practices are prepared to reveal their audit results. This has been achieved through the development of trust between the profession and their medical audit advisory group but has slowed down publication of the detail of the groups’ work.
What was the task set?
Health circular (FP)(90)8 did not specify what medical audit was, nor did it set medical audit advisory groups any guidelines for evaluating the activity.10 We were given no powers of any sort. We had to walk into the job with the confidence of the baby Mowgli walking into the wolf's lair. Mowgli had no concept of being eaten for supper; we are not as innocent as Mowgli so had few illusions. We had to carry the ketchup as an offering, but still expected to be the main course.
The lack of guidance and the lack of powers have probably been strengths, not weaknesses, in our evolution. Uniquely in the recent history of the NHS, the enthusiasm of clinical teams could be harnessed to review the work they do and to have resources to support this.
We had to bring about a major cultural change in general practice. Practices had to develop processes of review. They had to learn to question their activities. We had to help introduce the mechanics for gathering data and to teach the skills needed.
The Manchester group
As a group we spent much of 1991 educating ourselves and planning the task. We took the advice of the pilot medical audit advisory groups, who had been in place from the year before. Liverpool in particular reported that it should have established a support team of high quality lay staff. This we decided to do. We had to find suitable premises and set up systems that would allow us to record what we were doing, and resources to underpin the educational role that we had to undertake in general practice.
A key part of the preparatory work was the development of policy statements on, for example, computing and on confidentiality. Data protection registration required a ruling by the data protection registrar, and when we came to consider the potential for the group to uncover something that might place patients at risk, the duty of absolute confidentiality placed on us had to be qualified. Our local medical committee developed guidelines for us and these were subsequently confirmed by the General Medical Council.
These guidelines give insights into the complexity of the task as it developed. At the same time we were providing information to all our general practitioners about our plans, and holding meetings for general practitioners.
In 1991 at national medical audit advisory group meetings the consensus was that all audit should be practice based and developed. Groups that were considering large multipractice audits were heavily criticised. Our group rejected such criticism. We felt that collaborative audits would provide a tool for educating those practices that had never considered audit. They would provide an early baseline to permit us to measure improvement later. We would be able to structure such audits and they would also offer links to hospital and community services. Our responsibility to introduce systematic audit, could well best begin with multipractice audits.
Hughes and Humphrey clearly interpreted the Department of Health circular as requiring medical audit advisory groups to develop a strategy rather than proceed with ad hoc facilitation of audits within individual practices,9 and we agreed with this, though individual audits remain the cornerstone of the work. We developed the confidence to do what we felt would work best and we took account of the need to work closely with both the family health services authority and the local medical committee.
Sharing audit work between practices and with the hospital sector
General practice is an extremely isolated professional task. General practitioners may have left formal training behind many years ago, and though they have to comply with the updating requirements of the postgraduate education allowance there is some evidence that doctors prefer to attend courses in subjects that interest them already. We wanted to encourage collaboration between practices and to organise the sharing of information. Allowing practices to compare themselves with each other is a way of doing this. We also had the responsibility of developing links between primary care and the hospitals. In 1991 we bid successfully for national and regional money for two large projects; a multipractice audit of diabetes and a city-wide interface audit of antenatal care.
The topic of diabetes touches on Health of the Nation issues, and better management of this condition can improve outcome, so the benefit of audit could be evaluated by using diabetes as an index condition. We developed a stepwise audit approach to the management of the disease. Audit packs were produced which permitted practices to enter the audit no matter what level of diabetes management they currently achieved. These levels expanded as the confidence of the participants and the team grew. Now the range of audit questions runs from the basic entry level, “Who are my diabetic patients?” to “What complications have they and what do they think about the management of their disease?” This project had a target of recruiting 25 practices in the first year, and now has 80 of the 107 Manchester practices as active participants.
This audit has been facilitated in practices by field workers. The computerised database has 5448 registered diabetic patients and, with the extension to neighbouring medical audit advisory groups, we anticipate another 5000 being added. This database may have value for research, public health, and purchasing.
We have seen increases in the second complete year of the audit in the recorded prevalence of diabetes in participating practices and in the recording of monitoring measures such as foot pulse examination and funduscopy for the first 20 practices under surveillance (table I). These are markers for improved care. Our fieldworkers say that in the second year, practices involved in the project are much better organised and confident in the care of their diabetic patients. This is a real improvement. We hope to be able to follow up our diabetic patients and measure final outcome, but to show benefits here will take years.
Questioning of 900 of the professionals involved in antenatal care in the city resulted in the development of an audit check list. Next a retrospective audit measured the records of women who had delivered against the audit questions. The group then held uniprofessional and multidisciplinary meetings to discuss the work and produce a report. The audit showed that “shared care cards” were no such thing, and recommended hand held records. The move to hand held records in the city has been accelerated by our work. Our report matched the findings of the Winterton committee11 and the recent Changing Childbirth report of the Department of Health12 in recommending a woman centred and midwife led approach.
Because our work is locally based, links between the local professionals have been strengthened. We have reached the conclusions that are likely to be implemented nationally, so we understand the imperative. We are pushing for change through purchasing. Both midwives’ groups and obstetricians are nervous of the implications, but some have already met in seminars run by the medical audit advisory group and understand each others’ anxieties. When change towards a service led partly by midwives is required nationally, we will be most of the way there. That our group should have played a part shows how its role is developing fast and can be key to effective service change.
Visiting is a major task for the group. The ideas behind this process lie in the Royal College of General Practitioners’ report, What Sort of Doctor.13 We first had to ask what visiting was for, what questions should be asked, and how the results could be recorded. Although we agreed that the prime aim in the first round was to get to know practices and identify their needs, our visitors needed training to do this. We also wanted to elicit what work was going on and to record it in a consistent manner.
We felt that the visit could be as much benefit to the visitor as to the practice visited, so we offered the opportunity of being a visitor to anyone who had attended an audit seminar in the previous year. A training package was developed in conjunction with the department of general practice in Manchester. Role play was an important tool in this process. Twenty trained visitors then began our programme, which started in earnest at the end of 1992. All but three practices accepted a first round visit. We soon had records of audit activity on the database, but for the second round of visits we felt that we needed to define the quality of the audit work more closely, so modified the recording procedure to adopt the Oxford model.14 The percentage of audits completing the audit cycle has increased greatly (table II).
We learn from the experience of the visitors. They bring back messages of the stress in general practice at present, of the difficulties of coping with change, and we tailor our response as a result.
Our computerised audit recording database, MARS, now records 335 audit projects in the 104 Manchester practices, covering 44 major coded topics that reflect the breadth of interests in general practice (tables II and III). We now have coordinated multipractice audits of diabetes, hypertension, patient satisfaction (using Richard Baker's dialogue audit15), asthma, benzodiazepines, and back pain. When the health promotion banding system was announced we developed a clerical recording system for non-computerised practices to allow them to apply for the banding payments and we received requests for our pack from all over the country.
Quality and audit presuppose good record keeping. This was identified as an important aspect of audit in the General Medical Services Committee's paper on audit: “To facilitate reviews the MAAG will need to encourage the attaining by all practices of the minimum standard of recording approved by the Joint Committee on Postgraduate Training in General Practice.”16 Disorganised records lacking summaries are difficult to use for audit or for good practice.
We agreed with the family health services authority to develop a summarising team in Manchester which was funded separately from the medical audit advisory group's budget. We produced a training and support package for summarising medical records. A printed manual has been produced to help summarising. The first team began work in October 1991. This work has continued, has spawned training for general practice staff, who need the skills to keep the summaries up to date, and has spread to training in three other family health services authority areas. To date 74000 records have been completed and we have work to the end of 1995. Built into the process is an audit of quality control and an audit of participating practices to ensure that they keep the work going.
Managing and developing this work has been good for the image of the medical audit advisory group as it is evident to general practitioners that the group is helping them directly. The quality of this work has been of the highest order and totally refutes any argument that such work cannot be done by trained lay staff.
Relations with others
We have recognised our responsibility to work with Health of the Nation targets, and diabetes, hypertension, and benzodiazepines were chosen with this in mind. We are building links with the departments of epidemiology and public health to find ways that our growing information resource can be used to inform public health work. We are linking ourselves with bids for research, and the development of clinical research units in Manchester.
We have linked with the purchasing authority so that information from audit can inform purchasing and have accepted “wish lists” from purchasers of topics important to them, including stroke and glue ear. The MARS database will identify practices interested in particular topics and we hope that this will allow focused work for the purchaser to develop. During the past two years we have also organised evening audit presentations for general practitioners by general practitioners, and we twice presented the work of audit to a mixed group of councillors, authority members, and community health council members. We organised the North West Regional audit seminar in primary care in September 1993, which had 200 participants from practices from all over the region.
Six monthly and annual reports to our family health services authority are produced to account for our work, and we report to the regional health authority on the projects funded by them. Presentations to our family health services authority and to seminars organised in the NHS are all routine parts of our work. These activities are very labour intensive and place a heavy load on the group's secretariat.
Early on the needs of practice nursing and administrative staff were considered. A training programme for these staff has been developed and the courses are oversubscribed. They consist of three modules: an introduction to audit, devising an audit and reviewing the audit. This has been a direct and pragmatic “hands on” process, and enthusiastically enjoyed. In some practices the general practitioners have not shown great enthusiasm for audit but have nevertheless released their staff for such training, and audit has started in those practices.
Since the end of 1992 the group has begun to plan in the medium term and has taken the decision to set quality as the main objective for its activity. This is presenting us with new challenges, and we are working with the family health services authority and the local medical committee to start a process of setting standards in Manchester, which we call “Emerging Standards.” The medical audit advisory group itself would not set standards—that is for practices or groups of general practitioners to do.
The outcome of this initiative will not take shape until well into 1994, and even then it will only be the beginning of a rolling programme. The issue of monitoring for quality will emerge, and we foresee a voluntary system whereby practices determine the standards that they wish to set, and may request confirmation of attaining them.
Tables IV and V give some detail of our income and expenditure. We have won large sums for our multipractice audit work and this has greatly extended what we have been able to do. For comparison in the context of the spending on primary care, the Manchester group's core funding from the family health services authority, for example, is less than the annual spend by one average general practitioner on drugs.
The money is spent on salaries for our support staff, on renting premises and providing equipment and consumables, and on payments to doctors working on behalf of the group for the time they give. We decided not to pay practices directly for audits because this could not be sustained in the long run across all practices. Instead, we provided audit field workers to assist specific audit.
This process is in contrast with audit in the hospital sector, where dedicated time can be made available for audit. Some hospitals now suspend routine work for a day each month for audit to take place. There are costs to the practices in time and resources, and it is a tribute to general practitioners and their staff that this contribution has been made.
We have seen a major attitudinal shift over the past two years in Manchester. Doctors are much more open about the work that they are doing and the problems that they face. They are prepared to work in teams with their staff to review their work. The move to thinking about setting standards could only be taking place where these attitudinal shifts have prepared the ground.
This is not to say that there are no problems. The contract changes in general practice over the past three years have put enormous organisational strain on general practitioners and their staff. The effect of this on general practices cannot be overstated. This process has not stopped, and all practices are having to work at new and higher levels of data recording and practice management activity, with consequent increases in computer and paperwork time. Consultations, night visits, and patients’ expectations all are on the increase. Patient's charter pressures are there as well. We have tried to help practices with all the work that we have done, so that their medical audit advisory group shall be seen as facilitating, not frightening. To approach these teams again with added challenge has been one of our hardest tasks, and we must not forget that the doctors working with the group have all the same stresses themselves.
The amount of hard data on improved outcomes so far is small. It exists for our early multipractice audit of diabetes, but little of this work has been published. A major confounding factor is that the work done by general practitioners has been changed by their new contract. It will be difficult to separate out those changes brought about through audit and those resulting from contractual change.
Our critics can only go on what has been published, and I submit that the duties placed upon medical audit advisory groups have been so considerable that it is unlikely that many will have the skills or resources to publish at this early stage in their development. Many groups would welcome evaluation by outside bodies such as health economics units.
Soft data, as outlined in this article, are present on the ground in plenty. We have endeavoured, using our growing skills, to do responsibly what we can. We have built a relationship of trust with our family health services authority. But trust alone will not be enough in the future, and we have to work to find methods of critically evaluating the benefits of the money spent. These evaluation techniques will need to look at the work that the medical audit advisory group has done and will need to find measures that accurately reflect the outcome of such work. It is unlikely that simply counting audits done in general practice will provide an adequate measure of the changes that have been brought about.
I thank Manchester general practitioners and their teams and the medical audit advisory group members and staff for making it possible to record so much progress.