Rapid Responses to:

NEWS ROUNDUP: Susan Mayor NHS data are not accurate enough for monitoring doctors’ performance BMJ 2006; 333: 937-b [Full text]

Rapid Responses: Submit a response to this article

Rapid Responses published:

Cause and Effect

stephen black   (8 November 2006)

Our assessment of the doctor’s individual performance

Alvarez-Li Frank C., Espinosa-Brito AD, Ordúñez-García PO and García-Buchaca S.   (8 November 2006)

Conclusions based on opinion

Paul Aylin, Brian Jarman   (22 November 2006)

Use of consultant-level data: Authors' response

Giles P Croft, John G Williams   (2 December 2006)

Cause and Effect 8 November 2006
stephen black, management consultant london sw1w 9sr Send response to journal: Re: Cause and Effect	The RCP report concludes that we should not use HES as a way of comparing consultants as the data are not reliable enough. This lack of reliability of (some parts of) HES is well known to those of us who use it regularly. And the report makes some sensible suggestions for improvements to the design of HES. But the recommendation that we need to do something other than HES (perhaps by developing other clinical information systems) to support processes such as revalidation may not be the correct answer. The reason why HES is patchy and unreliable is because clinicians don't engage with the processes that get data into HES and validate that data. The reason why they don't engage is that they don't think the information in HES is useful to them. If it was clear that HES was going to have a significant impact on them as individuals, then they might well have a much better incentive to engage with data gathering and analysis. (Hospitals should already be incentivised via Payment by Results as miscoding activity costs real money.) The trouble is that engagement with data-related processes sounds like "management" which is assumed by default to be a bad thing by too many people in the NHS. But, if the benefit of consultant-consultant comparison on the basis of good information is a much bigger incentive to adopt best practice clinical standards, then even a significant amount of extra management would be well worth it for patients and their consultants. Competing interests: None declared
Our assessment of the doctor’s individual performance 8 November 2006
Alvarez-Li Frank C., Deputy Director Hospital Universitario, Espinosa-Brito AD, Ordúñez-García PO and García-Buchaca S. Send response to journal: Re: Our assessment of the doctor’s individual performance	We have carefully read the article ¨NHS data are not good enough to gauge doctor’s performance¨ published by BMJ (1) and we would like to make a brief comment on this issue. Performance assessment has been defined as a process which systematically provides an exact and reliable description of the worker’s individual efficiency, what he is able to do out of what he must do. This process gains its highest complexity degree when the medical staff is the subject of the evaluation process since the doctor’s technical competence and the interpersonal relationships rising during the assistance to the health-disease process are the goals to be appraised. In other words, the medical care performance develops during the health service and is given to actual patients in the everyday medical practice. (2,3) Therefore, the process becomes extremely difficult to be assessed and will not be entirely reliable if it is only supported on information provided by databases, no matter how complete they could be regarded. Moreover, almost all performance assessment models not only include objective but also subjective measurements and others that can be regarded as the evaluator’s opinion which could be obtained through systematic worker’s performances observations during their profession exercise. We developed our own performance evaluation model. (4) Using it, 751 doctors working in either primary or secondary medical care were evaluated in 2004, representing 37.6% of this professional category in Cienfuegos province. A 0-10 point quantitative scale with punctuation ranking determined by experts’ criteria was used to score the following elements: work quantity, quality and results; knowledge show up (competences); cooperation; human relationship ethics; behavioural attitude towards postgraduate studies; personal image; creativity, attendance, punctuality and working hours efficiency; fulfillment of the established regulations and level of patients satisfaction. Lowest scores (weaknesses) were found in the items related to post-graduate training and amount of work with 8.04 and 9.02 respectively. Individual performance assessment results of each medical doctor were analysed in private meetings with the participation of the evaluated professional and his or her immediate chief. The final marks in those individual interviews included a positive agreement between two different points of view: on one side, previous self evaluations of doctors and, in the other one, the evaluator’s criteria. This open procedure, face to face, helpfully solved the discomfort frequently reported in other type of performance assessments. To know the performance weaknesses and analyse them with the evaluated personnel allows the establishment of individual development programs reducing the existing gap between the physicians and the medical staff we have and the ones we need to have in order to keep on improving the medical care given to our patients. Frank C. Alvarez-Li, MD, MSc; Alfredo D. Espinosa-Brito, MD, PhD, Pedro O. Ordúñez García, MD; Susana García-Buchaca, MSc References: 1. Mayor S. NHS data are no accurate enough for monitoring doctors performance. BMJ 2006;333:937-b. 2. Alvarez-Li FC, Ordúñez-García PO, Espinosa-Brito AD. Introducción de la evaluación del desempeño individual en un hospital cubano. Metodología y resultados. Rev Calidad Asistencial. 2006:21(2):102-10. 3. Chiavenato I. ED humano. En su: Administración de Recursos Humanos. Santafé de Bogotá:McGraw-Hill. 1998:259-291. 4. Alvarez-Li FC, García-Buchaca S. Assessment of the doctor’s individual performance: the Cienfuegos Province model, Cuba. 8 October, 2006. Disponible en http://bmj.com/cgi/eletter-submit/333/7571/748. Competing interests: None declared
Conclusions based on opinion 22 November 2006
Paul Aylin, Clinical Senior Lecturer in Epidemiology and Public Health Dr Foster Unit at Imperial, Dept. of Primary Care & Social Medicine, Imperial College, London W6 8RP, Brian Jarman Send response to journal: Re: Conclusions based on opinion	Considering that the Royal College of Physicians' report appears to be based on clinicians’ opinions rather than formal validation of the data, it seems a rather bold statement by the authors that data routinely collected about patients attending NHS hospitals in England and Wales “are not suitable for routinely monitoring the performance of individual consultant physicians”.[1] The authors then go on to suggest that there is a need to develop new clinical information systems and processes to provide clinically meaningful measures of activity and performance. Again, the statement does not seem to be supported by findings of the report. The authors acknowledge that following visits to the iLab, physicians’ confidence about how accurately central returns reflected their clinical activity, actually improved. The report also states that coding staff are “very effective at accurately coding and entering information. However, the information clinicians provide – in patient notes and discharge summaries – can often be incomplete or unclear for the purposes of coding”.[1] It is hard to understand then, the suggestion that completely new clinical information systems are required to provide clinically meaningful measures of activity and performance. The cost of developing and maintaining these parallel clinical information systems has been estimated at between £10 (UK Cardiac Surgical Register) to £60 (Scottish Hip Fracture Audit) a record. By contrast, Hospital Episode Statistics cost around £1 a record.[2] This dual system of collecting data in the NHS in separate administrative and multiple clinical systems was criticised as “wasteful and anachronistic” in the final report of the Bristol Royal Infirmary Inquiry.[3] The view therefore, that routinely collected hospital data should be dismissed in favour of new clinical information systems is unfounded, particularly with no supporting evidence that these new systems would provide any better quality data. Although there are certainly doubts about using routinely collected hospital data to provide analysis down to individual consultant level, that does not mean that they should be discounted as not suitable for monitoring the performance of individual doctors. We would certainly agree with the report that hospitals should routinely share clinically relevant analyses of local activity data with consultants in order to increase their involvement in the collection, validation and use of these data, and that feedback and use of these routinely collected data is the key to improving their quality. Only when clinicians have access to timely and meaningful analyses of their activity, will they feel engaged in the process and be willing to correct any inaccurate coding, thus improving the quality of the data. Finally, we’d like to point out that data on outpatient appointments have been collected routinely for several years and are available for analysis.[4] References [1] The iLab Project Evaluation Report. Royal College of Physicians. Available from URL: http://hiu.rcplondon.ac.uk/ilab.asp [2] Raftery J, Roderick P, Stevens A. Potential use of routine databases in health technology assessment. Health Technol Assess 2005;9(20) [3] Learning from Bristol: the report of the public inquiry into children's heart surgery at the Bristol Royal Infirmary 1984 -1995. Available from: URL: http://www.bristol-inquiry.org.uk/ [4] The IC (HES) Outpatient data. Available from: URL: http://www.hesonline.nhs.uk Competing interests: The Doctor Foster Unit at Imperial is funded by a research grant from Dr Foster Intelligence
Use of consultant-level data: Authors' response 2 December 2006
Giles P Croft, Clinical Research Fellow RCP Information Laboratory, University of Wales Swansea, SA1 1UG, John G Williams Send response to journal: Re: Use of consultant-level data: Authors' response	We are disappointed to learn that Aylin et al. have misunderstood the methods and findings documented in the RCP iLab report [1], which specifically involved individual physicians in the validation of their own activity held on HES/PEDW; a practice recommended to improve data quality [2]. While we were heartened to find that the methods we employed did increase physician confidence, our analysis and validation of data attributed to individual participating physicians found that not only were 20% of all physicians not represented at all in these datasets, but that those who were engaged consistently identified inaccuracies, misallocations and incompleteness to a degree that precluded any meaningful comparisons against others – the cornerstone of performance monitoring. Indeed, a recent independent report of consultant-level HES data replicated our findings with regards variation in coding practices, questionable quality, distorting factors and fitness for the purposes of comparison, stating that the analyses produced were "meaningful only in the context of local interpretation" [3,p4]. Our discussion of potential solutions has also been misunderstood, for we agree wholeheartedly that the collection of multiple datasets is indeed "wasteful and anachronistic", as is clearly referenced in the full report [1,p33]. Information for secondary purposes should ideally flow from routinely collected data sources, not expensive purpose-built audit datasets. Our reference to the need for new "clinical information systems" relates to ongoing work with Connecting for Health in England and Informing Healthcare in Wales [1,p41]. If front-line clinicians are to be suitably engaged in improving quality, they require a data source which is clinically rich, clinically designed and representative of today's clinical practices. While many positive steps can be taken in improving its quality, even if 100% accurate, hospital episode statistics will not fulfil that role. We acknowledge that high-level activity data on outpatients have been available for some time. However, consultant-level outpatient data are not freely available for analysis. Trust-level HES outpatient data were made available as "experimental data", on 31/07/06 (the iLab study period concluded in 2005), and are currently limited to non-clinical attendance data only [4]. Our study has confirmed that physicians wish to see and use data that are clinically rich, attributable to individual practitioners and cover all contacts that they have with patients. Regrettably, routinely collected data do not at present meet these requirements. 1. Royal College of Physicians (2006) The iLab Project Evaluation Report. Available at: http://hiu.rcplondon.ac.uk/documents/iLabEvaluationReport.pdf 2. Audit Commission (2004) Information and data quality in the NHS: key messages from three years of independent review 3. University of York & Department of Health (2006) Delivering Quality and Value: Consultant Clinical Activity. Methodology and data quality exercise: 2004/05 data. Activity rates of consultants in five surgical and five medical specialties. Available at: http://www.hesonline.nhs.uk 4. The Information Centre (2006) Reporting patient journeys: Hospital outpatient activity in 2003-04 and 2004-05. First report and quality assessment of experimental data from patient level record systems. Available at: http://www.ic.nhs.uk/pubs/12monthspast Competing interests: None declared