Rapid Responses to:
|
|
Rapid Responses: Rapid Responses published:
-
![[Read Rapid Response]](/icons/misc/sectionDOWN.gif)
Accuracy versus Utility
- Jeremy E Rogers (29 May 2003)
-
Coding costs and must be justified
- Andrew Roberts (29 May 2003)
-
Authors' reply
- Philip JB Brown, Victoria Warmington, Michael Laurence, and A Toby Prevost (11 June 2003)
|
|
|||
|
Jeremy E Rogers, Clinical Research Fellow Manchester M13 9PL
Send response to journal:
|
Although this study uses real clinicians rather than coding experts, and so should be applauded for attempting to evaluate coding accuracy in less artificial circumstances than characterises most prior comparative studies, it does not use real clinicians in a real setting. If I read the paper correctly, it suggests that the average clinical consult in primary care is capable of generating an average of 10 unique concepts that could be coded, each of which would take an average of 30 seconds to assign a code for. Does this mean that 5 minutes of a 10 minute consultation should be taken up by coding it ? This seems unrealistic. Further, assessing only the ability of clinicians to use a coding scheme overlooks the fact that, in UK primary care at least, the doctors in a practice may be responsible for only the minority of all codes recorded. In my own data from local practices the GPs entered only about 20% of all codes recorded in a 12 month period (and most of these are diagnosis codes; many practices employ coding clerks to input procedure codes as the discharge letters are received from the hospital). Half of all code entries were remotely entered by the laboratory links systems, whilst the remainder were entered by an assortment of practice nurses, receptionists and other workers. In one 5-partner practice studied, 43 different named individuals from 11 professional groupings entered at least one code during the one year study period. Brown's conclusion that CTV3 'is better than' 5 byte READ also deserves closer inspection. Using CTV3, codings were judged to be exact representations of the original term in 70% of cases, but the same code had only been used 58% of the time. From the point of view of reliable aggegation of data, an interesting figure would be the percentage of cases where both coders chose the same code AND the code chosen was judged an exact match. However, although this metric may still be higher for CTV3 than for 5-Byte read, the real question isn't 'is the result higher' but 'is it high enough ?'. Whilst a certain amount of erroneous coding might be tolerable when aggregating patient data, much higher standards are required when trying to use decision support algorithms to analyse single patient records. If there is a 30% chance that the record doesn't correctly say which kind of diabetes a patient has (for example), what future is there for clinical decision support software trying to correctly trigger a type-I diabetes management protocol (e.g. a standard data entry form, of the kind alluded to by Brown) ? The false negative rate will make the software unreliable, and the false positive rate will make it irritating. Comparative measures of coding accuracy are all very well, but an unanswered question remains: how accurate does the data need to be to be useful. Until we know that, saying CTV3 is better than 5-byte Read may be similar to saying that Paris is closer to Sydney than London. Competing interests: The author is an OpenGALEN ontologist |
|||
|
|
|||
|
Andrew Roberts, Consultant Orthopaedic Surgeon Robert Jones & Agnes Hunt Orthopaedic Hospital, Oswestry, SY10 7AG
Send response to journal:
|
EDITOR - Brown et al examine two clinical coding systems and conclude that the Clinical Terms Version 3 code set is significantly better than the Read Codes 5 byte set(1). They concede that there is little evidence that using a standard clinical terminology will accrue benefits. Most general practitioners will not code all the possible features of a consultation thus the average of ten codes generated by each consultation at 30 seconds each for coding with Clinical Terms Version 3 is not realistic. If 5 minutes per consultation for coding is too much is 60 seconds to code a couple of terms too little? Gardner implies that standards are essential resources for future clinical decision support, audit, governance, research, education and training(2). If that can be proven then an extra 10% of clinical time spent on minimal coding might be worthwhile. The main question that remains unanswered is how do patients benefit from having their care coded. In patients who do not have their care coded such as the 37% reported by Gray et al are the uncoded diabetics less well cared for than those with a C10 code and if so is failure to code a marker for poor care, excessive patient mobility or some other factor(3). The live clinical situation is one of the most challenging for effective information technology. Busy clinicians are usually under significant time pressure, are often mobile within the care environment and have high demands of the services provided for them. Under these circumstances a ruthless Darwinian selection process occurs and unfit or slow technology is discarded. When designing any clinical IT solution three principles need to be heeded: patients permit; clinicians specify and technologists implement. Until clinicians who have to enter the codes can be convinced that their efforts will bring reward in terms of better clinical care or perhaps reimbursement, coding will always be incomplete. Similarly those who require coding to be performed must realise that their decisions have to balance a real cost in clinical time with the need for information. It is better to have a minimal level of coding performed thoroughly and expend resources on adding detail later where needed. No amount of standards committees will deliver effective coding solutions if clinicians are not in the driving seat to specify an extent of clinical coding that delivers benefit in excess of cost. 1. Brown PJB, Warmington V, Laurence M, Prevost AT. Randomised Crossover trial comparing the performance of Clinical Terms Version 3 and Read Codes 5 byte set coding schemes in general practice. BMJ 2003;326:1127-30. 2. Gardner M. Why clinical information standards matter. BMJ 2003;326:1101-2. 3. Gray J, Orr D, Majeed A. Use of Read codes in diabetes management in a south London primary care group: implications for establishing disease registers. BMJ 2003;326:1130-2. Competing interests: Director of Clinical IT |
|||
|
|
|||
|
Philip JB Brown, Honorary Lecturer in Healthcare Informatics School of Information Systems, University of East Anglia, Norwich, Norfolk, NR4 7TJ, Victoria Warmington, Michael Laurence, and A Toby Prevost
Send response to journal:
|
Editor-The study design details the use of a series of headings including reason for encounter and medical history employed for the collection of terms by the ten general practitioners. This methodology was planned to capture an unbiased corpus of concepts and no proposal was made that this represented an average number of codes generated by a consultation. In practice on many occasions the past medical history would have already been coded and each new consultation would generate considerably fewer additional terms. We also stated in the discussion that the coding times were only a proxy measurement but that this might be improved with better software. Roberts is correct in highlighting that there is a cost-benefit balance to be reached in relation to coding by clinicians in a practical setting and whether patients benefit from having their records coded: we are indeed investigating this aspect by examining trends in quality markers over time following the introduction of more intensive coding practices [1]. Rogers criticises the study for not using clinicians in a real setting supporting this claim with a report of his own. One of the most challenging aspects for evaluation in medical informatics’ study design is however to move away from such anecdotal reports to obtain “control” in comparative studies [2]. It was for these reasons that we chose a randomised crossover trial using general practitioners with documented differences in gender, age, self ranked computer literacy, frequency of coding and knowledge of schemes. The limitations of videotaping this activity separate from the patient encounter were also acknowledged in our discussion. The study was purposely limited to general practitioners to improve control, as it has been known for many years that the accuracy of coding is influenced by the user’s knowledge [3,4]. Rogers also suggests that our conclusions deserve closer inspection and that “an interesting figure would be the percentage of cases where both coders chose the same code and the code chosen was judged an exact match” – the statistics for this are available in the results section (consistency of coding schemes) of the paper with an additional percentage of 22% (P<0.001) in favour of Clinical Terms Version 3. In our discussion we acknowledged that we did not try to measure the potential clinical importance of non-exact matches and clearly if these were to occur in critical areas the triggering of decision support algorithms would be adversely effected. Rome (or Sydney for that matter) was not built in a day and we acknowledge that the performance of a coding scheme is only one of many factors affecting the accuracy of data in computer-based patient records [5] most of which deserve similarly detailed controlled investigation methods such as utilised in our study. 1 Brown PJB, Warmington V. Data quality probes - exploiting and improving the quality of electronic patient record data & patient care. International Journal of Medical Informatics 2002; 68/1-3:91-8. 2 Friedman CP, Wyatt JC. Evaluation methods in medical informatics. New York, Springer 1997:163-4. 3. Zuber TJ, Rhody CE, Muday TA et al. Variability in code selection using the 1995 and 1998 HCFA documentation guidelines for office services. J Fam Pract. 2000 Jul;49(7):642-5. 4. James NK Reid CD Plastic surgery audit codes: are the results reproducible? Br J Plast Surg. 1991;44(1):62-4. 5. Hogan WR, Wagner MM. Accuracy of data in computer-based patient records. 1997 J Am Med Inform Assn;4: 342-355. Competing interests: None declared |
|||