BMJ 2000;320:737-740 ( 18 March )

Papers

Reducing errors made by emergency physicians in interpreting radiographs: longitudinal study

James A Espinosa, medical director a,  Thomas W Nolan, statistician b. 

a Department of Emergency Medicine, Overlook Hospital, Atlantic Health System, 99 Beauvoir Avenue, Summit, NJ 07902, USA, b Associates in Process Improvement, 1110 Bonifant Street, Silver Spring, MD 20910, USA

Correspondence to: J A Espinosa jim010{at}aol.com


    Abstract
Top
Abstract
Introduction
Case study
Results
References

Objectives: To reduce errors made in the interpretation of radiographs in an emergency department.
Design: Longitudinal study.
Setting: Hospital emergency department.
Interventions: All staff reviewed all clinically significant discrepancies at monthly meetings. A file of clinically significant errors was created; the file was used for teaching. Later a team redesigned the process. A system was developed for interpreting radiographs that would be followed regardless of the day of the week or time of day. All standard radiographs were brought directly to the emergency physician for immediate interpretation. Radiologists reviewed the films within 12 hours as a quality control measure, and if a significant misinterpretation was found patients were asked to return.
Main outcome measures: Reduction in number of clinically significant errors (such as missed fractures or foreign bodies) on radiographs read in the emergency department. Data on the error rate for radiologists and the effect of the recall procedure were not available so reliability modelling was used to assess the effect of these on overall safety.
Results: After the initial improvements the rate of false negative errors fell from 3% (95% confidence interval 2.8% to 3.2%) to 1.2% (1.03% to 1.37%). After the processes were redesigned it fell further to 0.3% (0.26% to 0.34%). Reliability modelling showed that the number of potential adverse effects per 1000 cases fell from 19 before the improvements to 3 afterwards and unmitigated adverse effects fell from 2.2/1000 before to 0.16/1000 afterwards, assuming 95% success in calling patients back.
Conclusion: Systems of radiograph interpretation that optimise the skills of all clinicians involved and contain reliable processes for mitigating errors can reduce error rates substantially.

    Introduction
Top
Abstract
Introduction
Case study
Results
References

To better serve the needs of patients and to reduce costs, new ways of caring for patients must be designed. Many of these improvements will optimise the work of a team of clinicians, with team members working at their highest capability. Our aim is to illustrate a collaborative approach used by radiologists and emergency physicians to achieve a significant reduction in the errors made by emergency physicians in the interpretation of radiographs.

Rates of disagreement between emergency physicians and radiologists in the interpretation of radiographs range from 8-11%.1-5 A change of treatment was required for 1-3% of these patients. These errors in interpreting radiographs in the emergency department can also have significant clinical and legal consequences.6 As early as 1984, Trautlein published a study of 200 consecutive malpractice claims stemming from treatment in emergency departments. Radiographs were found to have been incorrectly interpreted in 38 of the cases7; these misinterpretations included missed fractures in 14 patients and missed foreign bodies in eight. Between 1974 and 1985 the liability programme of the American College of Emergency Physicians identified the most frequent cause of malpractice actions as the failure to diagnose fractures of the extremities and other fractures. The second most frequent cause was the failure to identify a foreign body in a wound. Errors related to the misdiagnosis of fractures accounted for the third largest amount of dollars paid out to settle malpractice claims.8

    Case study
Top
Abstract
Introduction
Case study
Results
References

Baseline analysis
In the third quarter of 1993, JAE initiated an analysis of the errors made by emergency physicians in interpreting radiographs at his hospital. Clinically significant errors in interpretation were defined, according to the method of Fleisher, as a false negative interpretation that would have resulted in a change in the patient's care.4 The rate of clinically significant errors was tracked each month and was based on an analysis of daily events. From July 1993 to December 1994 the average was 3% (figure). This was consistent with previously reported rates of clinically significant misinterpretations, although it was at the high end of the 1-3% range. Films that were not interpreted by emergency physicians, such as specialised scans, ultrasound scans, and intravenous pyelogram studies, were not included in the data.

At the time of the analysis, four separate radiology processes were in place for patients seen in the emergency department. One process operated from 8 00 am to 5 00 pm on weekdays; during this time radiologists interpreted the radiographs and transmitted the results to the emergency physician. From 5 00 pm until 10 00 pm radiology residents interpreted the radiographs and transmitted the results to the emergency physician. From 10 00 pm until 8 00 am the emergency physician interpreted the radiograph and a subsequent interpretation (an "over read") was done by a radiologist within 12 hours as a quality control measure. On weekends, a fourth system consisted of a mixture of the other systems. However, exceptions to the first process, in which primary responsibility for interpretation was with the radiologist, were frequent and resulted in the use of the third process, in which primary responsibility fell to the emergency physician even between 8 00 am and 5 00 pm on weekdays. The data available for the initial analysis were from cases in which the emergency physician was the primary interpreter of the radiograph and the radiologist provided the subsequent over read.

This variety of processes resulted in a standard of care that differed, at least implicitly, depending on the day of the week and the time of the day. This situation conformed to O'Leary et al's assessment of practice patterns in the interpretation of radiographs in emergency departments: "The interhospital variability of responsibilities within the course of a 168-hour week suggest that the evolution of these arrangements may have been driven by staffing constraints, scheduling practices and convenience, rather than by efforts to provide a consistent high standard of care."9

Initial improvement efforts
The initial aim was to reduce the number of interpretation errors made by emergency physicians. This initial work preceded a more extensive redesign of the system that would address the variation in the responsibility between emergency physicians and radiologists.

A quality improvement effort began in the first quarter of 1995, using an approach similar to that described by Berwick.10 Several improvements were implemented. All staff reviewed all clinically significant discrepancies at monthly meetings in a blame free environment. A common error discussed, for example, was the finding of small chip fractures on an ankle film series that had been interpreted by the emergency physician as negative. A file of clinically significant errors was created. This has been maintained since 1994 and is used for training. Mandatory study of the entire file has become part of the orientation of all new staff. Overall departmental patterns of error were identified from this file, and a focused review of these patterns took place at staff meetings. Patterns of errors made by each emergency physician were identified and were discussed as part of routine, ongoing communication and during performance reviews.

During the baseline period from July 1993 to December 1994 false negative errors had been made by the emergency physicians in 3% (95% confidence interval 2.8% to 3.2%) of the 28 161 cases reviewed. From August 1995 to August 1996, the rate of false negatives fell to 1.2%. (1.03% to 1.37%) for the 20 236 cases studied.

Fundamental redesign

Background
The interpretation of radiographs by emergency physicians had been made more reliable but the four different processes for having the radiographs interpreted were still in place. Long delays in taking and processing the films were common. Patients, emergency physicians, attending physicians, and nurses were unhappy with the process, as documented by data from both external and internal surveys of satisfaction. The hospital uses an externally benchmarked customer satisfaction survey for patients seen in the emergency department, inpatient services, and outpatient services (Press Ganey Associates, South Bend, Indiana). One question asks patients to rate their satisfaction with the time spent waiting for a radiograph. Before the processes were changed satisfaction was rated in the 20th centile. Internal data from private physicians and emergency physicians showed that there was considerable dissatisfaction with the process, especially with the difficulties in locating films needing to be reviewed.

The hospital managers commissioned an interdisciplinary team to redesign the process; it was composed of a staff radiologist, two radiology residents, a staff emergency physician, the managers of the emergency department and the diagnostic radiology department, the executive director of radiology, and the medical directors of the emergency department and radiology. The team's goal was to improve patients' satisfaction by significantly shortening the time they spent in the department waiting for the interpretation of their radiographs to become available. The team also aimed to further reduce the number of errors made in interpreting radiographs.

Changes
A system was developed for interpreting radiographs that would be followed regardless of the day of the week or the time of day. All standard radiographs were to be brought directly to the emergency physician for immediate interpretation. A radiologist would provide an interpretation within 12 hours as a quality control measure. When a clinically significant misinterpretation was found by the radiologist, staff from the emergency department would contact patients and ask them to return.

This process clearly assigned the primary, initial responsibility for interpreting plain radiographs to the emergency physician. The new system reduced the confusion that had been caused by ambiguously defined roles and engendered collaboration between the radiology and emergency departments to redesign other related processes to make the new system function reliably.

Other improvements to support the new system were also made. The process of communication between the radiologists and the emergency physicians was enhanced to better address situations in which a clinically significant error was detected. A new form was designed to provide feedback to the physicians, and a process for using the form was implemented. Potentially significant discrepancies are logged by the radiologist. These are then formally sent to the emergency physician on duty who is responsible for reviewing the films and the patient's record. If a discrepancy is identified, the emergency physician is responsible for contacting the patient and documenting the contact. This process embeds the ongoing training of emergency physicians into the normal process of care. Additionally, radiologists can call the emergency department and speak to an emergency physician when a discrepancy is identified.

The team redesigned the processes in the radiology film room to accommodate changes made in the emergency department. Additionally, a film alternator was placed in the emergency department, which allows the outputs of a series of radiographs to be viewed simultaneously and facilitates interpretation.

    Results
Top
Abstract
Introduction
Case study
Results
References

From November 1996 to December 1999 the rate of false negatives for the 67 111 cases studied was 0.3% (0.26% to 0.34%). This result is consistent with the 0.4% rate of clinically significant misinterpretation of radiographs reported by Preston et al after a similar improvement effort.11

Substantial improvements in patient satisfaction and a shortening of the turnaround time for interpretations also occurred as a result of the redesign. Improvements included a 50% reduction in the time it took from ordering plain films to having them returned to the emergency department. The centile ranking of the items on the patient satisfactions survey relating to radiology rose above the 90th centile. There was also a 50% reduction in the time it took for patients presenting to the emergency department with trauma to an extremity to be discharged. There was an improvement in the overall satisfaction of patients with their visit to the emergency department: satisfaction ratings rose from the 18th centile to above the 95th centile.

Reliability modelling
The safety of the system with respect to the misinterpretation of radiographs depends on the reliability of the interpretations of the emergency physician and the radiologist, the effectiveness of the recall procedure when a clinically significant error is identified, and the structure of the interaction among the components. The figure shows the error rate for the interpretation of radiographs by emergency physicians. Data on radiologists' error rates and on the effectiveness of the recall procedure were not available. In the absence of these data and to provide a quantitative prediction of what had been accomplished as a result of the improvements we modelled the reliability of the system. Reliability modelling is a common statistical and engineering technique used to ascertain the reliability and safety of alternative structures of systems.12 (Further information on reliability modelling is available in an appendix on the BMJ's website.)



View larger version (23K):
[in this window]
[in a new window]
  		Trends in the rate of errors (clinically significant false negative interpretations) made by emergency physicians in interpreting radiographs. Details of the changes are described in the text

For the purposes of this analysis, a potential adverse event was defined as a case in which a clinically significant error of misinterpretation is made on the first reading of the radiograph either by the emergency physician or the radiologist. An unmitigated potential adverse event is defined as a case in which an error of misinterpretation is made and the error goes undetected or the error is detected but the patient does not return for treatment. The rates of potential adverse events and unmitigated potential adverse events provide a measure of the reliability of the system.

The model was based on the following assumptions:

(1) The error rates for emergency physicians can be obtained from the data in the figure

(2) The error rates for the radiologists are assumed to be stable at 0.3%, which was the final error rate achieved by the emergency physicians

(3) If a radiologist makes an error of interpretation, the emergency physician will not identify it

(4) Before the new system was implemented, a radiologist interpreted 40% of the radiographs and an emergency physician interpreted 60% with a subsequent reading by a radiologist occurring within 12 hours

(5) The radiologist's interpretation was not influenced by the emergency physician's interpretation

(6) Not all patients whose radiographs were found to have a clinically significant misinterpretation would return to the emergency department. A success rate of 95% of patients returning was assumed but the model was also run for a 99% success rate for comparison.

The predictions obtained by the model are shown in the table.


                              
View this table:
[in this window]
[in a new window]
 

Predictions of error rates obtained by reliability modelling. Results were multiplied by 1000 to define rates per 1000 cases

Nolan has outlined a strategy that includes three components for increasing the safety and reliability of a system.13 The first is to design the system to prevent errors. The second is to design the procedures to make errors visible when they do occur so that they can be intercepted. And the third is to design procedures for mitigating the adverse effects of errors when they are not detected and intercepted.

The table illustrates the effect of each of these three components on the reliability of the system. The effect of the decrease in the emergency physicians' error rate on the rate of potential adverse events can be seen by comparing the weighted average of the error rates for emergency physicians and radiologists with the improved error rate for emergency physicians and the error rate for emergency physicians after the system was redesigned. By comparing the rate of potential adverse events with the associated rate of unmitigated adverse events for each process the impact of the identification of errors by the radiologist can be seen.

The mitigation strategy in this study was to have the patients who represented potential adverse events return to the emergency department for treatment. The effect of the mitigation (the recall process) on the rate of unmitigated potential adverse events can be seen by comparing the third and fourth columns in the table. The success rate for the recall process has a substantial effect on the rate of unmitigated potential adverse events when the interpretation errors are few (table).

Conclusions
Efforts to reduce errors in health care can be directed at a wide variety of processes. We have described efforts to reduce the rate of error among emergency physicians interpreting radiographs. The validation of such efforts will largely depend on replication by other centres. This change has been sustained over time, which suggests that the changes were sufficiently robust to meet the challenge of a complex and changing environment.


What is already known on this topic

Most previous studies have estimated the rates of misinterpreting radiographs

These studies have usually focused on one professional group, such as emergency physicians or radiologists, over a short interval

What this study adds

This six year, longitudinal study takes a systems approach to the problem of misinterpreting radiographs

Rather than comparing the performance of two groups of professionals, this study shows the impact of cooperation between emergency physicians and radiologists in reducing errors and the potential adverse events that result from them

The error rate for emergency physicians who are interpreting radiographs can be reduced to the level of incidents per thousand. Systems for interpreting radiographs that are designed to optimise the skills of both emergency physicians and radiologists and that contain reliable processes for mitigating errors can reduce the rate of unmitigated potential adverse events to the level of parts per hundred thousand. This rate is substantially lower than the rate that could be achieved by either an emergency physician or a radiologist acting alone. As error rates are reduced to the level of parts per thousand, the process of mitigating errors has a substantial effect and further reduces the incidence of unmitigated potential adverse events.

This approach seems to have generalisable features. The design principles might also apply to reducing other adverse events in emergency departments, such as the preventable adverse events associated with the management of patients with chest pain. When any large effort at redesigning systems is undertaken or significant clinical change is made, it is prudent to study errors as an important outcome measure of undesirable side effects.

    Acknowledgments

   Contributors: JAE provided leadership for the changes and performed the study at his hospital. TWN recognized the broader systems issues addressed by the study and developed the approach to reliability modelling used to understand the broader systems impact of the work. Both authors shared equally in writing the paper. JAE is guarantor.

    Footnotes

Funding: None.

Competing interests: None declared.

website extra: Additional information about reliability modelling appears on the BMJ's website www.bmj.com

    References
Top
Abstract
Introduction
Case study
Results
References

1. Robinson PJ, Wilson D, Coral A, Murphy A, Verow P. Variation between experienced observers in the interpretation of accident and emergency radiographs. Br J Radiol 1999; 72: 323-330[Abstract].
2. Lufkin KC, Smith SW, Matticks CA, Brunette DD. Radiologists' review of radiographs interpreted confidently by emergency physicians infrequently lead to changes in patient management. Ann Emerg Med 1998; 31: 202-207[CrossRef][Medline].
3. Scott Jr WW, Bluemke DA, Mysko WK, Weller GE, Kelen GD, Reichle RL, et al. Interpretation of emergency department radiographs by radiologists and emergency medicine physicians: teleradiology workstation versus radiograph readings. Radiology 1995; 195: 223-229[Abstract/Free Full Text].
4. Fleischer G, Ludwig S, McSorley M. Interpretation of pediatric x-ray films by emergency department pediatricians. Ann Emerg Med 1983; 12: 153-158[CrossRef][Medline].
5. Rhea JT, Potsaid MS, DeLuca SA. Errors of interpretation as elicited by a quality audit of an emergency department facility. Radiology 1979; 132: 277-280[Abstract].
6. George JE, Espinosa JA. Legal issues in emergency radiology. Practical strategies to reduce risk. Emerg Med Clin North Am 1992; 10: 179-203[Medline]
7. Trautlein JJ. Malpractice in the emergency department-review of 200 cases. Ann Emerg Med 1984; 13: 709-711[CrossRef][Medline].
8. Fastow JS. Medico-legal risks: identification and reduction. In: American College of Emergency Physicians comprehensive guide to effective practice management. Dallas: ACEP, 1986.
9. O'Leary MK, Smith M, Olmsted WW, Curtis DJ. Physicians' assessment of practice patterns in emergency department radiograph interpretation. Ann Emerg Med 1988; 17: 1019-1023[CrossRef][Medline].
10. Berwick DM. A primer on leading the improvement of systems. BMJ 1996; 312: 619-622[Free Full Text].
11. Preston CA, Marr JJ, Amaraneni KK, Suthar BS. Reduction of "call-backs" to the emergency department due to discrepancies in the plain radiograph interpretation. Am J Emerg Med 1998; 16: 160-162[CrossRef][Medline].
12. Ushakov I. Handbook of reliability engineering. New York: John Wiley, 1994.
13. Nolan TW. System changes to improve patient safety. BMJ 2000; 320: 771-773[Free Full Text].

(Accepted 23 February 2000)

© BMJ 2000
Add to CiteULike CiteULike   Add to Complore Complore   Add to Connotea Connotea   Add to Del.icio.us Del.icio.us   Add to Digg Digg   Add to Reddit Reddit   Add to Technorati Technorati    What's this?

Related Articles

Reducing error, improving safety
Victor Barley, Graham Neale, Christopher Burns-Cox, Paul Savage, Sam Machin, Adel El-Sobky, Anne Savage, Aziz Sheikh, Brian Hurwitz, C K Connolly, Pia Maria Jonsson, Göran Tomson, Lars Räf, Ken McCune, Ron Law, Roger M Goss, Anonymous, Carlos A Estrada, James Carter, Clyde Brooks, Ann C Jobe, Jonathan Aldridge, Peter Freeland, Brenda Ashcroft, Max Eistein, Nicholas Boreham, Peter J Shirley, Leo Strunin, Maldwyn Morgan, Paul Cartwright, and Robert P Coffey
BMJ 2000 321: 505. [Extract] [Full Text] [PDF]

Lower error rates in reading radiographs can be sustained by redesigning the system
BMJ 2000 320: 0. [Full Text]

This article has been cited by other articles:

  • Brown, C, Hofer, T, Johal, A, Thomson, R, Nicholl, J, Franklin, B D, Lilford, R J (2008). An epistemology of patient safety research: a framework for study design and interpretation. Part 2. Study design. Qual Saf Health Care 17: 163-169 [Abstract] [Full text]  
  • Welch, S., Jensen, K. (2007). The Concept of Reliability in Emergency Medicine. American Journal of Medical Quality 22: 50-58 [Abstract]  
  • Shojania, K G, Burton, E C, McDonald, K M, Goldman, L (2005). Overestimation of clinical diagnostic performance caused by low necropsy rates. Qual Saf Health Care 14: 408-413 [Abstract] [Full text]  
  • Mucci, B, Brett, C, Huntley, L S, Greene, M K (2005). Cranial computed tomography in trauma: the accuracy of interpretation by staff in the emergency department. Emerg. Med. J. 22: 538-540 [Abstract] [Full text]  
  • Schmidek, J M, Weeks, W B (2005). Relationship between tort claims and patient incident reports in the Veterans Health Administration. Qual Saf Health Care 14: 117-122 [Abstract] [Full text]  
  • Hardy, M, Barrett, C (2004). Interpretation of trauma radiographs by radiographers and nurses in the UK: a comparative study. Br. J. Radiol. 77: 657-661 [Abstract] [Full text]  
  • Gatt, M E, Spectre, G, Paltiel, O, Hiller, N, Stalnikowicz, R (2003). Chest radiographs in the emergency department: is the radiologist really necessary?. Postgrad. Med. J. 79: 214-217 [Abstract] [Full text]  
  • Berlin, L. (2002). Communicating Findings of Radiologic Examinations: Whither Goest the Radiologist's Duty?. Am. J. Roentgenol. 178: 809-815 [Full text]  
  • Studdert, D. M., Brennan, T. A. (2001). No-Fault Compensation for Medical Injuries: The Prospect for Error Prevention. JAMA 286: 217-223 [Abstract] [Full text]  
  • Wallis, L. A, Guly, H. R (2001). Improving care in accident and emergency departments. BMJ 323: 39-42 [Full text]  
  • Alberti, K G M M (2001). Medical errors: a common problem. BMJ 322: 501-502 [Full text]  
  • Barley, V., Neale, G., Burns-Cox, C., Savage, P., Machin, S., El-Sobky, A., Savage, A., Sheikh, A., Hurwitz, B., Connolly, C K, Jonsson, P. M., Tomson, G., Räf, L., McCune, K., Law, R., Goss, R. M, Anonymous, , Estrada, C. A, Carter, J., Brooks, C., Jobe, A. C, Aldridge, J., Freeland, P., Ashcroft, B., Eistein, M., Boreham, N., Shirley, P. J, Strunin, L., Morgan, M., Cartwright, P., Coffey, R. P (2000). Reducing error, improving safety. BMJ 321: 505-505 [Full text]  
  • Leape, L. L, Berwick, D. M (2000). Safe health care: are we up to it?. BMJ 320: 725-726 [Full text]  

Rapid Responses:

Read all Rapid Responses

reducing errors of radiographic interpretation
Elizabeth Miller
bmj.com, 20 Mar 2000 [Full text]
System works
Jonathan Aldridge
bmj.com, 10 Apr 2000 [Full text]
Radiograph systems
Gavin D Perkins
bmj.com, 19 Apr 2000 [Full text]
Emergency X-ray Studies
Kim Bullock
bmj.com, 9 May 2000 [Full text]
radiographer input.
Nia Leake
bmj.com, 25 Aug 2001 [Full text]

Student BMJ

Intimate examinations

Israeli students are refusing to perform intimate examinations on anaesthetised women without their informed consent.

www.student.bmj.com

Listen to the latest BMJ Interview