Reducing errors made by emergency physicians in interpreting radiographs: longitudinal study

Reliability modelling

Table 1 contains predictions of the potential adverse events (PAE) and the unmitigated potential adverse events (UPAE) that result from the rates of misinterpretation of the emergency physician and the radiologist, the effectiveness of the recall procedure, and the structure of the system in which these components interact. This appendix provides the basis for the numbers in the table and some suggestions on using reliability modelling to improve the safety of patients.

Method

The first figure (Figure A1) illustrates the chains of events that could occur in the system that was in place after the initial improvements but before the redesign. The diagram makes each chain of events leading to a PAE and UPAE visible by using a tree diagram and conditional probability. Each branch of the diagram contains three bits of information: an event, the frequency of the event given the preceding events, and the frequency of the chain of events including and preceding it. For example, one chain of events in the diagram is that the emergency physician reads the radiograph, a false negative error is made, and the radiologist identifies the error during the over read. Given that the emergency physician has read the radiograph and made a false negative interpretation, the frequency that the radiologist will identify it is 0.997. The frequency of this entire chain of events is 0.00718, which is (0.6)(0.012)(0.997).

Whenever a chain of events leads to a potential adverse event, the frequency of the chain is put in bold and marked with an asterisk. If the potential adverse event goes unidentified or unmitigated at a subsequent step another asterisk is added to the corresponding frequency. If the system mitigates the potential adverse event at some step in the chain, then the bold and asterisks are removed. When the chain of events reaches the point that the patient leaves the emergency department without the potential adverse event being mitigated, an exclamation point marks the frequency of the chain of events.

The frequency of potential adverse events and unmitigated potential adverse events can be computed by summing the frequencies of chain of events that led to them. From the diagram the frequency of potential adverse events is 0.0084. This corresponds to the rate of 8.4 potential adverse events per thousand cases in cell (d) of the table. The frequency of unmitigated adverse events is 0.00158 and corresponds to the rate in cell (e). The reader could reproduce the rates in cells (b) and (c) by substituting 0.03 for the frequency of a false negative interpretation by the emergency physician for 0.012 and recomputing the frequencies of these chains of events.

The second figure (Figure A2) contains the diagram of the chains of events for the system after redesign and produces frequencies of potential adverse events and unmitigated potential adverse events that correspond to the rates in cells (f) and (g).

Applications

In this case study we used the reliability modelling to predict retrospectively the effect of a reduced rate of error and a new structure of the system. The reliability modelling also can be used prospectively to predict the impact of potential changes to the system. Data on serious but rare adverse events provide slow feedback on the safety of a system. With some data on potential adverse events and some reasonable assumptions, modelling can help the designer to choose among alternative systems or courses of action. Modelling has the potential to predict which of the systems under consideration will be the safest. This seems preferable to evaluating data from injured patients.

Modelling can also guide decisions concerning where the next efforts at improvement should be focused. Comparing the recall rates in the last row of the table indicates that an effective recall process will have a substantial impact. Some data on the current level of effectiveness of the call back process would be a worthwhile.