BMJ  2004;328:199 (24 January), doi:10.1136/bmj.328.7433.199

Paper

Comparison of three methods for estimating rates of adverse events and rates of preventable adverse events in acute care hospitals

¹ Comité de Coordination de l'Evaluation Clinique et de la Qualité en Aquitaine, Hôpital Xavier Arnozan, 33604 Pessac, France

Abstract

Objectives To compare the effectiveness, reliability, and acceptability of estimating rates of adverse events and rates of preventable adverse events using three methods: cross sectional (data gathered in one day), prospective (data gathered during hospital stay), and retrospective (review of medical records).

Design Independent assessment of three methods applied to one sample.

Setting 37 wards in seven hospitals (three public, four private) in southwestern France.

Participants 778 patients: medical (n = 278), surgical (n = 263), and obstetric (n = 237).

Main outcome measures The main outcome measures were the proportion of cases (patients with at least one adverse event) identified by each method compared with a reference list of cases confirmed by ward staff and the proportion of preventable cases (patients with at least one preventable adverse event). Secondary outcome measures were inter-rater reliability of screening and identification, perceived workload, and face validity of results.

Results The prospective and retrospective methods identified similar numbers of medical and surgical cases (70% and 66% of the total, respectively) but the prospective method identified more preventable cases (64% and 40%, respectively), had good reliability for identification ( = 0.83), represented an acceptable workload, and had higher face validity. The cross sectional method showed a large number of false positives and identified none of the most serious adverse events. None of the methods was appropriate for obstetrics.

Conclusion The prospective method of data collection may be more appropriate for epidemiological studies that aim to convince clinical teams that their errors contribute significantly to adverse events, to study organisational and human factors, and to assess the impact of risk reduction programmes.

Introduction

Review of medical records is considered the benchmark for estimating the extent of medical injuries in hospitals.^1-7 The limitations of this retrospective method and the assessment of other objectives, such as the impact of risk reduction programmes and organisational and human factors, mean that alternative methods are needed.^8-10 We compared prospective and cross sectional methods for data collection with review of medical records for assessing rates of adverse events and rates of preventable adverse events in acute care hospitals in France.

Methods

An adverse event was defined as an unintended injury caused by medical management resulting in death, life threatening illness, disability at time of discharge, admission to hospital, or prolongation of hospital stay.^2-5 Preventable adverse events would not have occurred if the patient had received ordinary standards of care appropriate for the time of the study.

Our population sample was inpatients in medical, surgical, and obstetric wards in acute care hospitals in Aquitaine, France. The sample was obtained by a two stage cluster stratified process (see figure on bmj.com).

Independent investigators consecutively applied the three methods to the sample (figure). We used two questionnaires, which were adapted from an English survey.⁵ One was used for detection (by nurses and midwives) and one for confirmation (by doctors).

View larger version (29K): [in this window] [in a new window]   Method and timing of data collection

 

Data collection
Cross sectional method
For the cross sectional method the detection investigators interviewed the head nurse and if necessary consulted the patient's medical records on the basis of 17 criteria (box 1). Patients who screened positive were referred to the physician investigator who interviewed the doctor who managed the patient on the day of data collection, and consulted the patient's medical records if necessary. That day constituted the first day of the prospective method.

Prospective method
For the prospective method the investigators visited the ward on day one of the survey and on two other occasions during the first seven days, then once a week for up to a month. The doctor involved in the prospective method visited the ward at the end of the first week then when the last patient was discharged or on day 30 if patients were still present.

Retrospective method
For the retrospective method, review of the medical records began 30 days after the cross sectional method. The proportion of cases (incidence rate for retrospective and prospective methods, point prevalence for cross sectional method) was computed by taking into account the hospital's cluster effect.

Box 1: Criteria used when consulting medical records Unplanned admission as a result of any healthcare management during the 12 months before index admission Hospital incurred unintentional patient injury Adverse drug reaction or drug error Hospital acquired infection or sepsis Unplanned removal, injury, or repair of organ or structure during surgery, invasive procedure, or vaginal delivery Unplanned return or visit to the operating theatre Unplanned open surgery after closed or laparoscopic surgery Cardiac or respiratory arrest, low Apgar score Development of neurological deficit not present on admission Injury or complications related to termination of labour, and delivery including neonatal complications Other patient complications including myocardial infarction, deep vein thrombosis, cerebrovascular accident, pulmonary embolism Patient or family dissatisfaction with care received documented in the medical record, or documentation of claim or litigation Unplanned transfer from general care to intensive care or higher dependency Unplanned transfer to another acute care hospital Unexpected death (that is, not an expected outcome of the disease during hospital stay) Documented pain or psychological or social injury Any other undesirable outcomes (not covered by any of the other criteria)

Outcome measures and statistical analysis
Effectiveness of the methods was determined by the proportion of cases identified in relation to a reference list. This list was based on the adverse events identified by any one of the three methods. We calculated the proportion of preventable cases detected by each method in relation to the reference method.

We used paired ² tests to compare the effectiveness of the prospective and retrospective methods for identifying cases, preventable cases, and their subgroups (adverse events occurring during hospital stay, those responsible for all or part of hospital admission, and the most serious—associated with death or life threatening illness). For these subgroup analyses, we aggregated data from the medical and surgical wards using an equivalent weight, because in France the number of beds in these specialties is similar.

Reliability of the prospective method was assessed from cases detected on the first day of the study and confirmed twice—by the doctors performing the cross sectional and prospective methods. Acceptability and face validity of the results were assessed during sessions with each clinical team. Questions were open ended and participants could express their perception of the workload and of the truthfulness of the results of each method.

Results

We selected 37 wards in three public and four private hospitals: medical (n = 15), surgical (n = 14), and obstetric (n = 8). The three methods were applied to 778 patients: 278 in medicine, 263 in surgery, and 237 in obstetrics. The adverse event rates found by the prospective and retrospective methods were similar, and the point prevalence obtained by the cross sectional method was about one third lower than that of the other two methods (table). The incidence of preventable adverse events assessed prospectively was 25% higher than that assessed retrospectively.

View this table: [in this window] [in a new window]   Proportion of inpatients with at least one adverse event according to epidemiological method (n=778)

 

Thirteen of the 254 adverse events were excluded from the reference list and considered as false positives. The final list comprised 241 adverse events in 174 patients (110/80 in medicine, 114/80 in surgery, and 17/14 in obstetrics). Of these 174 patients, 122 (70%) had one adverse event, 38 (22%) had two adverse events, and 14 (8%) had three adverse events (125 patients had at least one adverse event on the day of cross sectional data collection). See bmj.com for details.

The prospective method was significantly more effective than the retrospective method at identifying cases in medicine (paired ² = 8.64, P < 0.01) and less effective at identifying cases in surgery (paired ² = 3.24, P < 0.07). The prospective method was significantly more effective at identifying preventable cases in medicine (paired ² = 17.8, P < 0.005) but no difference was observed in surgery. In obstetrics, effectiveness was not different but there were too few adverse events to draw conclusions.

When the medical and surgical cases were aggregated, the prospective and retrospective methods showed similar effectiveness (70% and 66% of 160 patients identified, respectively), but the prospective method was more effective at identifying preventable cases (64% and 40% of 71 patients, respectively; P < 0.02). Effectiveness was similar for adverse events occurring during hospital stay (68% and 64% of 110 patients), those responsible for hospital admissions (63% and 59% of 75 patients), and the most serious events (44% and 33% of 52 patients).

Reliability of cases identified among the 145 cases detected on the first day of study was good (global agreement 91.7%; = 0.83, 95% confidence interval 0.67 to 0.99), but agreement about preventability was low (67.8%; = 0.31, 0.05 to 0.57). See box 2 for advantages and disadvantages of the three methods.

Discussion

The retrospective method of data collection is as effective as the prospective method for estimating adverse event rates. The prospective method has several advantages over retrospective and cross sectional methods. It identified more preventable cases, was more reliable, had better face validity, and involved an acceptable workload.

Limitations of study
The external validity of perception of workload is questionable because it was assessed in a limited number of care teams without using a structured interview. The external validity of effectiveness and reliability may have been biased due to the small number of hospitals and wards. In medicine the preventable cases may have been under-represented because two of the 15 wards specialised in oncology and accounted for 23 of the 110 (21%) adverse events, of which only five were preventable. Because there is no gold standard for the reference list,¹¹ it may have contained errors because it was established by doctors, and not all the doctors involved in adverse events participated in this process. However, the validity of the list would have been questionable if doctors had agreed with cases identified by the prospective method (because they were involved in their identification) and had invalidated many of the cases identified by the retrospective method; only one case was invalidated. Eight patients who were still in the clinical wards at the beginning of the retrospective method (so medical record review could not be performed) had adverse events according to the prospective method. This may have resulted in bias in favour of the prospective method.

Strengths of study
One strength of our study was the different results shown according to ward type by stratification. Unlike our results, another study found that the prospective method detected twice to four times more complication rates in surgical patients than did a review of medical records.¹² These results may be due to differences in definitions or in data collection. The prospective method has proved more effective than the retrospective method for specific risks.^{13 14}

Data on the incidence of obstetric adverse events are sparse.¹⁵ Most studies focus on a limited number of events.¹⁶ The performance of the three methods was low. Further studies are needed since adverse events in obstetrics are of major importance because the patients are generally healthy women.

Overall, 11 of the 28 nosocomial infections in our study that should have been identified by the cross sectional method were missed. This method should be used cautiously because of the high rate of false positives and false negatives. Because we expected the cross sectional method to be less effective, we aimed to assess the extent of the lack of effectiveness.

Conclusions
Our results provide new insights into the epidemiology of adverse events. Firstly, they suggest ways to improve prospective assessment. For example, under identification could be reduced by providing doctors with detailed information before data collection and by reviewing medical records before interviewing the doctors. Secondly, our results suggest that the prospective method is preferable for assessing the impact of risk reduction programmes (better reliability), for convincing clinical teams that their errors contribute significantly to adverse events (better face validity resulting partly from team's involvement in data collection), for improving the assessment of consequences, and for studying organisational and human mishaps (especially better effectiveness in identifying preventable adverse events).

Box 2: Advantages and disadvantages of three methods used to estimate adverse event rates Prospective method (data collected during hospital stay) Advantages Best effectiveness for identifying preventable events Good reliability of judgment of iatrogenic nature of events Staff sufficiently involved to understand notion of iatrogenic risk and search for causes Preferred because of their pedagogical and communicative virtues Good appreciation of chain of events and their consequences Possible role as "red flag" for care providers during data collection Disadvantages Most expensive Heaviest workload, although perceived as acceptable: Several visits for investigators Staff must be available Cross sectional method (data gathered on given day) Advantages Least expensive Seamless continuation of former methodological approaches to iatrogenic risk Methodological approach fully understood by professionals and appreciated because it is rapid and easily renewed May be sufficient to justify implementation of a risk reduction policy and to define priorities Good reliability of judgment of iatrogenic nature of events; possible role as "red flag" for care providers during data collection Disadvantages Consequences of lack of follow up during patient's hospital stay: Lowest effectiveness Lack of validity due to measurement errors (false positives and false negatives) Prevalence biased by underestimation of frequency, particularly of deaths, and by over-representation of short stays Believed by unit staff to involve an excessive workload for obtaining imprecise estimations Inadequate to serve as initial estimation when evaluating impact of risk reduction policy Retrospective (review of medical records) Advantages Good effectiveness, even superior in surgery for estimating adverse event incidence Almost no workload for staff Data collection easily planned Method sometimes favoured by surgical teams and centres Disadvantages Difficulty to judge iatrogenic and preventable nature on basis of sometimes piecemeal data. Therefore: Measurement errors due to quality of medical records and to lesser reliability of judgment of iatrogenic nature Underestimation of preventable events Lower face validity of results, especially for preventability judgment (no involvement of staff)

What is already known on this topic Estimates of adverse event rates from large studies are based on review of medical records, despite limitations The epidemiology of adverse events is moving towards more analytical and evaluative goals for which these limitations may be serious issues What this study adds No reference method exists for identifying adverse events The retrospective method is appropriate for estimating rates of adverse events The prospective method, based on data gathered from wards, should be preferred for describing causes and consequences of adverse events and for evaluating risk reduction programmes The type of epidemiological method should be chosen according to objectives

---

Details of randomisation and data collection are on bmj.com

This is an abridged version; the full version is on bmj.com

We thank for their advice Charles Vincent, Maria Woloshynowych, and Graham Neale; Dominique Baubeau, Anne Broyard-Farge, Chantal Cases, Mireille Elbaum, Brigitte Haury, Jacques Massol, and Florence Veber (Ministry of Health); Pascal Astagneau, Françoise Haramburu, Lionel Pazart, and Marie-Laure Pibarot (national working group); the participating hospitals (university hospital of Bordeaux, Libourne and Langon public hospitals, Saint Augustin, Bordeaux Nord, Bordeaux Caudéran and Cenon Rive Droite private hospitals); the investigators, Françoise Zaro-Goni and Xavier Pelloquin (nurses), Delphine Caute and Nathalie Besson (midwives), Danielle Dreyfus, Sandrine Harston, and Michel Marcos (medicine), Alain Chabrol, Jean Pierre Claverie, and Astrid Sieber-Roth (surgery), and Jeanine Schirumberro, Philippe Cormier, and Richard Torrieli (obstetrics).

Contributors: See bmj.com

Funding: Financial support was provided by the Ministry of Health (Direction de la Recherche, des Etudes, de l'Evaluation et des Statistiques).

Competing interests: None declared.

Ethical approval: Not required.

References

1. Weingart SN, Wilson RM, Gibberd RW, Harrison B. Epidemiology of medical error. BMJ 2000;320: 774-7.[Free Full Text]
2. Wilson RM, Runciman WB, Gibberd RW, Harrisson BT, Newby L, Hamilton JD. The quality in Australian Health-Care Study. Med J Aust 1995;163: 458-71.[Web of Science][Medline]
3. Thomas EJ, Studdert DM, Burstin HR, Orav EJ, Zeena T, Williams EJ, et al. Incidence and types of adverse events and negligent care in Utah and Colorado. Med Care 2000;38: 261-71.[CrossRef][Web of Science][Medline]
4. Brennan TA, Leape LL, Laird NM, Hebert L, Localio AR, Lawthers AG. Incidence of adverse events and negligence care in hospitalized patients: results of the Harvard Medical Practice Study I. N Engl J Med 1991;324: 370-6.[Abstract]
5. Vincent C, Neale G, Woloshynowych M. Adverse events in British hospitals: preliminary retrospective record review. BMJ 2001;322: 517-9.[Abstract/Free Full Text]
6. Schioler T, Lipczak H, Pedersen BL, Mogensen TS, Bech KB, Stockmarr A, et al. [Incidence of adverse events in hospitals. A retrospective study of medical records]. Ugeskr Laeger 2001;163: 5370-8.[Medline]
7. Davis P, Lay-Yee R, Schug S, Briant R, Scott A, Johnson S, et al. Adverse events regional feasibility study: indicative findings. NZ Med J 2001;114: 203-5.[Medline]
8. Neale G, Woloshynowych M. Retrospective case record review: a blunt instrument that needs sharpening. Qual Safety Health Care 2003;12: 2-3.
9. Thomas EJ, Studdert DM, Brennan TA. The reliability of medical record review for estimating adverse event rates. Ann Intern Med 2002;136: 812-6.[Abstract/Free Full Text]
10. Weingart SN, Davis RB, Palmer RH, Cahalane M, Hamel MB, Mukamal K, et al. Discrepancies between explicit and implicit review: physician and nurse assessments of complications and quality. Health Serv Res 2002;37: 483-98.[CrossRef][Web of Science][Medline]
11. Brennan TA, Localio AR, Laird NM. Reliability and validity of judgments concerning adverse events and negligence. Med Care 1989;27: 1148-58.[Free Full Text]
12. Healey MA, Shackford SR, Osler TM, Rogers FB, Burns E. Complications in surgical patients. Arch Surg 2002;137: 611-7.[CrossRef][Web of Science][Medline]
13. Glenister HM, Taylor LJ, Bartlett CLR, Cooke EM, Sedgwick JA, Mackintosh CA. An evaluation of surveillance methods for detecting infections in hospitals inpatients. J Hosp Infect 1993;23: 229-42.
14. Flynn EA, Barker KN, Pepper GA, Bates DW, Mikeal RL. Comparison of methods for detecting medication errors in 36 hospitals and skilled-nursing facilities. Am J Health-Syst Pharm 2002;59: 436-46.[Web of Science][Medline]
15. Mantel GD, Biuchmann E, Rees H, Pattinson RC. Severe acute maternal morbidity: a pilot study of a definition for a near-miss. Br J Obstet Gynaecol 1998;105: 985-90.[Abstract/Free Full Text]
16. Waterstone M, Bewley S, Wolfe C. Incidence and predictors of severe obstetric morbidity: case control study. BMJ 2001;322: 1089-93.[CrossRef][Web of Science][Medline]

(Accepted 16 October 2003)

CiteULike    Complore    Connotea    Del.icio.us    Digg    Reddit    StumbleUpon    Technorati    What's this?

This article has been cited by other articles:

• Haller, G., Myles, P. S, Taffe, P., Perneger, T. V, Wu, C. L (2009). Rate of undesirable events at beginning of academic year: retrospective cohort study. BMJ 339: b3974-b3974 [Abstract] [Full text]  
• Smits, M, Wagner, C, Spreeuwenberg, P, van der Wal, G, Groenewegen, P P (2009). Measuring patient safety culture: an assessment of the clustering of responses at unit level and hospital level. Qual Saf Health Care 18: 292-296 [Abstract] [Full text]  
• Zegers, M, de Bruijne, M C, Wagner, C, Hoonhout, L H F, Waaijman, R, Smits, M, Hout, F A G, Zwaan, L, Christiaans-Dingelhoff, I, Timmermans, D R M, Groenewegen, P P, van der Wal, G (2009). Adverse events and potentially preventable deaths in Dutch hospitals: results of a retrospective patient record review study. Qual Saf Health Care 18: 297-302 [Abstract] [Full text]  
• Smits, M., Janssen, J., de Vet, R., Zwaan, L., Timmermans, D., Groenewegen, P., Wagner, C. (2009). Analysis of unintended events in hospitals: inter-rater reliability of constructing causal trees and classifying root causes. Int J Qual Health Care 21: 292-300 [Abstract] [Full text]  
• Mendes, W., Martins, M., Rozenfeld, S., Travassos, C. (2009). The assessment of adverse events in hospitals in Brazil. Int J Qual Health Care 21: 279-284 [Abstract] [Full text]  
• Soop, M., Fryksmark, U., Koster, M., Haglund, B. (2009). The incidence of adverse events in Swedish hospitals: a retrospective medical record review study. Int J Qual Health Care 21: 285-291 [Abstract] [Full text]  
• Camire, E., Moyen, E., Stelfox, H. T. (2009). Medication errors in critical care: risk factors, prevention and disclosure. CMAJ 180: 936-943 [Full text]  
• Aranaz-Andres, J M, Aibar-Remon, C, Vitaller-Murillo, J, Ruiz-Lopez, P, Limon-Ramirez, R, Terol-Garcia, E, the ENEAS work group, (2008). Incidence of adverse events related to health care in Spain: results of the Spanish National Study of Adverse Events. J. Epidemiol. Community Health 62: 1022-1029 [Abstract] [Full text]  
• Brown, C, Hofer, T, Johal, A, Thomson, R, Nicholl, J, Franklin, B D, Lilford, R J (2008). An epistemology of patient safety research: a framework for study design and interpretation. Part 3. End points and measurement. Qual Saf Health Care 17: 170-177 [Abstract] [Full text]  
• Hogan, H, Olsen, S, Scobie, S, Chapman, E, Sachs, R, McKee, M, Vincent, C, Thomson, R (2008). What can we learn about patient safety from information sources within an acute hospital: a step on the ladder of integrated risk management?. Qual Saf Health Care 17: 209-215 [Abstract] [Full text]  
• de Vries, E N, Ramrattan, M A, Smorenburg, S M, Gouma, D J, Boermeester, M A (2008). The incidence and nature of in-hospital adverse events: a systematic review. Qual Saf Health Care 17: 216-223 [Abstract] [Full text]  
• Sari, A. B.-A., Sheldon, T. A, Cracknell, A., Turnbull, A., Dobson, Y., Grant, C., Gray, W., Richardson, A. (2007). Extent, nature and consequences of adverse events: results of a retrospective casenote review in a large NHS hospital. Qual Saf Health Care 16: 434-439 [Abstract] [Full text]  
• Marang-van de Mheen, P. J., Hollander, E.-J. F., Kievit, J. (2007). Effects of study methodology on adverse outcome occurrence and mortality. Int J Qual Health Care 19: 399-406 [Abstract] [Full text]  
• Michel, P., Quenon, J. L., Djihoud, A., Tricaud-Vialle, S., de Sarasqueta, A. M. (2007). French national survey of inpatient adverse events prospectively assessed with ward staff. Qual Saf Health Care 16: 369-377 [Abstract] [Full text]  
• Wasson, J. H, MacKenzie, T. A, Hall, M. (2007). Patients use an internet technology to report when things go wrong. Qual Saf Health Care 16: 213-215 [Abstract] [Full text]  
• Olsen, S., Neale, G., Schwab, K., Psaila, B., Patel, T., Chapman, E J., Vincent, C. (2007). Hospital staff should use more than one method to detect adverse events and potential adverse events: incident reporting, pharmacist surveillance and local real-time record review may all have a place. Qual Saf Health Care 16: 40-44 [Abstract] [Full text]  
• Szekendi, M K, Sullivan, C, Bobb, A, Feinglass, J, Rooney, D, Barnard, C, Noskin, G A (2006). Active surveillance using electronic triggers to detect adverse events in hospitalized patients.. Qual Saf Health Care 15: 184-190 [Abstract] [Full text]  
• Marang-van de Mheen, P J, van Hanegem, N, Kievit, J (2005). Effectiveness of routine reporting to identify minor and serious adverse outcomes in surgical patients. Qual Saf Health Care 14: 378-382 [Abstract] [Full text]  
• Walsh, K E, Kaushal, R, Chessare, J B (2005). How to avoid paediatric medication errors: a user's guide to the literature. Arch. Dis. Child. 90: 698-702 [Abstract] [Full text]  
• Olsen, S., Neale, G. (2005). Clinical leadership in the provision of hospital care. BMJ 330: 1219-1220 [Full text]  
• Amalberti, R., Auroy, Y., Berwick, D., Barach, P. (2005). Five System Barriers to Achieving Ultrasafe Health Care. ANN INTERN MED 142: 756-764 [Abstract] [Full text]  
• Baker, G R (2004). HARVARD MEDICAL PRACTICE STUDY. Qual Saf Health Care 13: 151-152 [Full text]  

Rapid Responses:

Read all Rapid Responses

Issues on prospective method for estimating rates of adverse events

Dr Ali Baba-Akbari Sari
bmj.com, 26 Jan 2004 [Full text]

Prevention : the next step in the measurement of adverse events

Guy Haller, et al.
bmj.com, 5 Feb 2004 [Full text]

Other methods for rates of adverse events?

Roberto Natangelo
bmj.com, 20 Feb 2004 [Full text]

Comparison of Paired Proportions

Sanjeev Sarmukaddam, et al.
bmj.com, 1 Feb 2009 [Full text]

Online poll

Find out more>>