BMJ  2007;334:79 (13 January), doi:10.1136/bmj.39031.507153.AE (published 15 December 2006)

Research

Sensitivity of routine system for reporting patient safety incidents in an NHS hospital: retrospective patient case note review

¹ Department of Health Sciences, University of York, York YO10 5DD, ² Leeds General Infirmary, Leeds LS1 3EX, ³ York Hospital, York YO31 8HE

Abstract

Objective To evaluate the performance of a routine incident reporting system in identifying patient safety incidents.

Design Two stage retrospective review of patients' case notes and analysis of data submitted to the routine incident reporting system on the same patients.

Setting A large NHS hospital in England.

Population 1006 hospital admissions between January and May 2004: surgery (n=311), general medicine (n=251), elderly care (n=184), orthopaedics (n=131), urology (n=61), and three other specialties (n=68).

Main outcome measures Proportion of admissions with at least one patient safety incident; proportion and type of patient safety incidents missed by routine incident reporting and case note review methods.

Results 324 patient safety incidents were identified in 230/1006 admissions (22.9%; 95% confidence interval 20.3% to 25.5%). 270 (83%) patient safety incidents were identified by case note review only, 21 (7%) by the routine reporting system only, and 33 (10%) by both methods. 110 admissions (10.9%; 9.0% to 12.8%) had at least one patient safety incident resulting in patient harm, all of which were detected by the case note review and six (5%) by the reporting system.

Conclusion The routine incident reporting system may be poor at identifying patient safety incidents, particularly those resulting in harm. Structured case note review may have a useful role in surveillance of routine incident reporting and associated quality improvement programmes.

Introduction

Patient safety incidents (defined as any unintended event caused by the health care that either did or could have led to patient harm) have been shown to cause harm in between 3% and 17% of hospital inpatients.^{1 2 3 4 5}

After the development of the national risk management standards in 1995, most NHS hospitals in England and Wales established reporting systems as part of their risk management programme.⁶ People involved in or witnessing a patient safety incident complete a form that is sent to the local reporting system, where the incident is classified and entered into a database.⁷ The National Patient Safety Agency developed a national reporting and learning system in 2003 to collate reports of patient safety incidents from local organisations.⁸ This system aims to help the NHS to learn from patient safety incidents and to identify trends and patterns relating to patient safety.^{8 9} The system should, therefore, be able to identify a representative sample of patient safety incidents and provide adequate data about the cause, contributory factors, preventability, and impact of these incidents.^{9 10} In this paper we evaluate the relative performance of a local routine incident reporting system that feeds into the national reporting and learning system, by comparing it with a well validated method of systematically reviewing case notes.^{1 2 3}

Methods

We did the study in a large NHS hospital trust in England in 2005. We selected a stratified random sample of 1006 admissions (>24 hours' stay) between January and May 2004 from eight specialties: surgery; urology; orthopaedics; general medicine; medicine for the elderly; oncology; ear, nose, and throat; and ophthalmology. All data extracted were anonymised and kept confidential. The study consisted of using structured data extraction tools to do a two stage retrospective case note review of the sample admissions and reviewing the patient safety incidents reported by the routine hospital reporting system for the same admissions.

Review of medical records
We used previously described methods to do the case note review.^{1 2 3} Five trained nurses screened patients' records by using 18 explicit criteria (box). We used one (or more) positive criterion as an indicator of a patient safety incident and scrutinised these medical records in stage two. One of the other nurses independently reviewed a 10% sample to assess inter-rater reliability. In addition, medical staff fully reviewed 10% of admissions for which no positive criteria were identified to identify false negatives (fig 1).

Screening criteria used in the first stage of case note review1 Unplanned admission related to previous healthcare management Injury to patient incurred in hospital (for example, fall, burn, pressure ulcer) Other complications (for example, DVT, MI, CVA, PE, ARF) Hospital acquired infection/sepsis Cardiac/respiratory arrest; low Apgar score Adverse drug reaction Unplanned transfer from general care to intensive care Unplanned transfer to another acute care hospital Unplanned return to the operating theatre Unplanned removal of, injury to, or repair of an organ during surgery Development of neurological deficit not present on admission Inappropriate discharge to home Unexpected death Injury related to abortion or delivery Dissatisfaction with care documented in medical record Documentation or correspondence indicating litigation Any other undesirable outcomes not covered above Unplanned readmission after discharge from index admission ARF=acute renal failure; CVA=cerebral vascular accident; DVT=deep vein thrombosis; MI=myocardial infarction; PE=pulmonary embolism.

View larger version (29K): [in this window] [in a new window] [PowerPoint Slide for Teaching]   Fig 1 Summary of case note review process. RF1=review form 1; RF2=review form 2

 
In stage two, three hospital doctors reviewed the records that had one positive criterion in stage one. The doctors were trained to use a structured review form to judge if a patient safety incident had occurred and to assess its type and consequences. One of the other doctors independently reviewed 90 medical records to assess the inter-rater reliability (fig 1).

Review of patient safety incidents reported by routine reporting system
We inspected data on the routine adverse incident reporting system for the 1006 admissions in our sample to see if patient safety incidents had been reported. We calculated the number, percentage, and type of patient safety incidents identified by the case note review and routine reporting system and classified them into three groups according to the routine reporting system policy (table 1). We calculated the proportion of admissions with patient safety incidents identified by each method and the proportion of these incidents that were judged to have caused patient harm for each method, along with 95% confidence intervals. We used Cohen's to assess the inter-rater reliability.¹¹

View this table: [in this window] [in a new window]   Table 1  Type of patient safety incidents recorded by case note review (CNR) and routine adverse incident reporting system (AIRS). Values are numbers (percentages)

 

Results

Patient safety incidents—According to a combination of case note review and the reporting system, a total of 324 patient safety incidents were reported in 230 of the 1006 admissions (22.9%; 95% confidence interval 20.3% to 25.5%). Case note review identified 303 (94%) of the 324 incidents. The reporting system identified 54 (17%) of the total number of patient safety incidents, all of them of group I type (table 1).

Patient safety incidents causing harm to patients (adverse events)—Of the 1006 admissions, 110 (10.9%; 9.0% to 12.8%) had at least one patient safety incident resulting in harm to the patient (a total of 136 adverse events). In other words, 42% of patient safety incidents resulted in adverse events, of which all were detected by the case note review and 6 (5%) by the reporting system. All 21 patient safety incidents missed by case note review were minor (fig 2), whereas 130 (44.7%) incidents missed by the reporting system led to patient harm.

View larger version (23K): [in this window] [in a new window] [PowerPoint Slide for Teaching]   Fig 2 Group I patient safety incidents reported by case note review (CNR) and adverse incident reporting system (AIRS)

 

Discussion

We found that 23% of hospital admissions in eight specialties were associated with patient safety incidents and 11% with adverse events. This is similar to rates found in studies using similar methods in the United Kingdom (10.8%)¹ and internationally (7.5% to 16.6%).^{2 3 4 5}

The routine reporting system as implemented in this large hospital missed most patient safety incidents that were identified by case note review and detected only 5% of those incidents that resulted in patient harm. This suggests that the routine reporting system considerably under-reports the scale and severity of patient safety incidents.

Strengths and weaknesses
Structured case note review, when carried out by trained professionals, has been shown to reliably detect adverse events.^{1 2 3 6 12} The reviewers in this study were specifically trained, and inter-rater reliability was good at both stages¹¹: 84% between nurses in the first stage (=0.67) and 90% between doctors (=0.76) in the second stage (table 2).

View this table: [in this window] [in a new window]   Table 2  Agreement between nurses on presence of screening criteria and between doctors on presence of patient safety incident (PSI)

 
This study is based on data from one large hospital, where the performance of the incident reporting system may differ from that in other hospitals. However, this trust is a high reporter to the national reporting and learning system and the distribution of the types of patient safety incidents detected in this study was similar to that found in a recent analysis of patient safety incidents from 230 NHS organisations.⁸ Some hospitals report hospital acquired infections to systems other than the adverse incident reporting system¹³; however, even excluding infections, the reporting system detected only 24% of all patient safety incidents and only 5% of those resulting in patient harm. This suggests that our results may be generalisable.

Meaning of the study
A recent report by the House of Commons Committee of Public Accounts¹⁴ was critical of the adequacy of the national reporting and learning system. Our study provides empirical evidence that the data collected by the system may be biased. This is unlikely to be caused by teething problems, as the national reporting system was designed to complement pre-existing local reporting arrangements.⁷ If the NHS is to gather accurate information on serious injuries and deaths resulting from patient safety incidents, as recommended by the Committee of Public Accounts,¹⁴ then relying on voluntary reporting may not be sufficient. Voluntary reporting systems may under-report incidents, owing to lack of feedback; time constraints; fear of shame, blame, litigation, or professional censure; and unsatisfactory processes.^{15 16 17 18 19}

The results do not mean that the early themes emerging from the analysis of the national reporting and learning system data are not useful,⁸ but estimates of the type and severity of incidents are likely to be biased. More importantly, perhaps, the value of these data locally as a component of safety programmes is questionable.

Future research
More research is needed to help to develop a reporting system that is capable of providing an accurate picture of the type, nature, and severity of incidents and at reasonable cost. Even if detection is improved, this will not in itself result in improvements in patient safety. We need to develop and evaluate cost effective ways in which good data monitoring can be used as part of quality improvement.

Conclusion
The routine incident reporting system may not provide an accurate picture of the extent and severity of patient safety incidents, particularly those resulting in harm to patients. Healthcare organisations should consider routinely using structured case note review on samples of medical records as part of quality improvement.

What is already known on this topic Patient safety incidents are common in inpatients, and many of them lead to patient harm or extra cost Effective strategies are needed to identify, analyse, and learn from these incidents Case note review is useful but expensive for routine use; routine reporting systems have been introduced in several countries What this study adds Routine incident reporting systems may significantly under-report patient safety incidents, particularly those resulting in harm Structured case note review may have a role in surveillance and the monitoring of reporting systems

---

We thank Alan Maynard, Mike White, and Michael Porte for their support and advice. We also thank for their advice Denis Smith, Carl Thompson, Fiona Fylan, Richard Lilford, Charles Vincent, Graham Neale, Maria Woloshynowych, Martin Bland, Jeremy Miles, Ian Woods, Ann McEvoy, Donald Richardson, Glen Miller, Caroline Mosely, Dawn Taylor, Mary Nannary, and Sally Grabham. We are also grateful to clinical and administrative staff of the host hospital for their support.

Contributors: AB-AS designed and managed the project, wrote the research proposal, collected and analysed data, and wrote the final paper. TAS supervised the project; commented on the protocol, data collection, and analysis; was responsible for the quality control; and assisted on the paper. AC and AT piloted the instruments and process, assisted with stage two case note review, provided advice, and commented on the paper. Celia Grant, Eileen Richardson, William Gray, Yvonne Dobson, and Lorraine Wright screened the medical records and collected data in stage one and discussed the findings. AB-AS is the guarantor.

Funding: AB-AS was supported by a scholarship from the Iranian Ministry of Health and now works at the School of Public Health, Teheran University of Medical Sciences. All the researchers are independent from the Iranian Ministry of Health.

Competing interest: None declared.

Ethical approval: Hospital research ethics committee (reference number 04/Q1108/7).

References

1. Vincent C, Neale G, Woloshynowych M. Adverse events in British hospitals: preliminary retrospective record review. BMJ 2001;322:517-9.[Abstract/Free Full Text]
2. Brennan T, Leape L, Laird N, Hebert L, Localio A, Lawthers A, et al. Incidence of adverse events and negligence in hospitalized patients: results of the Harvard medical practice study I. N Engl J Med 1991;324:370-6.[Abstract]
3. Wilson R, Runciman W, Gibberd R, Harrison B, Newby L, Hamilton J. The quality in Australian health care study. Med J Aust 1995;163:458-71.[ISI][Medline]
4. Davis P, Lay YR, Briant R, Ali W, Scott A, Schug S. Adverse events in New Zealand public hospitals I: occurrence and impact. N Z Med J 2002;115:U271.[Medline]
5. Baker G, Norton P, Flintoft V, Blais R, Brown A, Cox J, et al. The Canadian adverse events study: the incidence of adverse events among hospital patients in Canada. CMAJ 2004;170:1678-86.[Abstract/Free Full Text]
6. Vincent C. Clinical risk management. London: BMJ Books, 2001.
7. Vincent C. Patient safety. Edinburgh: Churchill Livingstone, 2006.
8. National Patient Safety Agency. Building a memory: preventing harm, reducing risk and improving patient safety. The first report of the national reporting and learning system and the patient safety observatory. London: NPSA, 2005.
9. Kilbridge P, Classen D. Automated surveillance for adverse events in hospitalized patients: back to the future. Qual Saf Health Care 2006;15:148-9.[Free Full Text]
10. Giles S, Fletcher M, Baker M, Thomson R. Incident reporting and analysis current status and future challenges. In: Walshe K, Boaden R, eds. Patient safety: research into practice. Maidenhead: Open University Press, 2006.
11. Campbell MJ, Machin D. Medical statistics: a commonsense approach. 3rd ed. Chichester: J Wiley, 1999.
12. Thomas E, Petersen L. Measuring errors and adverse events in health care. J Gen Intern Med 2003;18:61-7.[CrossRef][ISI][Medline]
13. National Audit Office. Improving patient care by reducing the risk of hospital acquired infection: a progress report. London: Stationery Office, Comptroller and Auditor General, 2004.
14. House of Commons Committee of Public Accounts. A safer place for patients: learning to improve patient safety. London: Stationery Office, July 2006.
15. Uribe C, Schweikhart S, Pathak D, Dow M, Marsh G. Perceived barriers to medical-error reporting: an exploratory investigation. J Healthc Manag 2002;47:263-79.[ISI][Medline]
16. Evans S, Berry J, Smith B, Esterman A, Selim P, O'Shaughnessy J, et al. Attitudes and barriers to incident reporting: a collaborative hospital study. Qual Saf Health Care 2006;15:39-43.[Abstract/Free Full Text]
17. Kingston M, Evans S, Smith B, Berry J. Attitudes of doctors and nurses towards incident reporting: a qualitative analysis. Med J Aust 2004;181:27-8.[ISI][Medline]
18. Walsh T, Beatty P. Human factors error and patient monitoring. Physiol Meas 2002;23:R111-32.[CrossRef][ISI][Medline]
19. Firth Cozens J. Barriers to incident reporting. Qual Saf Health Care 2002;11:7.[Free Full Text]

(Accepted 3 November 2006)

Related Articles

Seeing the picture through "lean thinking"

David I Ben-Tovim
BMJ 2007 334: 169. [Extract] [Full Text] [PDF]

Routine reporting

Fiona Godlee
BMJ 2007 334: 0. [Extract] [Full Text]

Safety at what cost?

Douglas Kamerow
BMJ 2007 334: 0. [Extract] [Full Text]

Incident reporting and patient safety

Charles Vincent
BMJ 2007 334: 51. [Extract] [Full Text] [PDF]

Adverse events in British hospitals: preliminary retrospective record review

Charles Vincent, Graham Neale, and Maria Woloshynowych
BMJ 2001 322: 517-519. [Abstract] [Full Text] [PDF]

This article has been cited by other articles:

• Sari, A. B. A., Cracknell, A., Sheldon, T. A. (2008). Incidence, preventability and consequences of adverse events in older people: results of a retrospective case-note review. Age Ageing 37: 265-269 [Abstract] [Full text]  
• Oliver, D. (2008). Falls risk-prediction tools for hospital inpatients. Time to put them to bed?. Age Ageing 37: 248-250 [Full text]  
• Sari, A. B.-A., Sheldon, T. A, Cracknell, A., Turnbull, A., Dobson, Y., Grant, C., Gray, W., Richardson, A. (2007). Extent, nature and consequences of adverse events: results of a retrospective casenote review in a large NHS hospital. Qual Saf Health Care 16: 434-439 [Abstract] [Full text]  
• Munro, A J (2007). Hidden danger, obvious opportunity: error and risk in the management of cancer. Br. J. Radiol. 80: 955-966 [Full text]  
• Ben-Tovim, D. I (2007). Seeing the picture through "lean thinking". BMJ 334: 169-169 [Full text]  
• Vincent, C. (2007). Incident reporting and patient safety. BMJ 334: 51-51 [Full text]  

Rapid Responses:

Read all Rapid Responses

Patient safety is a symptom of our dysfunctional organisations

John Matthews
bmj.com, 21 Dec 2006 [Full text]

The incident that never happened

Christian P Subbe
bmj.com, 21 Dec 2006 [Full text]

Adverse incidents in emergency and elective patients

Ram Kumar
bmj.com, 21 Dec 2006 [Full text]

Incident reporting - but one spoke in the wheel of healthcare safety data

John H Williamson
bmj.com, 8 Jan 2007 [Full text]

Perhaps of some interest

Jeffrey C McILwain
bmj.com, 12 Jan 2007 [Full text]

Case review in practice

Grace Barden, et al.
bmj.com, 14 Jan 2007 [Full text]

Seeing the whole picture through Lean thinking eyes.

David I Ben-Tovim
bmj.com, 14 Jan 2007 [Full text]

Radiotherapy safety needs lean thinking

Katharine J Tylko, et al.
bmj.com, 16 Jan 2007 [Full text]

critical incident reporting from primary care

Mark F Lambert, et al.
bmj.com, 23 Jan 2007 [Full text]

Giving incident reporting systems a chance

Guy Haller
bmj.com, 24 Jan 2007 [Full text]