BMJ  2004;329:720 (25 September), doi:10.1136/bmj.38232.646227.DE (published 14 September 2004)

Information in practice

Surgical wound infection as a performance indicator: agreement of common definitions of wound infection in 4773 patients

¹ Department of Clinical Microbiology, University College London Hospitals, London WC1E 6DB, ² Department of Epidemiology and Population Health, London School of Hygiene and Tropical Medicine, London, ³ Department of Medical Physics, University College London Hospitals, ⁴ Department of Surgery, Medical School, Aberdeen, ⁵ Department of Public Health, Medical School, Aberdeen, ⁶ Nosocomial Infection Surveillance Unit, HPA Central Public Health Laboratory, London, ⁷ Health VFM Audit, National Audit Office, London

Abstract

Objective To assess the level of agreement between common definitions of wound infection that might be used as performance indicators.

Design Prospective observational study.

Setting London teaching hospital group receiving emergency cases as well as tertiary referrals.

Participants 4773 surgical patients staying in hospital at least two nights.

Main outcome measures Numbers of wound infections based on purulent discharge alone, on the Centers for Disease Control (CDC) definition of wound infection, on the nosocomial infection national surveillance scheme (NINSS) version of the CDC definition, and on the ASEPSIS scoring method.

Results 5804 surgical wounds were assessed during 5028 separate hospital admissions. The mean percentage of wounds classified as infected differed substantially with different definitions: 19.2% with the CDC definition (95% confidence interval 18.1% to 20.4%), 14.6% (13.6% to15.6%) with the NINSS version, 12.3% (11.4% to 13.2%) with pus alone, and 6.8% (6.1% to 7.5%) with an ASEPSIS score > 20. The agreement between definitions with respect to individual wounds was poor. Wounds with pus were automatically defined as infected with the CDC, NINSS, and pus alone definitions, but only 39% (283/714) of these had ASEPSIS scores > 20.

Conclusions Small changes made to the CDC definition or even in its interpretation, as with the NINSS version, caused major variation in estimated percentage of wound infection. Substantial numbers of wounds were differently classified across the grades of infection. A single definition used consistently can show changes in percentage wound infection over time at a single centre, but differences in interpretation prevent comparison between different centres.

Introduction

Surveillance of surgical site infection became mandatory in the NHS for orthopaedics in April 2004, and this will soon spread to other specialties.¹ The feedback of infection data to surgeons clearly reduces infection rates.^{2 3} Given that the percentage of wounds classified as infected will probably be used as a performance indicator,⁴ the new surveillance system must allow reliable comparisons to be made across NHS institutions, and with overseas health institutions.

There has been little or no critical evaluation of the definition of surgical site infection that is to be used for surveillance in England, namely the nosocomial infection national surveillance scheme (NINSS) version of the definition set out by the Centers for Disease Control (CDC) in 1992.⁵ Moreover, the version or interpretation of the definition used varies between hospitals and regions.^{6 7} Designers of a national surveillance system must judge the available definitions by their ability to identify infections that matter most to patients and to health services and by the practicability of collecting the required information.

We therefore compared agreement between four common definitions of surgical site infection—namely (a) the CDC 1992 definition, (b) the NINSS modification of the CDC definition, (c) the presence of pus, and (d) the ASEPSIS scoring method⁸—applied to the same series of surgical wounds. We also compared the percentage of infection based on the CDC definition and on the NINSS modification to investigate the potential effect of subjective CDC criteria and of variation between hospitals in data collection methods.

Participants and methods

Since May 2000, surgical wound surveillance has been conducted at University College London Hospitals. Cardiac, thoracic, orthopaedic, general, obstetric, gynaecological, urological, maxillofacial, plastic, and vascular surgical specialties have participated, each for at least six months each year. Only patients staying in hospital for at least two nights are included. Information is collected on patients and their surgical wounds, allowing us to apply the different definitions of wound infection.^{5 6 8 9}

Definitions of surgical site infection
The 1992 CDC definition requires the observation of 16 wound or patient characteristics in order to classify infection and has two subjective criteria, namely a surgeon's diagnosis of infection and the culture of microorganisms from the wound.⁵ The US national nosocomial infections surveillance system (NNISS) recommends that the latter criterion should be based only on positive cultures of fluid and tissue rather than wound swabs,^{5 7} but this interpretation does not seem to be applied generally.⁷ The English NINSS method modified the CDC definition to exclude the need for a surgeon's diagnosis and required that pus cells be present to satisfy the criterion of micro-organisms cultured from the wound.⁶ Another definition of infection simply requires the presence of pus, even though some infections are missed.⁹ ASEPSIS is a quantitative scoring method that provides a numerical score related to the severity of wound infection using objective criteria based on wound appearance and the clinical consequences of the infection.^{7 8}

For purposes of comparison, we classified ASEPSIS scores > 20 as infected. ASEPSIS scores of 10-20 ("disturbance of healing") are known to describe some infections, but most reflect wound breakdown due to other causes.¹⁰ Moderate to severe infections score > 30. The CDC definition also describes the severity of infection, classifying infections as "none," "superficial," or "deep or organ space" (termed "deep" in this article). Both definitions purport to describe the importance of an infection with respect to the patient's morbidity and the likely clinical consequences.

Data collection
Surveillance staff assessed patients every two or three days by direct observation, case note review, and questioning of nurses. We contacted patients by post or telephone one to two months after their operations to complete a questionnaire designed to ascertain late infections. Thus, we followed up patients either until their wounds had healed without infection or until an infection was detected. We therefore classified wounds as infected or not and recorded the proportion of wounds classified as infected at any time during follow up.

Statistical analysis
Counts and percentages presented are of wounds unless otherwise indicated. Confidence intervals for proportions of infection were adjusted for clustering on patient. We summarised agreement between the different definitions of infection using the statistic and the proportional agreement of ASEPSIS and CDC respectively for positive (Ppos) and negative (Pneg) diagnoses of infection.¹¹ Confidence intervals for the agreement statistics were adjusted for clustering on patient and calculated by bootstrap methods. The values shown are "bias-corrected."

Results

A total of 5804 surgical wounds in 4773 patients were assessed during 5028 separate hospital admissions to all surgical specialties in the hospital group between May 2000 and July 2003. The patients' median age was 53.5 years (interquartile range 37.5-69.6), and 2281 (48%) of the patients were female. The median hospital stay was 8 days (6-14), and duration of operation 111 minutes (62-180).

The mean percentage of wound infection differed substantially with the different definitions; 19.2% (95% confidence interval 18.1% to 20.4%) with the CDC definition, 14.6% (13.6% to 15.6%) with the NINSS version, 12.3% (11.4% to 13.2%) with pus alone, and 6.8% (6.1% to 7.5%) with an ASEPSIS score > 20. The table shows the level of agreement between the ASEPSIS and CDC systems. When superficial infections (according to CDC category) were included, 13% (778) of all observed wounds received conflicting diagnoses, and 6% were classified as infected by both definitions. When superficial infections were excluded, the two definitions estimated about the same overall percentage infection (6.8% and 7.0% respectively), but there were almost twice as many conflicting infection diagnoses (n=371) as concordant ones (n=215).

View this table: [in this window] [in a new window]   Comparison of crude rates of surgical site infection reported with Centers for Disease Control (CDC) 1992 definition and with ASEPSIS scoring method. Wounds were considered to be infected if they met the CDC criteria for either superficial or deep infection (top half of table) or if they met the criteria for deep infection only (bottom half of table). Values are numbers (percentages) of wounds, with 95% confidence intervals for percentages, adjusted for multiple wounds in the same patients

 

Wounds with pus were automatically diagnosed as infected by the CDC, NINSS, and pus alone definitions, but only 39% of these (283/714) had ASEPSIS scores > 20 (see figure). For these wounds, the CDC scale also consistently diagnosed greater infection severity than did ASEPSIS.

View larger version (38K): [in this window] [in a new window]   Comparison of diagnoses of surgical site infection in 5804 wounds reported with Centers for Disease Control (CDC) 1992 definition and with ASEPSIS scoring method, for wounds with and without pus

 

In wounds without pus the relation of ASEPSIS and CDC scales was less consistent (figure). For example, 42% (177/421) of wounds classified only as "disturbance of healing" by ASEPSIS were classified as infected by the CDC definition. Conversely, four of the six wounds classified as "severe wound infections" by ASEPSIS were classified as superficial by the CDC definition.

Comparison of wound classification with the CDC definition and with the NINSS version also revealed discrepancies. For example, more than 30% of wounds defined as superficially infected with CDC were classified as not infected with NINSS (229/709). In the CDC "superficial infection" category 94% (222/237) of the observed discrepancy was attributable to the NINSS modification of the CDC criterion related to positive bacterial cultures. In the CDC "deep infection" category the discrepancy observed was due to the exclusion of infections based solely on a surgeon's diagnosis.

Discussion

The four different definitions of surgical site infection varied widely in the estimated percentage of wounds infected. Comparing the 1992 CDC definition and the ASEPSIS scoring method, we found more than twice as many wounds were classified as infected by only one definition (n=778) as were classified as infected by both (n=366).

Potential limitations of this study
For the CDC definition, we often assumed the requirement for a surgeon's diagnosis of infection to be satisfied when a decision was made to start specific antibiotic treatment or to provide surgical treatment. In other studies, differences in results between CDC and other surveillance methods have been associated with lack of follow up, use of positive culture results, or clinical criteria.¹² Although our study was conducted in a single group of hospitals, data came from multiple sites, many surgical specialties, and a large number of surgeons, so that most of the relevant sources of variation were represented.

What is already known on this topic Surgical site infections are a major cause of morbidity and increased costs in health care The percentage of surgical wounds classified as infected is an obvious potential performance indicator, but common definitions have not been validated or compared What this study adds To assess the robustness of four common definitions of wound infection, their agreement in wound classification was determined Classifications with different definitions disagreed for more than twice as many wounds as those for which they agreed, and small changes in the interpretation of a definition caused substantial variation in the percentage of wounds classified as infected Although feedback of rates of wound infection within an institution using a consistent definition is effective in reducing infection rates, infection rates cannot be used as a performance indicator to compare hospitals without a more robust definition

Comparison of the different definitions
Other health outcome measures have been psychometrically evaluated, but similar information is lacking for most definitions of wound infection.¹³ ASEPSIS in its original form was reported to be repeatable and related to outcome,^{10 14} but it has since been modified and reproducibility is currently being reassessed. Choosing an optimal definition is therefore extremely difficult.

The absence of a clear pattern to the type of wounds classified as infected by CDC but as not infected by NINSS supports the view that the CDC criteria responsible for the discrepancy are difficult to apply consistently. Small changes made to the CDC definition or even to its interpretation, as with the NINSS version, cause substantial variation in the apparent percentage of infected wounds. Although the CDC definition has been adopted in many countries to allow international comparison, this faith seems unwarranted.

Conclusions
Using wound infection rates as a performance indicator to compare centres or countries is premature. Without a means to interpret absolute rates, such comparisons will be compromised by discrepancies in the way that infections are defined. External agencies should not judge the quality of medical care on these measures.¹⁵ Comparative performance tables should be reported only once a scientifically based and agreed definition has been produced.

---

This is the abridged version of an article that was posted on bmj.com on 14 September 2004: http://bmj.com/cgi/doi/10.1136/bmj.38232.646227.DE

We thank the members of the wound surveillance team (D Archibald, J Leach, and E O'Donnell).

Contributors: See bmj.com

Funding: Wound surveillance was supported by a start up grant from UCLH trustees for the first two years. Subsequently it has been funded directly by UCLH Trust. The analysis by CG reported in this paper has been funded by a grant from the National NHS R&D Research Methodology Programme. None of these funding sources have contributed to, or influenced the interpretation of, the analyses reported.

Competing interests: None declared.

Ethical approval: This was not deemed necessary as the surveillance was part of the hospital audit programme.

References

1. Donaldson L, Mullally S. Surveillance of healthcare associated infections. PLCMO2003/4, PLCNO2003/4. London: Department of Health, 2003.
2. Gil-Egea MJ, Pi-Sunyer MT, Verdaguer A, Sanz F, Sitges-Serra A, Eleizegui LT. Surgical wound infections: prospective study of 4,468 clean wounds. Infect Control 1987;8: 277-80.[Web of Science][Medline]
3. Mead PB, Pories SE, Hall P, Vacek PM, Davis JH Jr, Gamelli RL. Decreasing the incidence of surgical wound infections. Validation of a surveillance-notification program. Arch Surg 1986;121: 458-61.[Abstract/Free Full Text]
4. Department of Health. Indicator 23003. In: NHS performance indicators: a consultation. London: DoH, 2001: Annex 1, p 57.
5. Horan TC, Gaynes RP, Martone WJ, Jarvis WR, Emori TG. CDC definitions of nosocomial surgical site infections, 1992: a modification of CDC definitions of surgical wound infections. Infect Control Hosp Epidemiol 1992;13: 606-8.[Web of Science][Medline]
6. Wilson JA, Ward VP, Coello R, Charlett A, Pearson A. A user evaluation of the nosocomial infection national surveillance system: surgical site infection module. J Hosp Infect 2002;52: 114-21.[CrossRef][Web of Science][Medline]
7. Bruce J, Russell EM, Mollison J, Krukowski ZH. The measurement and monitoring of surgical adverse events. Health Technol Assess 2001;5(22): 1-194.[Medline]
8. Wilson AP, Treasure T, Sturridge MF, Gruneberg RN. A scoring method (ASEPSIS) for postoperative wound infections for use in clinical trials of antibiotic prophylaxis. Lancet 1986;i: 311-3.
9. Cruse PJ, Foord R. The epidemiology of wound infection. A 10-year prospective study of 62,939 wounds. Surg Clin North Am 1980;60: 27-40.[Web of Science][Medline]
10. Armitage P, Berry G, Matthews JNS. Statistical methods in medical research. 4th ed. Oxford: Blackwell, 2001: 703.
11. Wilson AP, Helder N, Theminimulle SK, Scott GM. Comparison of wound scoring methods for use in audit. J Hosp Infect 1998;39: 119-26.[CrossRef][Web of Science][Medline]
12. Colson M, Bolsin S. The use of statistical process control methods in monitoring clinical performance. Int J Qual Health Care 2003;15: 445.[Free Full Text]
13. Smith BH, Penny KI, Purves AM, Munro C, Wilson B, Grimshaw J, et al. The chronic pain grade questionnaire: validation and reliability in postal research. Pain 1997;71: 141-7.[CrossRef][Web of Science][Medline]
14. Byrne DJ, Malek MM, Davey PG, Cuschieri A. Postoperative wound scoring. Biomed Pharmacother 1989;43: 669-73.[CrossRef][Medline]
15. Lilford R, Mohammed MA, Spiegelhalter D, Thomson R. Use and misuse of process and outcome data in managing performance of acute medical care: avoiding institutional stigma. Lancet 2004;363: 1147-54.[CrossRef][Web of Science][Medline]

(Accepted 4 August 2004)

CiteULike    Complore    Connotea    Del.icio.us    Digg    Reddit    StumbleUpon    Technorati    What's this?

Relevant Articles

Mortality after Staphylococcus aureus bacteraemia in two hospitals in Oxfordshire, 1997-2003: cohort study

David H Wyllie, Derrick W Crook, and Tim E A Peto
BMJ 2006 333: 281. [Abstract] [Full Text] [PDF]

Wound infections are not classified consistently
BMJ 2004 329: 0. [Full Text]

This article has been cited by other articles:

• McCulloch, P. (2009). Developing appropriate methodology for the study of surgical techniques. JRSM 102: 51-55 [Full text]  
• Curran, E., Wilson, J. (2008). Using data effectively to prevent and control infection. British Journal of Infection Control 9: 26-33 [Abstract]  
• Wyllie, D. H, Crook, D. W, Peto, T. E A (2006). Mortality after Staphylococcus aureus bacteraemia in two hospitals in Oxfordshire, 1997-2003: cohort study. BMJ 333: 281- [Abstract] [Full text]  
• Phillips, J. E., Crane, T. P., Noy, M., Elliott, T. S. J., Grimer, R. J. (2006). The incidence of deep prosthetic infections in a specialist orthopaedic hospital: A 15-YEAR PROSPECTIVE SURVEY. J Bone Joint Surg Br 88-B: 943-948 [Abstract] [Full text]  

Rapid Responses:

Read all Rapid Responses

Surgical wound infection as a performance indicator: lack of methodological detail prevents interpretation

Emily S Petherick, et al.
bmj.com, 15 Oct 2004 [Full text]

Online poll

Find out more here>> and here>>