Information In Practice

How to evaluate and improve the quality and credibility of an outcomes database: validation and feedback study on the UK Cardiac Surgery Experience

BMJ 2003; 326 doi: http://dx.doi.org/10.1136/bmj.326.7379.25 (Published 04 January 2003) Cite this as: BMJ 2003;326:25
  1. Leon G Fine, professor and heada (l.fine{at}ucl.ac.uk),
  2. Bruce E Keogh, consultant surgeonb,
  3. Shan Cretin, senior scientistc,
  4. Maria Orlando, associate behavioural scientistc,
  5. Mairi M Gould, project coordinatora

    for the Nuffield-Rand Cardiac Surgery Demonstration Project Group

  1. a Department of Medicine, Royal Free and University College Medical School, University College London, London WC1E 6JJ
  2. b Department of Cardiothoracic Surgery, Queen Elizabeth Hospital, Birmingham B15 2TH
  3. c RAND Health, PO Box 2138 Santa Monica, CA 90407-2138, USA
  1. Correspondence to: L G Fine
  • Accepted 4 October 2002

Abstract

Objectives: To assess the quality and completeness of a database of clinical outcomes after cardiac surgery and to determine whether a process of validation, monitoring, and feedback could improve the quality of the database.

Design: Stratified sampling of retrospective data followed by prospective re-sampling of database after intervention of monitoring, validation, and feedback.

Setting: Ten tertiary care cardiac surgery centres in the United Kingdom.

Intervention: Validation of data derived from a stratified sample of case notes (recording of deaths cross checked with mortuary records), monitoring of completeness and accuracy of data entry, feedback to local data managers and lead surgeons.

Main outcome measures: Average percentage missing data, average κ coefficient, and reliability score by centre for 17 variables required for assignment of risk scores. Actual minus risk adjusted mortality in each centre.

Results: The database was incomplete, with a mean (SE) of 24.96% (0.09%) of essential data elements missing, whereas only 1.18% (0.06%) were missing in the patient records (P<0.0001). Intervention was associated with (a) significantly less missing data (9.33% (0.08%) P<0.0001); (b) marginal improvement in reliability of data and mean (SE) overall centre reliability score (0.53 (0.15) v 0.44 (0.17)); and (c) improved accuracy of assigned Parsonnet risk scores (κ 0.84 v 0.70). Mortality scores (actual minus risk adjusted mortality) for all participating centres fell within two standard deviations of the mean score.

Conclusion: A short period of independent validation, monitoring, and feedback improved the quality of an outcomes database and improved the process of risk adjustment, but with substantial room for further improvement. Wider application of this approach should increase the credibility of similar databases before their public release.

What is already know in this topic

What is already know in this topic Release of healthcare outcomes into the public domain has altered referral patterns and has led to improvement in some centres and elimination of others

The tacit assumption is that such outcomes data are accurate and can be relied on by the public and by healthcare providers to guide improvements

What this study adds

What this study adds Sampling of a published national cardiac surgery database in England revealed it to be both incomplete and unreliable in its ability to yield accurate, risk adjusted outcomes data

An independent short process of monitoring, validation, and feedback improved the quality of the database

Such databases probably require an ongoing process of monitoring in order to allow data of adequate quality to be generated for the purpose of improving healthcare outcomes

Footnotes

  • Funding This project was supported by a grant from the Nuffield Trust. The Nuffield Trust had a representative on the steering committee but played no role in determining the final design of the study or in the analysis and presentation of the results.

  • Competing interests None declared.

  • Embedded Image Members of the project group are listed on bmj.com

    Editorial by Black

  • Accepted 4 October 2002
View Full Text