How to evaluate and improve the quality and credibility of an outcomes database: validation and feedback study on the UK Cardiac Surgery Experience

doi:10.1136/bmj.326.7379.25

BMJ 2003;326:25-28 ( 4 January )

Information in practice

How to evaluate and improve the quality and credibility of an outcomes database: validation and feedback study on the UK Cardiac Surgery Experience

Leon G Fine, professor and head, ^a Bruce E Keogh, consultant surgeon, ^b Shan Cretin, senior scientist, ^c Maria Orlando, associate behavioural scientist, ^c Mairi M Gould, project coordinator, ^a for the Nuffield-Rand Cardiac Surgery Demonstration Project Group.

^a Department of Medicine, Royal Free and University College Medical School, University College London, London WC1E 6JJ, ^b Department of Cardiothoracic Surgery, Queen Elizabeth Hospital, Birmingham B15 2TH, ^c RAND Health, PO Box 2138 Santa Monica, CA 90407-2138, USA

Correspondence to: L G Fine l.fine{at}ucl.ac.uk

Objectives: To assess the quality and completeness ofa database of clinical outcomes after cardiac surgery and to determinewhether a process of validation, monitoring, and feedback couldimprove the quality of thedatabase.
Design: Stratified sampling of retrospective datafollowed by prospective re-sampling of database after interventionof monitoring, validation, andfeedback.
Setting: Ten tertiary care cardiac surgery centresin the UnitedKingdom.
Intervention: Validation of data derived from a stratifiedsample of case notes (recording of deaths cross checked with mortuaryrecords), monitoring of completeness and accuracy of data entry,feedback to local data managers and leadsurgeons.
Main outcome measures: Average percentage missing data, average $kappa$ coefficient, and reliability score by centre for 17 variablesrequired for assignment of risk scores. Actual minus risk adjustedmortality in eachcentre.
Results: The database was incomplete, with a mean (SE)of 24.96% (0.09%) of essential data elements missing, whereasonly 1.18% (0.06%) were missing in the patient records (P<0.0001).Intervention was associated with (a) significantly less missingdata (9.33% (0.08%) P<0.0001); (b) marginal improvement in reliabilityof data and mean (SE) overall centre reliability score (0.53 (0.15)v 0.44 (0.17)); and (c) improved accuracy of assigned Parsonnetrisk scores ( $kappa$ 0.84 v 0.70). Mortality scores (actual minus riskadjusted mortality) for all participating centres fell withintwo standard deviations of the meanscore.
Conclusion: A short period of independent validation,monitoring, and feedback improved the quality of an outcomes databaseand improved the process of risk adjustment, but with substantialroom for further improvement. Wider application of this approachshould increase the credibility of similar databases before theirpublicrelease.

What is already know in this topic
Release of healthcare outcomes into the public domain has altered referral patterns and has led to improvement in some centres and elimination of others

The tacit assumption is that such outcomes data are accurate and can be relied on by the public and by healthcare providers to guide improvements

What this study adds
Sampling of a published national cardiac surgery database in England revealed it to be both incomplete and unreliable in its ability to yield accurate, risk adjusted outcomes data

An independent short process of monitoring, validation, and feedback improved the quality of the database

Such databases probably require an ongoing process of monitoring in order to allow data of adequate quality to be generated for the purpose of improving healthcare outcomes

CiteULike

Complore

Connotea

Del.icio.us Add to Digg

Digg

StumbleUpon Add to Technorati

Technorati What's this?

Relevant Article

Surgical outcomes database is incomplete and unreliable
BMJ 2003 326: 0. [Full Text]

This article has been cited by other articles:

Harboe, K. M., Anthonsen, K., Bardram, L. (2009). Validation of data and indicators in the Danish Cholecystectomy Database. Int J Qual Health Care 21: 160-168 [Abstract] [Full text]
Bridgewater, B, Keogh, B (2008). Surgical "league tables". Heart 94: 936-942 [Full text]
McCarthy, M., Datta, P., Sherlaw-Johnson, C., Coleman, M., Rachet, B. (2008). Is the performance of cancer services influenced more by hospital factors or by specialization?. J Public Health (Oxf) 30: 69-74 [Abstract] [Full text]
Coory, M., Duckett, S., Sketcher-Baker, K. (2008). Using control charts to monitor quality of hospital care with administrative data. Int J Qual Health Care 20: 31-39 [Abstract] [Full text]
Falcoz, P. E., Conti, M., Brouchet, L., Chocron, S., Puyraveau, M., Mercier, M., Etievent, J. P., Dahan, M. (2007). The Thoracic Surgery Scoring System (Thoracoscore): Risk model for in-hospital death in 15,183 patients requiring thoracic surgery. J. Thorac. Cardiovasc. Surg. 133: 325-332 [Abstract] [Full text]
Niland, J. C., Rouse, L., Stahl, D. C. (2006). An Informatics Blueprint for Healthcare Quality Information Systems. J. Am. Med. Inform. Assoc. 13: 402-417 [Abstract] [Full text]
Al-Mallah, M. H., Tleyjeh, I. M., Abdel-Latif, A. A., Weaver, W. D. (2006). Angiotensin-Converting Enzyme Inhibitors in Coronary Artery Disease and Preserved Left Ventricular Systolic Function: A Systematic Review and Meta-Analysis of Randomized Controlled Trials. J Am Coll Cardiol 47: 1576-1583 [Abstract] [Full text]
Leandro, G, Rolando, N, Gallus, G, Rolles, K, Burroughs, A K (2005). Monitoring surgical and medical outcomes: the Bernoulli cumulative SUM chart. A novel application to assess clinical interventions. Postgrad. Med. J. 81: 647-652 [Abstract] [Full text]
Ambler, G., Omar, R. Z., Royston, P., Kinsman, R., Keogh, B. E., Taylor, K. M. (2005). Generic, Simple Risk Stratification Model for Heart Valve Surgery. Circulation 112: 224-231 [Abstract] [Full text]
McDonnell, A., Nicholl, J., Read, S. (2005). Exploring the impact of Acute Pain Teams (APTs) on patient outcomes using routine data: Can it be done?. Journal of Research in Nursing 10: 383-402 [Abstract]
Hilton, C. J., Hamilton, J.R. L., Vitale, N., Haaverstad, R. (2005). Effects of 'Bristol' on surgical practice in the United Kingdom. ICVTS 4: 197-199 [Abstract] [Full text]
Keogh, B., Spiegelhalter, D., Bailey, A., Roxburgh, J., Magee, P., Hilton, C. (2004). The legacy of Bristol: public disclosure of individual surgeons' results. BMJ 329: 450-454 [Full text]
Treasure, T. (2004). Congenital heart disease. BMJ 328: 594-595 [Full text]
Sheldon, T A (2003). Basing decisions on better quality routine clinical data. Qual Saf Health Care 12: 327-328 [Full text]
Treasure, T., Utley, M., Bailey, A. (2003). Assessment of whether in-hospital mortality for lobectomy is a useful standard for the quality of lung cancer surgery: retrospective study. BMJ 327: 73- [Abstract] [Full text]
Bridgewater, B., Grayson, A. D, Jackson, M., Brooks, N., Grotte, G. J, Keenan, D. J M, Millner, R., Fabri, B. M, Jones, M. (2003). Surgeon specific mortality in adult cardiac surgery: comparison between crude and risk stratified data. BMJ 327: 13-17 [Abstract] [Full text]
Marshall, M. N., Shekelle, P. G., Davies, H. T.O., Smith, P. C. (2003). Public Reporting On Quality In The United States And The United Kingdom. Health Aff (Millwood) 22: 134-148 [Abstract] [Full text]