Re: How can we get high quality routine data to monitor the safety of devices and procedures?
Campbell et al, suggest an interesting series of measures that could be used to monitor the safety of devices and procedures . We agree that single measures may mislead but the problem of identifying a problem extends further than considering devices in isolation from the rest of clinical practice.
In 2012, there was a press outcry about the breast histopathology service at King’s Mill Hospital, Mansfield. Many patients were alleged to have been given incorrect treatment because an audit of breast tumour oestrogen-receptor positivity suggested that there was a significant difference between that Trust and others in the region. Consequently, the Care Quality Commission asked for a review of the pathology service to be carried out. This revealed that Kings’ Mill Hospital appeared to be an outlier only because of the small sample size of this service’s throughput. This made statistical evaluation of performance difficult  and the findings of the original audit invalid.
Similar problems were identified in the field of antenatal Down’s syndrome screening. These were resolved by the imposition of a minimum workload threshold for laboratories providing such screening tests. That minimum workload was based on statistical analysis of the volume work needed to provide a robust estimate of performance in a reasonable time period (1 year), and lead to the introduction of Down's Syndrome Screening Quality Assurance Support Service (DQASS) . Definition of minimum workloads forced reconfiguration of screening and has improved National performance.
All sorts of decisions made by departments and individuals can be monitored provided there are sufficient data. This cannot be achieved within the current parochial boundaries that define functional units in the NHS. Some reconfiguration of catchment areas and their volumes of work are needed to ensure sufficient data are available to make desirable and statistically valid interpretations of the quality of that work. It should be possible to identify with confidence that a unit is significantly different from its peers within a short enough timescale for patients to benefit from that knowledge. Thus, workload sizes need to be determined statistically so that the minimum number to give an acceptable confidence interval can be identified.
When a large number of a particular type of examination is carried out, it may also be possible to define a minimum acceptable workload of a clinician. This will allow outliers to be evaluated. This approach has been applied by DQASS with equal success to ultrasonographic measurement of nuchal thickness for first trimester Down’s screening. There is no reason to believe this cannot be applied to other areas of medical practice, whether for an objective numeric result or for a subjective opinion of an observation (interpreted as binary positive / negative) such as with oestrogen-receptor testing. Assessment of devices and procedures is insufficient without such sophisticated analysis.
1) Campbell B, Stainthorpe A, Longson C. How can we get high quality routine data to monitor the safety of devices and procedures? BMJ 2013;346:f2782
2) Royal College of Pathologists. Review of cellular pathology governance, breast reporting and immunohistochemistry at Sherwood Forest Hospitals NHS Foundation Trust: A report prepared for the Care Quality Commission in respect of diagnostic and screening procedure. 20 February 2013.
http://www.cqc.org.uk/sites/default/files/media/documents/20130313_full_... (accessed 13th May 2013).
3) DQASS. Homepage on the NHS Fetal Anomaly Screening Site. http://fetalanomaly.screening.nhs.uk/dqass
Competing interests: No competing interests