Performance league tables: the NHS deserves betterBMJ 2002; 324 doi: https://doi.org/10.1136/bmj.324.7329.95 (Published 12 January 2002) Cite this as: BMJ 2002;324:95
Text for web only
Constructing control charts
The first step in constructing control charts is ensuring that all providers (such as the 37 hospitals in figs 2 and 3) comprise parts of the same single system. The use of control charts over several evaluation cycles allows us to check whether the single system premise is true. If it is not, hospitals can be regrouped into different systems.
The second step is ensuring that all providers have high quality monitoring systems, with standardised data collection. Providers that are not able to comply with this standard should not be included in the performance monitoring exercise.
The third step is the calculation of control limits. Suitable limits, like the limits set for any diagnostic test, must balance the disadvantages of erroneously concluding that a special cause is present when it is not (the clinical equivalent of a false positive diagnosis) and the disadvantages of failing to detect the presence of a special cause (false negative result). Years of experience from industry have shown that control limits with high specificity are the most appropriate.w1 Experience gained from the health service also suggests that, where the prevalence of a condition is low (as far as we know truly poor performance among NHS providers is rare), the consequences of inappropriate intervention are serious, and the cost of intervention is high, a test with high specificity is appropriate.w2
Constructing control limits
Control limits may be drawn in two ways.
Eyeballing—When many readings are available eyeballing has been shown to be successful for constructing control limits. Figure A shows over 200 readings of the annual rainfall in the United Kingdom since 1766. Most readers, if asked to sketch in upper and lower control limits for normal rainfall, would draw lines close to those shown (range 600-1220 mm/year). All would agree that 1872 was a wet year.
(F1) Fig A Control chart showing annual rainfall in England and Wales during 1766-1999
Statistical—When fewer readings are available the eyeball method becomes somewhat arbitrary. In such circumstances the estimated point performance of the system is calculated from the pooled performance data of all providers. An assumption is now made about the distribution of the performance parameter. Performance limits are then calculated using a specific 3-σ test criterion (equivalent to 98.8%). When these rules are applied to binomial performance data such as "proportion of patients who die within 30 days" these control limits are the same as the conventional 99.8% confidence intervals.
- Shewhart WA. Economic control of quality of manufactured product. New York: D Van Norstrand, 1931. (Reprinted by ASQC Quality Press, Wisconsin, 1980).
- Sackett DL, Haynes RB, Guyatt GH, Tugwell P. Clinical epidemiology. A basic science for clinical medicine. Boston: Little Brown, 1991.
- River blindness in East Africa: gains and lossesBMJ January 21, 2020, 368 m155; DOI: https://doi.org/10.1136/bmj.m155
- Association of ideal cardiovascular health at age 50 with incidence of dementia: 25 year follow-up of Whitehall II cohort studyBMJ August 07, 2019, 366 l4414; DOI: https://doi.org/10.1136/bmj.l4414
- Genetic predisposition to increased serum calcium, bone mineral density, and fracture risk in individuals with normal calcium levels: mendelian randomisation studyBMJ August 01, 2019, 366 l4410; DOI: https://doi.org/10.1136/bmj.l4410
- Dietary fats and mortality among patients with type 2 diabetes: analysis in two population based cohort studiesBMJ July 02, 2019, 366 l4009; DOI: https://doi.org/10.1136/bmj.l4009
- Physical activity trajectories and mortality: population based cohort studyBMJ June 26, 2019, 365 l2323; DOI: https://doi.org/10.1136/bmj.l2323
- FRCS first pass variance: deanery and specialty contrariety
- Validation of quality indicators for end-of-life communication: results of a multicentre survey
- Hospital variation in allogeneic transfusion and extended length of stay in primary elective hip and knee arthroplasty: a cross-sectional study
- Random variation and rankability of hospitals using outcome indicators
- Displaying random variation in comparing hospital performance
- Plotting basic control charts: tutorial notes for healthcare practitioners
- Effect of an acute medical admission unit on key quality indicators assessed by funnel plots
- Measuring the performance of urban healthcare services: results of an international experience
- Control, compare and communicate: designing control charts to summarise efficiently data from multiple quality indicators
- Using statistical process control to improve the quality of health care
- Mortality control charts for comparing performance of surgical units: validation study using hospital mortality data
- Using routine comparative data to assess the quality of health care: understanding and avoiding common pitfalls
- Monitoring mortality rates in general practice after Shipman
- Funnel plots for institutional comparison