How to identify when a performance indicator has run its course

Comment on David Reeves and colleagues, “How to identify when a
performance indicator has run its course” (BMJ 2010;340:c1717)

Recently David Reeves and colleagues explained the rationale behind
the removal of eight performance indicators from the UK Quality and
Outcomes Framework. (1)
The indicators needed to be reviewed and were evaluated for reaching the
‘ceiling of achievement’. During the revision process the indicators were
assessed in at least five ways: what was the average rate and trend of
achievement; is there substantial variation in achievement rates; what is
the rate and trend in exception reporting and what is the variation in
exception reporting. When the achievement rate is high and constant, the
exception rate is low and constant and when there is little variation
among hospitals, the performance indicator has reached its plateau and was
a candidate for replacement. The decision for removal could be influenced
by additional contextual factors such as policy arguments, feasibility,
requests of patient groups or new scientific insights and revised
guidelines.

In the Netherlands, performance indicators are currently under
development and revision as well and the same issues are considered for
the replacement of indicators. Though, in both the Dutch and the UK
discussion, a critical aspect is underexposed: the validity of performance
indicators. Does the indicator measure what it is intended to measure? To
ensure validity there are several requirements for indicators, concerning
the reliability of the data, the comparability of the results, and the
statistical reliability.(2) This should be evaluated when revising the
indicators.

A reliable hospital comparison requires that the data source is
complete and the gathering of data is done in a uniform way. Because of
that, registrations should include all pre-defined patients. And
definitions of the required data should be clear. In addition, the
comparability of the results requires that differences in patient
populations are taken into account. Therefore, the results should be
adjusted for differences in case mix. Finally, the results have to be
statistically reliable. This means that when the results of hospitals are
compared, statistical random variability should be taken into account.

We will illustrate this with a study we performed in collaboration
with the comprehensive cancer centre west, to test the validity of the
quality indicator ‘tumour positive margin rate after breast conserving
surgery for breast neoplasm’, which is used by two large healthcare
parties in the Netherlands: the Dutch Healthcare Inspectorate and the
Healthcare Transparency Program.(3) We tested the validity of the
indicator, in relation to the used definition (focal involvement,
extensive involvement or re-resection rate), in relation to adjustment for
differences in case mix, and in relation to random variation. Therefore we
evaluated a cohort of 762 patients in 9 hospitals in the Leiden region in
the Netherlands. At first sight, the tumour positive margin rates varied
widely between the 9 hospitals. Though, most variation could be explained
by random variation, as shown in the funnel plot (figure 1).(4) After
adjusting for differences in case mix, the variation of hospitals
decreased further and a hospital could be considered an outlier or not
depending on the used definition.

However, two outlier hospitals could still be identified, and the
question rose why these hospitals had an outlier status. Was it because
they performed better or worse than the others? Or was there a
registration bias? We observed large differences among the hospitals in
the interpretation of ‘tumour margin positivity’ reflected by
discrepancies in focal involvement rate, extensive involvement rate (when
a re-resection is indicated) and the re-resection rate. These
discrepancies indicated that hospitals did not use uniform definitions and
therefore a valid hospital comparison was undermined.

Quality indicators are usually not registered in a uniform way and
there is limited control if the registration is complete and reliable.
Therefore quality indicators bare the risk of registration and selection
bias, which limits the validity. When quality indicators are developed and
revised, the validity of the indicator should thus be considered as a
critical aspect.
Quality indicators are a usable tool for quality measurement and have
provided more transparency on the quality of care. However, there are more
restraints. A better performance on a single quality indicator does not
necessarily relate to better quality of care. This can induce a ‘perverse
incentive’: defensive behaviour to optimize the results on the indicator.
Secondly, quality indicators give only a proxy for quality and do not
provide information to improve the quality if care.

An alternative instrument for quality measurement is the clinical
audit, which does provide a complete overview of the quality of care. An
audit is a detailed registration of clinical data from different
healthcare providers, concerning all aspects of treatment: structure,
process and outcomes. Moreover, it contains baseline characteristics which
can be used for risk adjustment. Individual providers receive on a regular
basis the results of their own performance compared to the mean. This has
an intrinsic effect of improving quality (‘the Hawthorne effect’). In
addition to performance monitoring and feedback, audits have the
possibility to identify ‘best practices’. Identifying and adopting these
‘best practices’ might improve the overall quality of care.(5)

In several countries, including the US, the UK, the Nordic Countries
and the Netherlands, audits have been successfully implemented.(6) In the
Netherlands, participating in the Dutch Surgical Colorectal Audit is used
as a performance indicator by the health care inspectorate and the health
care transparency program. This can be an example for other countries.

The development and revision process of quality indicators is
expensive and time consuming, while the benefits of indicators are
limited. In our opinion, clinical audits are a more effective strategy to
provide both transparency on the quality of care, and generate tools for
quality improvement.

Figure 1 Funnel plot of the margin involvement rates and re-resection
rates of nine hospitals in the region of the comprehensive cancer centre
west in the Netherlands, between 1 July 2008 and 30 June 2009. At the x-
axe is the hospital volume for breast conserving surgery.
The results of
each hospital are presented in a vertical line in similar colours.

The blue line represents the consensus based standard of maximum 20%
margin involvement after breast conserving surgery for breast cancer. The
red and green lines represent the 95% and 99.8% confidence intervals
around the standard. A hospital outside the confidence limits, performs
significantly better or worse than the standard.

1. Reeves D, Doran T, Valderas JM, Kontopantelis E, Trueman P, Sutton
M, et al. How to identify when a performance indicator has run its course.
BMJ 2010;340:c1717.

2. Dimick JB. What makes a "good" quality indicator? Arch Surg
2010;145(3):295.

3. Gooiker GA, Veerbeek L, van der Geest LG, Stijnen T, Dekker JW, Nortier
JW, et al. [The quality indicator 'tumour positive surgical margin
following breast-conserving surgery' does not provide transparent insight
into care]. Ned Tijdschr Geneeskd 2010;154:A1142.

4. Mayer EK, Bottle A, Rao C, Darzi AW, Athanasiou T. Funnel plots and
their emerging application in surgery. Ann Surg 2009;249(3):376-83.

5. Hall BL, Hamilton BH, Richards K, Bilimoria KY, Cohen ME, Ko CY. Does
surgical quality improve in the American College of Surgeons National
Surgical Quality Improvement Program: an evaluation of all participating
hospitals. Ann Surg 2009;250(3):363-76.

6. van Gijn W, Wouters MW, Peeters KC, van de Velde CJ. Nationwide outcome
registrations to improve quality of care in rectal surgery. An initiative
of the European Society of Surgical Oncology. J Surg Oncol 2009;99(8):491-
6.

Competing interests:
None declared