Intended for healthcare professionals

Rapid response to:

Information In Practice

Performance league tables: the NHS deserves better

BMJ 2002; 324 doi: https://doi.org/10.1136/bmj.324.7329.95 (Published 12 January 2002) Cite this as: BMJ 2002;324:95

Rapid Response:

Let's not write off league tables yet

Performance management, including the use of indicators, is an agenda
that is seemingly here to stay. It has to be a multidisciplinary endeavour
and therefore the involvement of clinical professionals, such as public
health in this case, is crucial.

The article is stimulating and does highlight many of the undoubted
problems of league tables and control charts do seem to be a useful
addition to the possible tools and techniques of performance management.
However the case may not be as strong as is suggested.

As the authors state, “some arguments against their (league tables)
use can be used against any monitoring or assessment system.” What they
don’t go onto to fully explore is the extent to which these arguments
apply to control charts, after all statistical process control would
appear to be a monitoring system itself.

Below I want to consider some of the problems of league tables that
they highlight and whether they may also apply to control charts.

Quality of data – this still applies to control charts, they use the
same data. (However, often one of the best ways of improving data quality
is to publish in some form.)

Not all valued outcomes are measurable – again applies to control
charts e.g. no of deaths is used in the example for both, leaving open the
issue of the many other possible outcomes that could be included under
either scheme. (The authors refer to the 1990 article by Smith and the
conclusion that most NHS indicators are chosen “on the basis of what is
available and practical rather than what is meaningful.” Whilst it is
always crucial to bear this in mind, many developments have taken place
since 1990 in this field. There has been a considerable amount of work
done by professionals within the NHS itself and by the DoH/NHSE e.g. the
NHS Performance Assessment Framework, the accompanying indicators and the
national consultation on this last year for example.)

Unintended consequences e.g. gaming – could still apply as control
charts emphasise an outcome.

Lack of use by regulators – more could be done here. However, for
example, the NHSE (through Regional Offices) do question
providers/purchasers on specific indicators, CHI do review them for their
clinical governance reviews, and the next set of NHS PIs is supposed to be
accompanied by a commentary from CHI and the Audit Commission. There is
also no reason to assume that the situation would be significantly
different with control charts except if they are accepted as being a
better way of doing things then maybe their use would be more widespread.

Someone must come bottom – someone may still appear as an outlier in
a control chart. It is also entirely possible to look at rankings over
time – league tables haven’t done this yet, perhaps because they have only
been around in their current form for a couple of years. You could also
look at sets of indicators, a balanced scorecard approach, to help
overcome this issue for a particular indicator.

Poor presentation – the league table chart as presented is probably
less clear than the control chart. However alternative presentations of
the league table are possible – for instance the overall figure could be
presented as a line or band instead of a bar, allowing an easier
comparison with individual units.

Control charts
The concepts of “noise” and “signals” are very helpful but need their use
be restricted to control charts? Isn’t an equivalent the use of casemix
adjustment with performance indicators? They emphasise the need to
identify controllable and uncontrollable elements of performance as
appropriate for the unit you are looking at – be it the national system, a
provider or a team within a service.

Difference between control charts and league tables.
League tables are said to assume there are performance differences between
providers and that control charts assume all providers are part of a
single system. The case for providers being in a system highlights certain
features and so suggests “large acute trusts” may be considered a system.
A different case could be constructed if we accept that cultures, for
instance, are very important in determining provider performance – as the
clinical governance literature suggest, and that we shouldn't treat all
providers or a group as part of a system. Or maybe it is just that we
need to consider how much of a performance variation could be attributed
to "common cause" and how much to "special cause" - and need this be the
same for all systems? (The use of “system” is problematic in that the NHS
might be a system but a unit could also be seen as a system - so what is
"common" may easily be very different depending how what you take as your
system.)

Surely there is a need to consider both performance differences
between units and the need to increase overall performance. Control charts
have a role to play but so probably do league tables – and in both cases
there is still work to be done in their construction and use.

Andrew Barraclough

Competing interests: No competing interests

14 January 2002
Andrew J Barraclough
Performance Manager
Nottingham HA (NG1 6GN)