Targets, inspections, and transparencyBMJ 2004; 328 doi: https://doi.org/10.1136/bmj.328.7440.598 (Published 11 March 2004) Cite this as: BMJ 2004;328:598
- Gwyn Bevan, professor of management science (, )
- Christopher Hood, Gladstone professor of government
- Department of Operational Research, London School of Economic and Political Science, London WC2A 2AE
- All Souls College, University of Oxford, Oxford OX1 4AL
For the past three years, the performance of NHS organisations in England has been assessed by performance (star) ratings based on targets—“key targets” and indicators in a “balanced scorecard.”1–4 In the first two years, NHS managers knew only (until about a week before publication) that ratings would reflect stated government priorities. In response to demands for greater transparency the Commission for Health Improvement, when it took over responsibility for the third set of ratings from the Department of Health, published lists of the targets to be used in advance of publication of ratings but during the year to which the targets applied. Demands have been made for greater transparency in the NHS—for example, by publishing full specifications of targets and how ratings are to be calculated before the start of each financial year. Such demands are similar to those for every speed camera to be marked with information revealing when each camera is in operation and at what threshold of speed it is triggered. So how much transparency is enough and when does transparency need to be traded off against effective control?
As with speed cameras, complete transparency in setting targets leads to problems in securing effective control. The most vivid, if apocryphal, examples are from the Soviet Union, one being of the nail factory that made only enormous nails because its target was set in tonnes. Recently, the Public Administration Select Committee found many examples of inaccuracies in data used for targets and gaming in response to targets across government.5 In the NHS problems have been noticed with targets for waiting times, patients in emergency departments, and for ambulance response times. 5–9
But the Public Administration Select Committee concluded that these difficulties do not mean that targets should now be scrapped.4 Striking improvements in measured performance that have accompanied the use of targets are unlikely to be wholly attributable to problems with data or gaming. Moreover, targets have been introduced because alternative forms of control have been tried and found wanting. As Winston Churchill said of democracy, regulating performance through targets is the worst control system ever devised—except for all the others. What is needed are ways of limiting gaming. And one way of doing so is to introduce more randomness in the assessment of performance, at the expense of transparency.
Combining randomness with inspection systems to counter gaming was advocated by Jeremy Bentham nearly 200 years ago.10 Inspectors of care homes are statutorily obliged to make unannounced visits today. The Commission for Health Improvement's successor, the Commission for Health Care Audit and Inspection, has proposed making random unannounced visits in its vision of inspecting the NHS in England.11 Other ways of introducing unpredictability into regulation are to make it hard for managers to ascertain in advance who will assess them, and, as we advocate here, to introduce some uncertainty into targets and rating systems.
The analogy is with the use of unseen examinations, where the unpredictability of what the questions will be means that it is safest for students to cover the syllabus. This is similar to the first two years of NHS performance ratings, when managers knew priorities and key targets but not exactly how their performance would be assessed. A variant on that approach would be to make the relative weights of targets unpredictable, so that the manager of the Soviet nail factory would not know whether performance would be assessed in terms of tonnage or numbers of nails. Targeting NHS waiting times is an obvious application of this principle.
Of course we do not advocate total reliance on randomised controls. Announcing dates of assessments in advance can have the beneficial effect of making organisations look carefully at themselves, and that benefit would be lost if all inspections were randomly timed.12 Moreover, to make managers' worlds too unpredictable in the object of assessment is a recipe for producing fatalistic, lottery playing responses rather than the strategy and direction good managers provide. So our argument is not for total reliance on randomised controls, but for more randomness as part of a broader mix. That is because too much predictability in the name of transparency weakens control by the gaming responses it invites.
Competing interests GB was director of the office for information on healthcare performance at the Commission for Health Improvement until September 2003.