Hospital tables “should prompt authorities to investigate”
BMJ 2001; 322 doi: https://doi.org/10.1136/bmj.322.7279.127 (Published 20 January 2001) Cite this as: BMJ 2001;322:127
All rapid responses
Rapid responses are electronic comments to the editor. They enable our users to debate issues raised in articles published on bmj.com. A rapid response is first posted online. If you need the URL (web address) of an individual response, simply click on the response headline and copy the URL from the browser window. A proportion of responses will, after editing, be published online and in the print journal as letters, which are indexed in PubMed. Rapid responses are not indexed in PubMed and they are not journal articles. The BMJ reserves the right to remove responses which are being wilfully misrepresented as published articles or when it is brought to our attention that a response spreads misinformation.
From March 2022, the word limit for rapid responses will be 600 words not including references and author details. We will no longer post responses that exceed this limit.
The word limit for letters selected from posted responses remains 300 words.
The analysis by Dr Foster Ltd, published in the Sunday Times amounts
to little more than seriously flawed research. Until a proper critique of
it has taken place I propose that no NHS organisation rushes to take
action that might turn out to be as misdirected as it was unnecessary.
I would like to raise the following questions
1. Was the proposal to conduct this study properly evaluated at the
design stage? Ideally a fully worked up protocol should have been
subjected to external review and been scrutinised by a truly independent
ethics committee. As a chair of such a committee I don't think the in-
house committee that we are told considered this study would stand up to
outside scrutiny. The data that underpins the study is confidential
clinical information and its use requires proper and accountable
oversight. The method of dissemination of the analysis raises further
questions about the use of the data.
2. Are SHMRs valid measure of quality of care with mortality as the
end point of interest?. SMRs were originally public health measures
intended to apply to whole area populations that are relatively static.
Hospitals don't have such a logical population. Admissions to a hospital
do not constitute a predefined population, they are arbitrary and depend
heavily on admissions policy, availability of support and other community
services locally and propensity of surgeons to operate on high risk cases.
SHMRs cannot be used to compare hospitals, one with the other (1,2).
Therefore league tables based on SHMRs are flawed in concept. This might
have been picked up by an independent review of the study at the protocol
stage.
3. Who benefits from the study? So far as I can tell no one benefits.
In fact the whole exercise is supremely unhelpful. It doesn't tell
managers and clinical leaders what to do in order to improve quality.
Should they look for rogue surgeons or killer nurses or systemic failings
in clinical care that could have been prevented. If we are all agreed that
it is in fact the last of this list that we ought to be working on, then I
have to ask, 'What's new?'. We were doing that anyway with clinical
governance and Organisation with a memory'. All that the Dr Foster study
has done is
a)divert managers in 'bad' hospitals into answering hysterical queries
from the press,
b)induce a sense of self-righteous complacency in so called 'good'
hospitals, and
c) encourage lawyers to chase after every death, expected or otherwise, to
look for what they can make from the so called bad hospitals trusts. That
will tie up the doctors and managers in these trusts, diverting them even
further from the things they ought to be doing.
Nor does it tell future patients anything even remotely useful. How
does knowing that a hospital's mortality index is say 120 help? That is a
crude and arguably imprecise estimate of the a priori average risk of
dying while in hospital. Who does it apply to? It applies only to the
statistically 'average' patient - a nice esoteric concept for the risk
modeling enthusiast but of no help the individual patient. The only effect
of this information on patients is to make them worry. What the patient
really needs is an estimate of his individual chances of a successful
outcome and he and his surgeon or physician need then to decide where the
balance of benefit lies. This process, also known as good clinical
medicine, has only been hampered by the Dr Foster analysis.
4. I can only characterise the whole exercise as the epidemiological
equivalent of shouting 'Fire!' in a crowded theatre. There people looking
either for a better view or a comfortable seat are diverted from both
pursuits by a mischief maker. Here patients and clinicians struggling for
a decent outcome to hospital care or at least for a half decent experience
are similarly distracted.
5. At least, is the approach logical? Can the analysis be improved?
My answers, again, are no and no. The reason we got into this mess is
because Dr Foster Ltd inverted the process of logical analysis. This is
yet another example of a project which starts from data that happened to
be there, goes on to refined and sophisticated (and therefore seductively
persuasive) analysis , suggests a few answers (If you torture a data set
enough it will confess to whatever you want), and then the researchers
ask: 'Now what possible question is this the answer to?'. Dr Foster says
that the question being answered is 'Which hospital's have poor quality of
care as measured by mortality?' Wrong, it doesn't answer this question.
Ideally we should start with the question, refine it as far as
possible, determine what data we need in order to answer it with a given
and acceptable degree of validity, collect the relevant data and then
analyse it.
By inverting this rational approach, the Dr Foster Ltd / Sunday Times
report is of little or no value, it has done nobody any good and it may
even have done a great deal of harm. Clinicians and patients alike should
be appalled, as I am, that it has allowed some people to make money from
the new sport of NHS-bashing.
References
1. Howell J. Standardised mortality ratios. Lancet 1995;346:904 [letter]
2. Court BV, Cheng KK. Pros and cons of standardised mortality ratios.
Lancet 1995;346:1432
Competing interests: No competing interests
Sir,
"In the year 1854 a terrible recrudescence of cholera, due, as was
supposed, to the contamination of the water furnished by the notorious
Broad Street pump, in the parish of St James's, Westminster, ravaged the
Metropolis and particularly the immediate neighbourhood of the Hospital
(Golden Square). Twenty-two years before, cholera had sprung suddenly upon
a profession utterly unprepared to deal with it and destitute of a
principle to guide them in organising the best defence against the new
foe. In 1849 it was found that their experience had not taught them much.
In 1854 they had still to search among their record of cases for any
agreement as to the best way out of their perplexities. Mean-while the
homœopathic section of the profession, relying upon the principle which
directs them to seek for medicines capable of producing physiological
phenomena similar to those exhibited by the disease, had never been in
doubt. Hahnemann, on receiving a detailed description of the disease in
its various stages from a disciple who sought his guidance, had, without
seeing a case, but relying solely on his law of drug-selection, prescribed
a course of remedies which, alike in 1832, 1849, and 1854, proved pre-
eminently successful, and which to the present day constitute the
treatment mainly relied upon by homœopathic practitioners. Thus fortified
the homœopaths did not shrink from the issue offered by the cholera
outbreak of 1854. The whole of the wards of the London Homœopathic
Hospital were devoted to the treatment of the epidemic, and 64 cases of
cholera and 331 of choleraic and simple diarrhœa were treated.
Of the 61 cases of cholera treated, 10 died, a percentage of 16.4; of
the 331 cases of choleraic and simple diarrhœa treated, 1 died. The
neighbouring Middlesex Hospital received 231 cases of cholera and 47 cases
of choleraic diarrhœa. Of the cholera patients treated 123 died, a
fatality rate of 53.2 per cent., among the victims being one of the
nurses.
Dr Macloughlin, one of the medical inspectors appointed by the
General Board of Health, visited the wards, examined the cases under
treatment, and watched their progress. His statement, addressed to Mr.
Hugh Cameron, a member of the medical staff, was as follows
"You are aware that I went to your hospital prepossessed against the
homœopathic system, that you had in me in your camp an enemy rather than a
friend... and I need not tell you that I have taken some pains to make
myself acquainted with the rise, progress and medical treatment of
cholera, and that I claim for myself some right to be able to recognise
the disease, and to know something of what the medical treatment ought to
be, and that there may, therefore, be no misapprehension about the cases
saw in your hospital, I will add that, all I saw were true cases of
cholera, in the various stages of the disease, and that I saw several
cases which did well under your treatment which I have no hesitation in
saying would have sunk under other. In conclusion I must repeat to you
what I have already told you, and what I have told everyone whom I have
conversed, that although in allopath by principle, education and practice
yet were it the will of Providence to afflict me with cholera, and deprive
me of the power of prescribing for myself, I would rather be in the hands
of a homœopathic than an allopathic adviser."
Dr. Macloughlin, as shown by his researches and publications, was
undoubtedly well informed as to the nature of cholera.
Now, a circular was addressed by the President of the Board of Health
to various Metropolitan hospitals and to qualified practitioners,
requesting returns of cholera cases, with details of the circumstances,
treatment and results. The object was to determine by comparison, for the
public good, what treatment experience showed to be the best for the new
plague. Returns were sent in from the London Homœopathic Hospital, giving
the names and addresses of the patients treated, the symptoms, remedies,
and result in each case, and a summary of those results. This was not a
question of theory, or of any particular school; it was a question of
facts and statistics affecting the public health. But the report of the
Board of Health was presented to Parliament without the slightest
reference to the London Homœopathic Hospital or to the brilliant results
which its physicians had achieved by undaunted self-sacrifice in a time of
great public calamity. Complaint was, of course, made to the Board of
Health and duly referred to its Medical Committee, with the result that
the Board received from the committee a resolution, which, for ingenuity
of disingenuousness and illiberality, can hardly ever have been equalled.
It was this :
"That by introducing the returns of homœopathic practitioners they
(the Treatment Committee) would not only compromise the value and utility
of their averages of Cure, as deduced from the operation of known
remedies, but they would give an unjustifiable sanction to an empirical
practice, alike opposed to the maintenance of truth and to the progress of
science."
In the first place, the remedies "unknown" to the Treatment Committee
were such as Camphor, Copper, Hellebore, Arsenic, and other drugs well
known to medicine. In the second place, it was their bounden duty to
"compromise" the averages of old methods by more successful new methods in
their search for the best results. Thirdly, the interference with
empirical practice was no part of the statistical duty before them. And
lastly, the "progress of science" was de facto obstructed by their refusal
to "compromise" their averages by a factor which contained the very object
of their search. The perversity was too plain, and Lord Robert Grosvenor
(afterwards Lord Ebury) moved on May 17, 1855, in the House of Commons for
"Copies of Letters addressed to the General Board of Health complaining of
the omission of any notice of certain returns in relation to the treatment
of cholera and correspondence between the President of the Board and the
Medical Council, with copies of the returns which have been rejected by
the Medical Council." The House of Commons, which was more anxious for the
"progress of science" and the "value and utility of averages" than for
"the operation of known remedies," to say nothing of its great duty to the
people it represented, forthwith ordered a special return of the ignored
homœopathic statistics, which was in due course made by the Board of
Health, and these returns were ordered by the House to be printed on May
21, 1855. They remain among Parliamentary Papers to this day, a standing
monument alike of the success of the new policy and of the obscurantism of
the old."
[from "Sixty-five years work, An Historical Sketch of the London
Homeopathic Hospital", 1914]
Competing interests: No competing interests
Your report on the hospital "league tables (BMJ, 20th January, p.127)
states "The biggest predictor of death rates was the number of doctors in
the hospital..." - a conclusion made by Professor Brian Jarman, and (in
the Sunday Times) underpinned by data from Greenwich District Hospital.
Unfortunately for the conclusion, the data are wrong. The Department
of Health's figures for Greenwich appear to be for consultants only, and
do not include junior doctors. This introduces an error of an order of
magnitude between 2 and 3 fold. One may ask why the trainee figures are
missing; but all hospital's figures are likely to be distorted further by
the fact that staff in unrecognised posts (often posts with strange titles
such as "Trust doctors") are not counted at all.
When such a fundamental and massive data error passes unchecked and
results in false deductions it must cast doubt on the whole process. We
cannot blame Dr Foster Ltd, who issued a disclaimer on data accuracy in
the small print, but it is in my view quite wrong of the Department of
Health to allow publication without looking closely at figures that
departed significantly from the mean.
Garbage in, garbage out. Pity really, because the idea is not bad.
Competing interests: No competing interests
Good Hospital Guide: Triumph of rhetoric over reason
Good Hospital Guide: Triumph of rhetoric over reason.
I am amazed by the near hysterical reaction of both political leaders
and
the media to the publication of hospital mortality figures in Britain
(1).
Similar figures were published in the United States 15 years ago and
initially caused a similar reaction (2). The annual release of these
adjusted
figures in the States soon became an annual "non-event". The hope that
these figures would be useful in identifying hospitals with poor
performance soon faded away. The methodology of analysing the figures
did not have a scientific basis. In fact, there has not been a single
study
that has convincingly linked variations in hospital mortality rates to
underlying difference in quality of care (3, 4).
Dr. Foster Ltd. has used the large administrative databases that were
built up over the last 10 years to create a league table of the clinical
performance of all hospitals in Britain. The data were never collected for
that purpose. It does not matter how the private company ( Dr. Foster )
massages the data, it will never reflect the clinical performance of the
hospitals. This methodology has no scientific basis – only political and
commercial one. No wonder that Dr. Foster Ltd. chose to present this
pseudoscience in the popular press.
The hospital league table has been a meaningless exercise and waste
of the taxpayer’s money. This money could have been used in hospitals to
collect clinically validated data reflecting quality of care as suggested
by the Federation of Surgical Specialty Associations last month (5).
Yours faithfully,
Audun Sigurdsson
Consultant General Surgeon
Walsall Manor Hospital,
37 Buchanan Avenue,
Walsall WS4 2ER
LapSurg@Compuserve .Com
(1) The Sunday Times, Jan 14, 2001, p 1.
(2) New York Times, Mar 12, 1986, p1.
(3) Knaus WA, Wagner DP. Interpretation of Hospital Mortality Rates : The
Current State of the art. Mayo Clin Proc 65 :1627 – 1629,1990.
(4) Park RE, Brook RH,Koesecoff J,Rubenstein L et al. Explaining
variations in hospital death rates: randomness, severity of illness,
quality of care. JAMA 264:484 – 490, 1990.
(5) Response to the Prime Minister, following the publication of the NHS
Plan. The Federation of Surgical Associations at the Association of
Surgeons of Great Britain and Ireland
Competing interests: No competing interests