Plea for caution in the interpretation of single-centre quality improvement reports.
Sir,
We question the cause and effect inferences in the above report [1]
for the following reasons:
1. This is only one report from a large population of potential
reports and hence should be considered at high-risk of publication bias.
Negative single-centre quality improvement reports are notable by their
absence.
2. The study was uncontrolled – controlled multi-centre studies
provide much more modest results.[2;3] A controlled before and after multi
-centre evaluation of an intervention that included IHI bundles will be
reported in due course.
3. The intervention effect is implausible for two related reasons:
a. The size of the observed effect greatly exceeds the total
preventable death rate of 0.3%, which can be calculated from the figures
cited by the authors in the introduction.
b. It seems implausible for an intervention so simple that it could
be introduced across a whole hospital within one month, to have had such a
massive effect.[4]
4. Reported death rates for certain diseases started to improve in
January 2007 (well before the intervention date in April 2007), suggesting
that other (confounding) factors were at work.
5. The topics were chosen for intervention on the grounds that they
had “led to the largest number of deaths” in the previous year and the
findings may therefore have been subject to regression to the mean.
6. We do not accept that differences in use of bundles by the
hospitals within the trust generates valid internal controls for the
following related reasons:
a. This was not an intention-to-treat analysis;
b. There are many potential confounding differences between the
hospitals;
c. The method by which use of the bundles was measured was not itself
audited and is likely to have been subject to measurement error.
There is a fashion for giving credence to single-centre studies that
would not normally be regarded as likely to be valid, provided they can be
described as ‘quality improvement’.[5] Yet the risks of bias are not
contingent on how a study is labelled.
References:
[1] Robb E, Jarman B, Suntharalingam G, Higgens C, Tennant R, Elcock
K. Using care bundles to reduce in-hospital mortality: quantitative
survey. BMJ 2010;340:c1234.
[2] Landon BE, Hicks LS, O'Malley AJ, Lieu TA, Keegan T, McNeil BJ et
al. Improving the Management of Chronic Disease at Community Health
Centers. N Engl J Med. 2007;356(9):921-34.
[3] Landon BE, WIlson IB, McInnes K, Landrum MB, Hirschhorn L,
Marsden PV et al. Effects of a Quality Improvement Collaborative on the
Outcome of Care of Patients with HIV Infection: The EQHIV Study. Ann
Intern Med. 2004;140(11):887-96.
[4] Bosk CL, Dixon-Woods M, Goeschel CA, Pronovost PJ. Reality check
for checklists. Lancet. 2009;374:444-5.
[5] Berwick DM. The Science of Improvement. JAMA. 2008;299(10):1182-
4.
Competing interests:
RL is senior author on controlled before and after multi-centre evaluation mentioned in response.
Rapid Response:
Plea for caution in the interpretation of single-centre quality improvement reports.
Sir,
We question the cause and effect inferences in the above report [1]
for the following reasons:
1. This is only one report from a large population of potential
reports and hence should be considered at high-risk of publication bias.
Negative single-centre quality improvement reports are notable by their
absence.
2. The study was uncontrolled – controlled multi-centre studies
provide much more modest results.[2;3] A controlled before and after multi
-centre evaluation of an intervention that included IHI bundles will be
reported in due course.
3. The intervention effect is implausible for two related reasons:
a. The size of the observed effect greatly exceeds the total
preventable death rate of 0.3%, which can be calculated from the figures
cited by the authors in the introduction.
b. It seems implausible for an intervention so simple that it could
be introduced across a whole hospital within one month, to have had such a
massive effect.[4]
4. Reported death rates for certain diseases started to improve in
January 2007 (well before the intervention date in April 2007), suggesting
that other (confounding) factors were at work.
5. The topics were chosen for intervention on the grounds that they
had “led to the largest number of deaths” in the previous year and the
findings may therefore have been subject to regression to the mean.
6. We do not accept that differences in use of bundles by the
hospitals within the trust generates valid internal controls for the
following related reasons:
a. This was not an intention-to-treat analysis;
b. There are many potential confounding differences between the
hospitals;
c. The method by which use of the bundles was measured was not itself
audited and is likely to have been subject to measurement error.
There is a fashion for giving credence to single-centre studies that
would not normally be regarded as likely to be valid, provided they can be
described as ‘quality improvement’.[5] Yet the risks of bias are not
contingent on how a study is labelled.
References:
[1] Robb E, Jarman B, Suntharalingam G, Higgens C, Tennant R, Elcock
K. Using care bundles to reduce in-hospital mortality: quantitative
survey. BMJ 2010;340:c1234.
[2] Landon BE, Hicks LS, O'Malley AJ, Lieu TA, Keegan T, McNeil BJ et
al. Improving the Management of Chronic Disease at Community Health
Centers. N Engl J Med. 2007;356(9):921-34.
[3] Landon BE, WIlson IB, McInnes K, Landrum MB, Hirschhorn L,
Marsden PV et al. Effects of a Quality Improvement Collaborative on the
Outcome of Care of Patients with HIV Infection: The EQHIV Study. Ann
Intern Med. 2004;140(11):887-96.
[4] Bosk CL, Dixon-Woods M, Goeschel CA, Pronovost PJ. Reality check
for checklists. Lancet. 2009;374:444-5.
[5] Berwick DM. The Science of Improvement. JAMA. 2008;299(10):1182-
4.
Competing interests:
RL is senior author on controlled before and after multi-centre evaluation mentioned in response.
Competing interests: No competing interests