Comparison of three methods for estimating rates of adverse events and rates of preventable adverse events in acute care hospitals
BMJ 2004; 328 doi: https://doi.org/10.1136/bmj.328.7433.199 (Published 22 January 2004) Cite this as: BMJ 2004;328:199
All rapid responses
Rapid responses are electronic comments to the editor. They enable our users to debate issues raised in articles published on bmj.com. A rapid response is first posted online. If you need the URL (web address) of an individual response, simply click on the response headline and copy the URL from the browser window. A proportion of responses will, after editing, be published online and in the print journal as letters, which are indexed in PubMed. Rapid responses are not indexed in PubMed and they are not journal articles. The BMJ reserves the right to remove responses which are being wilfully misrepresented as published articles or when it is brought to our attention that a response spreads misinformation.
From March 2022, the word limit for rapid responses will be 600 words not including references and author details. We will no longer post responses that exceed this limit.
The word limit for letters selected from posted responses remains 300 words.
What do you think of linking methods for estimating rates of adverse
events and patients complaints? In Italy, the National Health Service does
not support an independent agency or patient representative. In Milan, we
carried out a survey to explore if an independent patient’s agency could
facilitate local learning and action to improve the quality of health
services from users’ perspectives.
We analysed 140 forms of complaints people reported to our voluntary
citizen’s association (Cittadinanza attiva). Every complaint enclosed one
or more clinical records. Our aim was to compare allegations contained in
the written complaints and the results of a retrospective case records
review.
The main causes of the complaint (according to the patients) were:
*A
delay in diagnosis and treatment.
*The failure or a complication in the
technical performance of an indicated operation or invasive procedure.
*Lack of care or attention, failure to attend, lack in monitoring of a
patient.
According to the physician record review, 51 out of 140 complaints were
associated with a preventable adverse event (mainly occurring in
hospital); 9 of them were serious and preventable.
We know that there are a number of potential shortcomings in our study.
Nevertheless, the survey show what kind of useful information about
substandard medical care we can gather also from this type of
documentation. The complaints handled by an independent agency will good
represent the views of the people in this field.
Roberto Natangelo
Retired Physician.
(roberto.natangelo@libero.it)
Cittadinanzaattiva (Citizenshipfoundation.),
Via Mecenate 25.,
20138 Milan (Italy).
Competing interests:
None declared
Competing interests: No competing interests
The cornerstone of any quality strategy is prevention. If preventable
adverse events can be reliably detected, improvement strategies can be
designed. Unfortunately, none of the methods tested in the study by Michel
et al 1 gives a definite answer to this issue. The prospective method
detects preventable adverse events more effectively in medicine. The
retrospective method is equally effective when surgical cases are added.
This warrants further comments.
The gold standard in assessing the preventability of an adverse event
was for a long time the so called ‘Bolam Test’: the judgement of a
responsible body of medical peers. The validity of the Bolam Principle
was questioned by the House of Lords in the Bolitho case 2. It resulted in
a major controversy without definite answer. There is no real consensus on
the definition of a preventable adverse event.
Secondly, judgement on the preventability of an adverse event changes
according to the kind of data that is collected. From related studies, it
appears that preventable adverse events are found more often in incident
reports than in chart review processes.3,4
Finally, information on adverse events is mostly qualitative.
Classifying such events for standardised comparison and reliably assessing
their preventability, represents the real challenge.
The excellent study by Michel et al 1 opens the door for validating
methods of adverse event measurement; prevention has still a long way to
go.
REFERENCES
1 Michel P, Quenon JL, de Sarasqueta AM, Scemama O. Comparison of
three methods for estimating rates of adverse events and rates of
preventable adverse events in acute care hospitals. Bmj 2004; 328: 199.
2 Samanta A, Samanta J. Legal standard of care: a shift from the
traditional Bolam test. Clin Med 2003; 3: 443-6.
3 Thomas EJ, Petersen LA. Measuring errors and adverse events in
health care. J Gen Intern Med 2003; 18: 61-7
4 O'Neil AC, Petersen LA, Cook EF, Bates DW, Lee TH, Brennan TA.
Physician reporting compared with medical-record review to identify
adverse medical events. Ann Intern Med 1993; 119: 370-6.
Competing interests:
None declared
Competing interests: No competing interests
This study provides very useful information on the methodology of
detecting and measuring adverse events. However, there are some
potentially important issues that might need more consideration.
The reported proportion of adverse events in this study is
significantly greater than many previous studies 1-4. This is partly
because the authors have used a combination of methods (review of medical
records and interviews with medical team) to detect adverse events. This
is similar to the capture-recapture technique, which is usually used when
there is no single reliable source of information.
The possibility of a Hawthorne effect should also be considered when
using the prospective method. In this project, clinicians knew the study
patients from the beginning of their admissions, so it was possible to
both treat the study patients and complete their medical records
differently, and this might change the rate of adverse events.
Adverse events can happen anytime between the admission, discharge
and post-discharge, (i.e. surgical wound infection). The prospective
method could miss those adverse events that might happen after the
interviews.
In patients who had two or more adverse events (i.e. adverse event A
and B), each prospective or retrospective method might detect adverse
event A but miss adverse event B and vice versa. It is possible that the
two methods detect similar number of patients with adverse events;
however, the number and type of adverse events detected by each method
could be different. Therefore, comparing the prospective and
retrospective methods based on the number and types of adverse events may
provide some potentially useful information as well.
Reference List
1. Brenan TA, Leap LL, Laird NM, Hebert L, Localio AR, and Lawthers
AG. Incidence of adverse events and negligence in hospitalised patients. N
Engl J Med 1991; 324, 370-376.
2. Gawande AA, Thomas EJ, Zinner MJ, and Brennan TA. The incidence
and nature of surgical adverse events in Colorado and Utah in 1992.
Surgery 1999; 126, 66-75.
3. Vincent Ch, Neale G, and Woloshynowych M. Adverse events in
British hospitals: preliminary retrospective record review. BMJ 2001;
322(7285), 517-519.
4. Wilson RM, Runciman WB, Gibberd RW, Harrison BT, Newby L, and
Hamilton JD. QAHCS_The Quality in Australian Health Care Study. Med J Aust
1995; 163, 458-471.
Competing interests:
None declared
Competing interests: No competing interests
Comparison of Paired Proportions
Many times in the field of medical/bio sciences the data are
collected meticulously with tremendous efforts but finally analyzed
hurriedly by wrong choice of method. One very often faced situation is
‘comparison of paired proportions’. It is necessary to make or understand
very clearly whether the data are from paired samples. Even when
observations are paired in some way, (ex. two proportions are measured on
the same individuals on two occasions say pre/post type or from
studies/trials in which a matched paired design has been used) comparison
of two proportions is very often done by two independent samples ‘normal
test’. It is not to claim that anything suggested below is new (except
simplification of formulae), the purpose is just to highlight this fact in
view of its importance in ‘epidemiology’ and medical education. The
material covered and the solutions offered are, to a large extent, known
in the literature.
Though a case of comparison of two proportions is discussed here,
when there are more than two related/paired proportions, appropriate test
for comparison is Chochran’s ‘Q’ test rather than simple, usual Chi-Square
test. That test is a sort of extension of McNemar test details of which
can be found in literature1. Published application of ‘Q test’ can also be
seen in literature2. ‘Normal test’ specifically applicable to compare two
paired proportions is also described in details in earlier quoted book1.
Analyzing data from crossover design with binary response and exact test
on matched pair proportions based on Binomial distribution is discussed
with examples in another exclusive text book on Medical Biostatistics3.
Case of two paired proportions:
The method is described here with the help of one numerical example. In a
study of ‘nutritional supplement’ efficacy, 300 children were examined
before start of the supplement and also at the end of the study. About 50%
children found to be under-nourished at baseline. This proportion came
down to 43.33% at the end. The data can be displayed in a 2×2 table as
follows :
For this example, r=150, s=20, t=0, and u=130. It would be incorrect
to arrange the data as in (b) part of the table and to apply the standard
chi-squared test, (which is same as two independent samples ‘normal test’)
as this would take no account of the paired nature of the data, namely
that it was the same 300 subjects examined twice and not 600 different
ones.
Then p1=0.5000, p2=0.5667, the difference is 0.0667, together with
its standard error, can be estimated from the numbers of discordant pairs,
s & t, and the total number of pairs, n. Difference between paired
proportions = [(s - t) / n] and approximate estimated Standard Error =
1/n * Ö{s+t-[(s - t)2 / n]}. Then test statistic Z is the ratio of
difference and SE which cannot be estimated unless r, s, t, & u are
known (not from p1, p2, and n alone). Nevertheless, it is possible to
estimate others if any one of r, s, t, u is known along with p1, p2, and
n. Lack of this information often limits the expected practice of making
“paired-unpaired” differentiation while dealing with proportions. For our
dataset SE= 0.0144 and Z=4.653, is highly significant. If you look at the
other-side i.e. if you consider proportions of “under-nourished” instead
of “normal”, the end result will remain same except that Z will bear
negative sign.
“t” can assumed to be a ‘structural’ zero reasonably since we expect
change only in one direction (however, ‘t’ cannot be assumed as structural
zero if we expect change in both the directions), then Standard Error =
1/n *Ö{ s-(s2/ n)} becomes approximately equal to Ö{(p2 - p1) / n} taking
s = (p2- p1)*n. For our example it is 0.0149 which is slightly more than
the actual. Note that ‘t’ zero (frequency of that cell being zero) is not
same as ‘structural zero’. McNemar's chi-squared test (c2paired) which is
based on the numbers of discordant pairs, s and t, is same (apart from
rounding error) as the square of the ‘Z’ value indicating that the two
tests are mathematically equivalent. It may be noted that two independent
samples ‘normal test’ or usual Chi-square test yields non-significant
result with these data (c2 = 2.6785 and Z =1.6429).
Another best way is to use the computer software accompanying an
excellent book by Altman et al.4 called CIA, to estimate 95% Confidence
Interval for difference (by recommended method due to Wilson for ‘paired’
samples) and check whether this CI includes zero as inclusion of ‘0’
indicates ‘non-significance’ (or exclusion of ‘0’ in CI indicating
significance) at 5% level of significance. For these data CI is 0.0377 to
0.0953. Another method (called traditional) in CIA also yields exact SE
from which Z can be estimated. This will help overcome another limitation
for making “paired-unpaired” differentiation while dealing with
proportions namely unavailability of computer software. However, while
using CIA, the set-up has to be correct (as shown in part ‘a’ of above
table). For data as in part ‘b’ of above table (incorrect layout) CI by
Newcombe’s method in CIA is from -0.0131 to 0.1452 which includes zero.
Sample size required for such studies is also discussed with examples in
literature5.
References
1. Sarmukaddam SB. Fundamentals of Biostatistics. New Delhi: Jaypee
Brothers Medical Publishers Ltd.; 2006 (pages 117 & 109).
2. Parlikar V., Sarmukaddam S., Agashe M., and Weiss M. Diagnostic
concordance of neurasthenia spectrum disorders in Pune, India. Soc
Psychiatry Psychiatr. Epidemiol. 2007; 42: 561-572.
3. Indrayan A. and Sarmukaddam SB. MedicalBiostatistics. New York: Marcel
Dekker, Inc.; 2001 (page 353).
4. Altman, DG., Machin, D., Bryant, TN., and Gardner, MS. Statistics with
confidence. London: BMJ Books; 2000 (page 50).
5. Sarmukaddam SB. and Kharshikar AV. Sample Size for Pre-Measure and Post
-Measure Panel Type Prevalence Study, In : Statistics in Health and
Nutrition, Hyderabad: National Institute of Nutrition 1990, pp 402-406.
Competing interests:
None declared
Competing interests: Table : Comparison of two paired proportions ¾ same 300 subjects were examined at both times(a) Correct layout (b) Incorrect layoutPre stage Normal Under-nourished Total Result Pre stage Post stage TotalPost stage Normal 150(=r) 20(=s) 170 Normal 150 170 260Under-nourished 0(=t) 130(=u) 130 Under-nourished 150 130 340Total 150 150 300 Total 300 300 600