Adab et al's suggestion that league tables be abandoned adds
additional voices to this rising and welcome chorus. Their recommendation
that control charts be used instead, however, raises a question that has
long bothered me, for which I hope the authors (or the BMJ's readers)have
the answer.
Control limits are set using an estimate of variability of the
process in question, typically some suitable multiple of the standard
deviation. The multiple chosen is associated with a probability using the
relevant statistical distribution. For example, 3 sigma limits imply that
the probability of a point appearing between the upper and the lower
limits is 0.997, assuming the normal distribution. However, that
probability is for a single, prespecified point. The probability that at
all of (say) 37 independent points would appear within the control limits
is only 0.905, even though all the points have the same individual
probability. In fact, the probability that 2 or more points are outside
the limits (0.005) is greater that of any individual point (0.003).
Attributing special cause variation to these points seems like an error.
Is there a method to take the multiplicity of comparisons into
account when creating and using control charts?
Rapid Response:
Performance charts and multiple comparisons
Adab et al's suggestion that league tables be abandoned adds
additional voices to this rising and welcome chorus. Their recommendation
that control charts be used instead, however, raises a question that has
long bothered me, for which I hope the authors (or the BMJ's readers)have
the answer.
Control limits are set using an estimate of variability of the
process in question, typically some suitable multiple of the standard
deviation. The multiple chosen is associated with a probability using the
relevant statistical distribution. For example, 3 sigma limits imply that
the probability of a point appearing between the upper and the lower
limits is 0.997, assuming the normal distribution. However, that
probability is for a single, prespecified point. The probability that at
all of (say) 37 independent points would appear within the control limits
is only 0.905, even though all the points have the same individual
probability. In fact, the probability that 2 or more points are outside
the limits (0.005) is greater that of any individual point (0.003).
Attributing special cause variation to these points seems like an error.
Is there a method to take the multiplicity of comparisons into
account when creating and using control charts?
Competing interests: No competing interests