Should we be wary of statistics in medicine?
I read with great interest and excitement the systematic review and
meta-analysis on prophylactic antibiotics for burns patients.
Looking at it closer, a long-standing question seems to emerge and it
is that of validity and generalisability, which is often encountered with
even well-conducted systematic reviews and meta-analyses
The inclusion of quasi-randomised controlled trials in this meta-
analysis does probably not remove anything from its value and a trade-off
between the trueness and the validity of findings is undoubtedly often
needed before conclusions are made.
Heterogeneity is probably not avoidable to a certain extent as
systematic reviews bring together studies that are diverse both clinically
and methodologically (1).
I still wonder if the question of heterogeneity is not inadequately
The Cochran’s Q value gives you only a dichotomous answer
(heterogeneous or not) but the I2 quantifies the effect of heterogeneity,
providing a measure of the degree of inconsistency in the studies’ results
Does the I2 value of 32% reflect a low, moderate or high
heterogeneity then? And how much is too much heterogeneity?
There is no hard rule to define the above but one generally accepted
rule of thumb characterizes an I2 of less than 25% as small heterogeneity,
25 to 50% as moderate, and more than 50% as large heterogeneity.
When the authors conclude that systemic prophylaxis is associated
with a significant reduction in all cause mortality without significant
heterogeneity, one could argue that the studies included are not
consistent enough to draw hard clinical conclusions.
The authors do recognise in their discussion that all types and
degree of burns were included and a subgroup analysis should have been
carried but unfortunately this was impossible because of paucity of
I wonder how generalisable their findings are on the general
population with the included studies being moderately heterogeneous.
(1) Higgins J, Thompson S, Deeks J, Altman D. Statistical
systematic reviews of clinical trials: a critical appraisal of guidelines
practice. J Health Serv Res Policy 2002;7:51-61.
(2) 4 Higgins JPT, Thompson SG. Quantifying heterogeneity in a meta-
analysis. Stat Med 2002;21:1539-58.
Competing interests: No competing interests