Meta-analyses: what is heterogeneity?BMJ 2015; 350 doi: https://doi.org/10.1136/bmj.h1435 (Published 16 March 2015) Cite this as: BMJ 2015;350:h1435
All rapid responses
Some statements should be modified, to be consistent with the literature of statistics and meta-analysis. Also, the random-effects meta-analysis underlying the example is prone to biases.
The phrase "Cochran's Q test" is not accurate. "Cochran's Q" is all right, but "test" is not. Cochran (1) did not propose the popular test, which refers Q to the chi-squared distribution on k-1 degrees of freedom (k is the number of trials).
In finite samples the null distribution of Q is not chi-squared on k-1 degrees of freedom (2). Complicating matters, the actual null distribution varies with the measure of effect (risk difference, risk ratio, odds ratio, standardized mean difference, etc.). Using the incorrect null distribution may be part of the reason that "Cochran's Q test … often fails to detect heterogeneity in the sample estimates." This issue has not yet been well studied.
I^2 is related to Q by the formula I^2 = [Q - (k-1)]/Q (or 0 if the right-hand side is negative). Under the chi-squared distribution usually assumed as the null distribution, Q has a substantial probability of exceeding k-1. Thus, in the absence of heterogeneity, I^2 has a substantial probability of being positive. This is a problem for the interpretation of I^2 as "the proportion of variation between the sample estimates that is due to heterogeneity rather than to sampling error."
Since I^2 is a function of Q, it does not provide "an additional test of heterogeneity." Except for the complication mentioned in the preceding paragraph, however, I^2 is easier to interpret than Q.
For random-effects meta-analysis, Gates et al. (3) used the popular DerSimonian-Laird procedure (DL) (4). That choice is unfortunate, because DL systematically underestimates the heterogeneity variance (tau^2) and may produce biased estimates of the overall effect. (5) One can avoid the shortcomings of DL by analyzing the binomial data from the studies, rather than basing the meta-analysis on the log-risk-ratios and their estimated variances.
1. Cochran WG. The combination of estimates from different experiments. Biometrics 1954;10:101-29.
2. Kulinskaya E, Dollinger MB, Bjorkestol K. On the moments of Cochran's Q statistic under the null hypothesis, with application to the meta-analysis of risk difference. Research Synthesis Methods 2011;2:254-70.
3. Gates S, Lamb SE, Fisher JD, et al. Multifactorial assessment and targeted intervention for preventing falls and injuries among older people in community and emergency care settings: systematic review and meta-analysis. BMJ 2008;336:130.
4. DerSimonian R, Laird N. Meta-analysis in clinical trials. Controlled Clinical Trials 1986;7:177-88.
5. Hamza TH, van Houwelingen HC, Stijnen T. The binomial distribution of meta-analysis was preferred to model within-study variability. J Clin Epidemiol 2008;61:41-51.
Competing interests: No competing interests