Intake of saturated and trans unsaturated fatty acids and risk of all cause mortality, cardiovascular disease, and type 2 diabetes: systematic review and meta-analysis of observational studies
Meta-analyses have also become popular in nutrition research. However, statistics behind a random-effects meta-analysis is a nuanced one (1). When random-effects model for meta-analysis is fitted, one needs to estimate between-study variance (τ2 i.e. tau-squared) with one of the many estimators available (1) to obtain summary effect estimate and confidence intervals (CIs) for this effect estimate.
Routinely, the random-effects modeling is based on approach proposed by DerSimonian and Laird (DL) (2) and CIs for the summary effect estimate are constructed using standard normal distribution. However, this approach might not be the most reliable approach, at least in some situations (3,4). Different analytical choices might make difference since research findings are typically based on a statistical significance threshold.
For example, a recent Cochrane review showed in prespecified subgroup analysis that saturated fats (SFA) replaced with polyunsaturated fats (PUFA) reduced cardiovascular events (risk ratio [RR] 0.73; 95% CI 0.58 to 0.92, P=0.007, Tau2=0.06, I2=69%, 7 studies) (5,6). A Mantel-Haenszel method was used in the Cochrane review, however, an inverse-variance method yielded identical results as depicted above.
I used recent editorial by Hooper et al considering their Cochrane review (6) to extract the relevant data, and re-analyzed it with R (R Core Team 2014, R Foundation For Statistical Computing, Vienna, Austria, version 3.1.2) using meta package (7) to adjust confidence interval and p-value for the summary effect estimate with the t-distribution based method by Hartung and Knapp and by Sidik and Jonkman (HKSJ) (see ref. 3 for details).
As shown in Figure, RR for cardiovascular events was 0.73 (95% CI 0.50 to 1.08, two-tailed P=0.095) with the method by HKSJ. As evident, the CI became wider and the P-value larger. By all means, this re-analysis should not be taken at face value, however, it illustrates that statistics matter. Evidently, if one is willing to accept the statistically significant versus non-significant framework, then two different conclusions abound.
In simulated meta-analyses, for example with 7 unequal sized trials and statistical heterogeneity (I2= 50-90%), error rates exceeded the 5% threshold and were around 10% with the DL approach (3). In addition, of 185 statistically significant binary outcome Cochrane meta-analyses with DL, 26 % (48/185) were non-significant with HKSJ (3). Interestingly, this is exactly what happened with the two different analytical choices. The width of CI might be too narrow in the aforementioned Cochrane review or any other meta-analysis with a statistical heterogeneity thereof (4). Naturally, other between-study variance estimators than DL can be used.
Undoubtedly, a Bayesian framework can provide broader interpretation of statistically significant research findings with the calculation of posterior probability for true positivity (8).
After all, the strength of evidence depends on who evaluates it and which methodology is adopted by the guideline panels for grading the evidence. Maybe statistics is much more important than acknowledged.
1. Veroniki AA, Jackson D, Viechtbauer W, et al. Methods to estimate the between-study variance and its uncertainty in meta-analysis. Res Synth Methods 2015 Sep 2. doi: 10.1002/jrsm.1164.
2. DerSimonian R, Laird N. Meta-analysis in clinical trials. Control Clin Trials. 1986; 7: 177-88.
3. IntHout J, Ioannidis JP, Borm GF. The Hartung-Knapp-Sidik-Jonkman method for random effects meta-analysis is straightforward and considerably outperforms the standard DerSimonian-Laird method. BMC Med Res Methodol 2014; 14: 25.
4. Cornell JE, Mulrow CD, Localio R, et al. Random-effects meta-analysis of inconsistent effects: a time for change. Ann Intern Med 2014; 160: 267-70.
5. Hooper L, Martin N, Abdelhamid A, Davey Smith G. Reduction in saturated fat intake for cardiovascular disease. Cochrane Database Syst Rev 2015; 6: CD011737.
6. Hooper L, Martin N, Abdelhamid A. Cochrane corner: what are the effects of reducing saturated fat intake on cardiovascular disease and mortality? Heart 2015 Oct 9. doi: 10.1136/heartjnl-2015-308521.
Rapid Response:
Does quality of dietary fats matter?
Meta-analyses have also become popular in nutrition research. However, statistics behind a random-effects meta-analysis is a nuanced one (1). When random-effects model for meta-analysis is fitted, one needs to estimate between-study variance (τ2 i.e. tau-squared) with one of the many estimators available (1) to obtain summary effect estimate and confidence intervals (CIs) for this effect estimate.
Routinely, the random-effects modeling is based on approach proposed by DerSimonian and Laird (DL) (2) and CIs for the summary effect estimate are constructed using standard normal distribution. However, this approach might not be the most reliable approach, at least in some situations (3,4). Different analytical choices might make difference since research findings are typically based on a statistical significance threshold.
For example, a recent Cochrane review showed in prespecified subgroup analysis that saturated fats (SFA) replaced with polyunsaturated fats (PUFA) reduced cardiovascular events (risk ratio [RR] 0.73; 95% CI 0.58 to 0.92, P=0.007, Tau2=0.06, I2=69%, 7 studies) (5,6). A Mantel-Haenszel method was used in the Cochrane review, however, an inverse-variance method yielded identical results as depicted above.
I used recent editorial by Hooper et al considering their Cochrane review (6) to extract the relevant data, and re-analyzed it with R (R Core Team 2014, R Foundation For Statistical Computing, Vienna, Austria, version 3.1.2) using meta package (7) to adjust confidence interval and p-value for the summary effect estimate with the t-distribution based method by Hartung and Knapp and by Sidik and Jonkman (HKSJ) (see ref. 3 for details).
As shown in Figure, RR for cardiovascular events was 0.73 (95% CI 0.50 to 1.08, two-tailed P=0.095) with the method by HKSJ. As evident, the CI became wider and the P-value larger. By all means, this re-analysis should not be taken at face value, however, it illustrates that statistics matter. Evidently, if one is willing to accept the statistically significant versus non-significant framework, then two different conclusions abound.
In simulated meta-analyses, for example with 7 unequal sized trials and statistical heterogeneity (I2= 50-90%), error rates exceeded the 5% threshold and were around 10% with the DL approach (3). In addition, of 185 statistically significant binary outcome Cochrane meta-analyses with DL, 26 % (48/185) were non-significant with HKSJ (3). Interestingly, this is exactly what happened with the two different analytical choices. The width of CI might be too narrow in the aforementioned Cochrane review or any other meta-analysis with a statistical heterogeneity thereof (4). Naturally, other between-study variance estimators than DL can be used.
Undoubtedly, a Bayesian framework can provide broader interpretation of statistically significant research findings with the calculation of posterior probability for true positivity (8).
After all, the strength of evidence depends on who evaluates it and which methodology is adopted by the guideline panels for grading the evidence. Maybe statistics is much more important than acknowledged.
1. Veroniki AA, Jackson D, Viechtbauer W, et al. Methods to estimate the between-study variance and its uncertainty in meta-analysis. Res Synth Methods 2015 Sep 2. doi: 10.1002/jrsm.1164.
2. DerSimonian R, Laird N. Meta-analysis in clinical trials. Control Clin Trials. 1986; 7: 177-88.
3. IntHout J, Ioannidis JP, Borm GF. The Hartung-Knapp-Sidik-Jonkman method for random effects meta-analysis is straightforward and considerably outperforms the standard DerSimonian-Laird method. BMC Med Res Methodol 2014; 14: 25.
4. Cornell JE, Mulrow CD, Localio R, et al. Random-effects meta-analysis of inconsistent effects: a time for change. Ann Intern Med 2014; 160: 267-70.
5. Hooper L, Martin N, Abdelhamid A, Davey Smith G. Reduction in saturated fat intake for cardiovascular disease. Cochrane Database Syst Rev 2015; 6: CD011737.
6. Hooper L, Martin N, Abdelhamid A. Cochrane corner: what are the effects of reducing saturated fat intake on cardiovascular disease and mortality? Heart 2015 Oct 9. doi: 10.1136/heartjnl-2015-308521.
7. Schwarzer Guido. meta: General Package for Meta-analysis. R-package version 4.3-0. 2015. https://cran.r-project.org/web/packages/meta/index.html. Cited October 20, 2015.
8. Pereira TV, Ioannidis JP. Statistically significant meta-analyses of clinical trials have modest credibility and inflated effects. J Clin Epidemiol 2011; 64: 1060-9.
jesper.m.kivela@helsinki.fi
Competing interests: I like statistics.