Type 2 diabetes and cancer: umbrella review of meta-analyses of observational studiesBMJ 2015; 350 doi: https://doi.org/10.1136/bmj.g7607 (Published 02 January 2015) Cite this as: BMJ 2015;350:g7607
All rapid responses
We thank Dr Stevens and Dr Riley for their comment on our umbrella systematic review (1), and totally agree with them that prediction intervals do not have a relationship with the statistical significance of the summary random effect in a meta-analysis (2). This is shown upfront in the definition of prediction intervals in the Methods of our umbrella review: "We also calculated the 95% prediction intervals for the summary random effects estimates, which further account for heterogeneity between studies and indicate the uncertainty for the effect that would be expected in a new study examining that same association. The 95% prediction interval shows where the true effects are for 95% of the studies from the population of studies that are synthesized or similar (exchangeable) studies that might be done in the future." We took extra care to avoid any conflation of prediction intervals with statistical significance in further sections of the manuscript, and it was our oversight that such a conflation remained in the second paragraph of the Discussion.
1. Tsilidis KK, Kasimis JC, Lopez DS, et al. Type 2 diabetes and cancer: umbrella review of meta-analyses of observational studies. BMJ 2015;350:g7607.
2. Riley RD, Higgins JP, Deeks JJ. Interpretation of random effects meta-analyses. BMJ 2011;342:d549.
Competing interests: No competing interests
Tsilidis et al., in an “umbrella” review of meta-analyses of diabetes and cancer, report prediction intervals from random effects meta-analyses . The reporting of prediction intervals after meta-analysis was suggested by Higgins et al.  and subsequently Riley et al. . However, clarification is needed regarding the use of statistical significance statements in regard to prediction intervals.
Following a random effects meta-analysis, the results are usually summarised by a summary estimate and its 95% confidence interval. These are typically indicated by the centre and width of a diamond at the bottom of a forest plot, respectively. The summary estimate of effect indicates the average treatment effect: in other words, it gives the best estimate of the average effect of treatment from a set of studies that represent different clinical or population settings, in which the effect of treatment may vary. The confidence interval indicates our degree of certainty about this average effect, based on the evidence available.
Confidence intervals, in meta-analysis as elsewhere, have a close relationship with statistical significance. In general, statistical significance is seen when the confidence interval excludes the null (for example, when a confidence interval for a relative risk lies wholly above or wholly below 1) and is not seen when the interval includes null (that is, in the case of a relative risk, includes the possibility that the relative risk is 1). Statistical significance at the usual 5% level correspond to 95% confidence intervals; a p-value less than 0.05 corresponds to a confidence interval that excludes the null. In a random effects meta-analysis, statistical significance is a statement about the average effect in different settings: can we reject the possibility that there is, on average, no effect? A p-value less than 0.05 indicates strong evidence that we can.
If a treatment performs well on average, it is also important to know whether it will perform well in the majority of settings themselves. A 95% prediction interval indicates the variability around the average effect, in terms of the potential treatment effect in just one setting. It answers the question (under assumptions) “what is the potential effect if we apply this treatment in a future setting”. The prediction interval is therefore a useful addition to the reporting of a random effects meta-analysis. Riley et al. suggested reporting prediction intervals in addition to confidence intervals, not as a replacement.
Prediction intervals address the range of possible effects, and do not have a relationship with the statistical significance of the average effect. Tsilidis’ Table 1 lists many meta-analyses in which the p-value shows significance but the prediction interval includes the null. For instance, for Zhu , the Table lists a random-effects p-value of p<0.001, but a prediction interval for the relative risk of 0.61 to 3.02. The correct interpretation is that we cannot be confident that the setting-specific relative risk will be greater than 1 in any future study (prediction interval includes values below 1), but we can be highly confident that the average relative risk across all settings is non-null (p-value small, and 95% confidence interval entirely above 1).
This point arose during peer review of the paper by Tsilidis, was agreed by authors and reviewer, and the paper adjusted accordingly. the conflation of prediction intervals with significance was corrected at almost every occurrence, but unfortunately, due to an oversight by reviewer and authors, one such remains. The second paragraph of the Discussion contains the following remark:
“… when calculating 95% prediction intervals … only the associations between type 2 diabetes and risk of developing breast cancer, intrahepatic cholangiocarcinoma, colorectal cancer and endometrial cancer remained nominally significant”.
We suggest a correct interpretation of these intervals might be:
Only for breast cancer, intrahepatic cholangiocarcinoma, colorectal cancer and endometrial cancer do we expect that the relative risk associated with diabetes will be greater than 1 in at least 95% of the settings of interest.
Taking a probabilistic (Bayesian) argument, one might alternatively say:
Only for breast cancer, intrahepatic cholangiocarcinoma, colorectal cancer and endometrial cancer is there a probability of at least 95% that the relative risk associated with diabetes will be greater than 1 in all the settings of interest.
This letter comes from the reviewer of the manuscript [RS] and an author of the paper recommending prediction intervals [RR] and is therefore offered to correct the record.
Richard J Stevens, Oxford
Richard D. Riley, Keele
1. Tsilidis KK, Kasimis JC, Lopez DS, Ntzani EE, Ioannidis JPA. Type 2 diabetes and cancer: Umbrella review of meta-analyses of observationlal studies. BMJ (Online). 2015;350.
2. Higgins JPT, Thompson SG, Spiegelhalter DJ. A re-evaluation of random-effects meta-analysis. Journal of the Royal Statistical Society.Series A: Statistics in Society. 2009;172(1):137-59.
3. Riley RD, Higgins JPT, Deeks JJ. Interpretation of random effects meta-analyses. BMJ. 2011;342(7804):964-7.
4. Zhu Z, Wang X, Shen Z, Lu Y, Zhong S, Xu C. Risk of bladder cancer in patients with diabetes mellitus: An updated meta-analysis of 36 observational studies. BMC Cancer. 2013;13.
Competing interests: No competing interests
The present study shows a significant risk of developing cancer in 20 sites, and mortality from seven cancer sites from 20 meta analyses (1). As reported by the authors, the study did not find any randomised control trials in relation to this observation. It is of significance since the previous studies focused on the observations which could be riddled with biases and hence the observed associations cannot be directly labelled as causal in nature. Many of the cancers such as that of breast, colorectal, gall bladder etc. have a common risk factor with diabetes type 2 such as high fatty intake in diet, obesity, sedentary lifestyle, alcohol intake etc. These factors may act as confounders and unless the studies are properly designed and analysed with appropriate statistical tests, one cannot comment on the role of diabetes in increasing the risk of developing cancers. Further, mere statistical significance cannot always point towards causality factors. We need to look at other aspects such as biological plausibility, strength of association, consistency, specificity, temporality, coherence, and analogy. We cannot apply for dose response relationship since the quantum of diabetes in terms of severity cannot be actually measured along with that of cancers.
Further, we also need to consider publication bias where chances of publishing articles with positive associations are likely to be published. The authors have also discussed that peer review selects the best studies and this may manifest as confirmation/allegiance bias. Considering the above observations it seems to be too premature to conclude any causal role of diabetes type 2 and cancers.
1. Tsilidi KK, Kasimis JC, Lopez DL, Ntzani EE, Ioannidis JPA. Type 2 diabetes and cancer: umbrella review of meta-analyses of observational studies. BMJ 2015;350:g7607
Competing interests: No competing interests