Concerns regarding the systematic review of vitamin D and mortality by Zhang and colleagues
There are several concerns about the systematic review by Zhang and colleagues of the effect of vitamin D supplementation on mortality,1
a. Data errors
1. The study by Lappe (2007) is stated to have 16 deaths in the analysis of all-cause mortality but 30 deaths from cancer in the analysis of cancer mortality. Likewise, for Prince (2008) and Zhu (2008), it is reported that there were more cancer-related deaths than total deaths. For all three studies, cancer incidence data have been used, not cancer mortality data.
2. Two of the studies (Ooms 1995 and Reid 2017) were sub-studies of larger trials (Lips 1996, ViDA 2017 respectively) and thus deaths in the sub-studies are counted twice.
3. Data from the large study, RECORD 2012, include post-trial follow-up for up to 3y when participants were not taking supplements, whereas no other study included extended post-trial follow-up. On-trial data for RECORD have been published previously.2-4
b. Discrepant subgroup and outcome analyses
Only 5 subgroup analyses derived from baseline characteristics were pre-specified in the PROSPERO protocol, yet 12 are described in the Methods and 14 in Table 2. Of 6 reported mortality outcomes (total and various subtypes), only 3 were specified in the PROSPERO protocol.
c. Multiple statistical tests
Multiple statistical analyses have been performed without consideration of the possibility of false positive results. If baseline variables used to define subgroups are independent, the probability of a false positive result from 14 subgroup analyses is about 50%.5 No adjustment was made for multiple statistical tests, nor did the authors acknowledge this limitation in their paper. In Table 2, 14 subgroup analyses based on baseline characteristics are reported, of which only 1 is statistically significant. Six mortality outcomes were reported, of which only one subgroup had a statistically significant benefit for vitamin D supplementation. Both these statistically significant but fragile results are highlighted in the Abstract.
d. Implausible research timeline
Based on the information in the PROSPERO document, the manuscript and the BMJ website, the authors started their review on 1 November 2018, registered the protocol on 13 December 2018, anticipated completing the review on 1 January 2019, conducted literatures searches up until 26 December 2018, and submitted the manuscript on 18 February 2019. Such a short timeline for a large systematic review involving screening of >20,000 titles that ultimately included 52 trials with multiple different outcomes is scarcely believable, especially when the authors state that they contacted researchers to obtain unpublished data when required, and contemporaneously the researchers were conducting other meta-analyses1,6-9- submitting four meta-analyses on a very diverse range of topics within 46 days in February and March 2019. 1,7-9
e. Authors’ conclusions
This is at least the 15th meta-analysis of vitamin D supplements and mortality or a subgroup of mortality in the past 12 years, and there remains no consistent evidence that vitamin D has any meaningful clinical benefit for these outcomes. The authors’ conclusion that “additional large clinical studies are needed to determine whether vitamin D3 supplementation is associated with lower all-cause mortality” is not justified by the existing evidence.
It is not practical for a reader to individually check data for each trial in all the analyses, but given the errors and issues described, it is difficult to have confidence in the integrity of the paper, and at the least a substantial correction is required, with every table and figure in the manuscript and 15 of 18 in the supplementary data requiring correction.
1. Zhang Y, Fang F, Tang J, et al. Association between vitamin D supplementation and mortality: systematic review and meta-analysis. BMJ 2019;366:l4673.
2. Grant AM, Avenell A, Campbell MK, et al. Oral vitamin D3 and calcium for secondary prevention of low-trauma fractures in elderly people (Randomised Evaluation of Calcium Or vitamin D, RECORD): a randomised placebo-controlled trial. Lancet 2005;365(9471):1621-8.
3. Ford JA, MacLennan GS, Avenell A, et al. Cardiovascular disease and vitamin D supplementation: trial analysis, systematic review, and meta-analysis. Am J Clin Nutr 2014;100(3):746-55.
4. Goulao B, Stewart F, Ford JA, et al. Cancer and vitamin D supplementation: a systematic review and meta-analysis. Am J Clin Nutr 2018;107(4):652-63.
5. Lagakos SW. The challenge of subgroup analyses--reporting without distorting. N Engl J Med 2006;354(16):1667-9.
6. Fang F, Zhang Y, Tang J, et al. Association of Corticosteroid Treatment With Outcomes in Adult Patients With Sepsis: A Systematic Review and Meta-analysis. JAMA Intern Med 2019;179(2):213-23.
7. Zhang Y, He J, Yuan Y, et al. Increased versus stable dose of inhaled corticosteroids for asthma exacerbations: A systematic review and meta-analysis. Clin Exp Allergy 2019.
8. Zhang Y, Jia L, Fang F, et al. General Anesthesia Versus Conscious Sedation for Intracranial Mechanical Thrombectomy: A Systematic Review and Meta-analysis of Randomized Clinical Trials. J Am Heart Assoc 2019;8(12):e011754.
9. Zhao Y, Yi C, Zhang Y, et al. Intensive follow-up strategies after radical surgery for nonmetastatic colorectal cancer: A systematic review and meta-analysis of randomized controlled trials. PLoS ONE 2019;14(7):e0220533.
Competing interests: We have published systematic reviews on vitamin D supplementation, generally concluding that vitamin D supplements do not have clinically relevant benefits for most individuals.