The editorialists respond to the authors
Professors Martineau and Camargo disagree with a number of points in our editorial on their individual patient data (IPD) meta-analysis on vitamin D supplements and respiratory tract infections (RTIs). As we indicated in our editorial, gathering IPD from 25 trials is an impressive achievement. However, when the authors make very strong claims in their paper “Our study reports a major new indication for vitamin D supplementation”, “Our results add to the body of evidence supporting the introduction of public health measures such as food fortification” and The Guardian reports that “… researchers calculate that daily or weekly supplements of vitamin D would mean 3.25 million fewer people in the UK having at least one respiratory infection a year…”, the veracity of these claims should be closely scrutinised.
To illustrate the issues in our editorial, we include here discussion of our trial-level meta-analyses of the raw unadjusted data from the authors’ Figure 2.
1. The authors present data using odds ratios rather than relative risks (also called risk ratios). Odds ratios overestimate risk reduction when the outcome is frequent.1 In the current meta-analysis, the incidence of RTIs in the controls was 42%. The headline result of an odds ratio of 0.88 (0.81-0.96) is equivalent to a relative risk (RR) of 0.93 (0.88-0.98), [using a proportion (P) of 0.422, and the formula RR = OR/(1-P+P*OR)].1 Performing a random-effects trial-level meta-analysis gives a relative risk of 0.93 (0.88-0.98), heterogeneity statistics: I2=48%, P=0.004, aligning closely to the IPD results. Thus, there was only a 7% reduction in the proportion of individuals with RTI with vitamin D. Unfortunately, in the BMJ’s press release and many media stories, the result was erroneously reported as a 12% reduction, wrongly nearly doubling the estimate of benefit. This error has not been corrected by the authors.
2. There is considerable heterogeneity in individual trial results, although the authors disagree with us about its importance. Of the 25 trials, six had statistically significant reductions in RTIs. These six trials included only a small proportion of the participants with RTI (557/4507, 12%) and total participants (1149/10933, 11%). In four of these six trials, RTI was not the primary endpoint, and all were small to moderately sized (all N<500). Four of the six trials were in children, and only one in adults >50y. Of the 14 trials with RTI as primary endpoint, only two had significant reductions in RTIs. Most of the 25 trials were small, and the four larger trials (N>500) reported neutral results. These data raise valid concerns about ‘small trial’ bias,2 and lack of generalisability to adult populations.
In our trial-level analysis, there was a significant interaction between baseline age, and trial results (P=0.03) using the four age bands chosen by the authors. This contrasts unexpectedly with the authors’ findings of no heterogeneity (P=0.61) even though risk estimates for each subgroup in the IPD analysis are similar to our trial-level results (≤1y RR 0.97, 0.90-1.05; 1.1-15.9y 0.75, 0.64-0.88; 16-65y 0.96, 0.90-1.03; >65y 0.98, 0.86-1.12). Therefore, there is a valid basis for considering age as a potential cause of heterogeneity.
Four of the 25 studies were carried out in developing countries - two in Afghanistan (their refs 35,42), one in Mongolia (ref 21) and the other in India (ref 44). Two of these four studies showed strong benefits for vitamin D (refs 21,42). It seems unwise to apply results from these trials in developing countries to developed countries, such as the UK. If these four trials are excluded from analyses, the pooled effect size is smaller and not statistically significant: RR 0.95 (0.89-1.00), and the heterogeneity decreases substantially (I2= 36%, P=0.05). Two studies were carried out in Afghanistan by the same group of researchers. The first, smaller study reported a benefit for vitamin D preventing recurrent pneumonia, but the second, larger study showed no effect in preventing incident pneumonia. These results raise questions about the reproducibility of results from a smaller secondary prevention trial in primary prevention in a lower risk population.
In summary, in the presence of significant heterogeneity, the pooled meta-analysis result, which represents an “average” result, may not apply to all populations studied. Here, there is significant heterogeneity that can be attributed to factors that probably cannot be separated because most of them are present in the trials with positive results: study population, location and size, age of participants, baseline vitamin D status and dosing regimen. One implication is clear: applying the pooled estimate to an unselected general population in developed countries with very different characteristics from the trials with positive results is inappropriate.
3. The VIDA trial completed in 2016 with more than 5000 participants and had RTIs as a co-primary endpoint (https://www.anzctr.org.au/Trial/Registration/TrialReview.aspx?id=336777). Some results from the trial have been submitted for publication. Because of its size, the trial will receive large weighting in meta-analyses. As soon as its results are available, the current meta-analysis will be outdated. If the trial results are neutral, the pooled meta-analysis estimate will likely be neutral, but if the results are positive, that would support the authors’ case. Given the very great public, media and academic interest, we encourage Professor Camargo, a principal investigator on VIDA, to immediately release the results for RTI from VIDA.
4. The authors say we should have used adjusted rather than raw data in our comments regarding absolute risk reduction. The difference is trivial. The same arguments apply whether the absolute risk reduction is from 42% to 40% or 42.2% to 39.1%.
5. The authors compare the results from vitamin D to influenza vaccination. Notwithstanding the inadvisability of conflating influenza with much less serious RTIs, their comparison is invalid. The two studies with influenza as the endpoint showed no effect of vitamin D (refs 26 and 27).
6. The authors argue on theoretical grounds that pooling data from a diverse group of endpoints (producing a composite endpoint) is reasonable. However, Table 4 also shows that there was no effect on the more specific endpoints of upper or lower RTI. We prefer to focus on trial results, which are quite inconsistent, rather than theoretical arguments. Clinical interpretation of composite endpoints is difficult. When there are large variations between components, in importance, frequency, and results, it is recommended that the composite endpoint be abandoned.3 We think these criteria are met for the composite endpoint of RTI, and therefore the composite endpoint results are not generalizable.
7. The starting point for our editorial was what this analysis adds to the 10 previous systematic reviews and meta-analyses, with eight since 2012, and why the current results differ from previous ones. Consistency of findings from systematic reviews lends credence to the robustness of evidence. When faced with the same data, different authors can make different decisions about study inclusion, study quality, and use different methods, and these differences can influence the results and conclusions.4,5 In this case, two studies included in previous systematic reviews6,7 that appeared eligible to us were not included, but two studies that retrospectively gathered data, which was an exclusion criterion, were included.8-10 Given that such decisions influence results, it is important that the decisions made regarding trial inclusion and study methods are explained fully and transparently.
The authors appear to acknowledge that two trials meet their exclusion criteria but justify inclusion because they were unlikely to bias the results. Perhaps this is true, but there may be other studies that also were ineligible for this meta-analysis which also were unlikely to have biased results. At the very least, all departures from the published protocol should be justified in the paper, particularly when inclusion or exclusion of such trials could influence the overall conclusions.
8. Professor Martineau makes the point our editorial was not peer reviewed. This is true, as usually occurs with editorials, but our previous publications on vitamin D, including several meta-analyses, have been. Our recent practice piece in the BMJ11 had nine reviewers. However, Professor Martineau was able to review our editorial ahead of publication. After seeing a draft version of the BMJ’s press release, he requested a copy of our editorial from us as a courtesy, which we provided in good faith. He then requested that we substantially modify it, initially directly to us, and then to Dr Fiona Godlee, the BMJ editor. The BMJ asked us to take a look at what he said and make any necessary changes. Many of the issues he raised are repeated in Professors Martineau and Camargo’s rapid response, which we have now responded to in public.
In summary, we do not think the points raised by Professors Martineau and Camargo justify altering the conclusion in our editorial that the current evidence is not sufficient to support prescribing vitamin D supplements or fortifying food with vitamin D to prevent RTIs. When large, well-conducted randomised controlled trials show benefits from vitamin D on RTI in appropriate populations, they should inform sensible clinical care and public health policy, but we do not have such trial results yet.
1. Grant RL. Converting an odds ratio to a range of plausible relative risks for better communication of research findings. BMJ 2014; 348:f7450.
2. Sterne JA, Egger M, Smith GD. Systematic reviews in health care: Investigating and dealing with publication and other biases in meta-analysis. BMJ 2001; 323:101-5.
3. Montori VM, Permanyer-Miralda G, Ferreira-Gonzalez I, Busse JW, Pacheco-Huergo V, Bryant D, et al. Validity of composite end points in clinical trials. BMJ 2005; 330:594-6.
4. Bolland MJ, Grey A. A case study of discordant overlapping meta-analyses: vitamin d supplements and fracture. PLoS One 2014; 9:e115934.
5. Bolland MJ, Grey A, Reid IR. Differences in overlapping meta-analyses of vitamin d supplements and falls. J Clin Endocrinol Metab 2014; 99:4265-72.
6. Aloia JF, Li-Ng M. Re: epidemic influenza and vitamin D. Epidemiol Infect 2007; 135:1095-6; author reply 7-8.
7. Bischoff-Ferrari HA, Dawson-Hughes B, Platz A, Orav EJ, Stahelin HB, Willett WC, et al. Effect of high-dosage cholecalciferol and extended physiotherapy on complications after hip fracture: a randomized controlled trial. Arch Intern Med 2010; 170:813-20.
8. Camargo CA, Jr., Ganmaa D, Frazier AL, Kirchberg FF, Stuart JJ, Kleinman K, et al. Randomized trial of vitamin D supplementation and risk of acute respiratory infection in Mongolia. Pediatrics 2012; 130:e561-7.
9. Neale R, Armstrong B, Ebeling P, English D, Kimlin M, van der Pols J, et al. Effect of vitamin D supplementation on antibiotic use and upper respiratory tract infection: a randomised controlled trial. Am J Epidemiol 2013; 177:S131-S, 521-S.
10. Tran B, Armstrong BK, Ebeling PR, English DR, Kimlin MG, van der Pols JC, et al. Effect of vitamin D supplementation on antibiotic use: a randomized controlled trial. Am J Clin Nutr 2014; 99:156-61.
11. Bolland MJ, Avenell A, Grey A. Should adults take vitamin D supplements to prevent disease? BMJ 2016; 355:i6201.
Competing interests: No competing interests