Implausible results in human nutrition research

BMJ 2013; 347 doi: https://doi.org/10.1136/bmj.f6698 (Published 14 November 2013) Cite this as: BMJ 2013;347:f6698

Re: Implausible results in human nutrition research

We read with great interest Dr. Ioannidis’s paper on “Implausible results in human nutrition research: Definitive solutions won’t come from another million observational papers or small randomized trials” (1). However, we would like to point out a few statements in his article that need to be read with caution.

Randomized trials and observational studies are designed to ask slightly different research questions. An intention-to-treat analysis from randomized trials can give us an estimate of the health effects of the intervention, but will tend to underestimate the true effect due to imperfect long-term compliance. Observational studies examine the effect of those actually reporting the exposure, which is a different question and may be more relevant to the study of the health effects of food or nutrition on disease. Also, the findings from trials will usually be very imprecise because studies are typically stopped for ethical reasons once a result is statistically significant, meaning the confidence interval for the effect will range from near the null value to implausibly high.

Dr. Ioannidis used the drug paradigm to evaluate the effects of dietary factors, which is entirely inappropriate. While RCTs remain the gold standard for evaluating drug efficacy, it is not the right approach to evaluate the relationship between diet and risk of chronic diseases, which may take years or even decades to manifest and thus participants’ compliance to a certain diet regimen is typically very low (2). Furthermore, unlike drug trials, diet interventions often involve trade-offs of nutrients or foods (3). For example, when fat consumption is decreased, intake of carbohydrate often goes up to make up for the decreased calories (3, 4, 5). These dietary changes can only be achieved through changes in the overall dietary patterns. For nutrition and chronic diseases research, three main conceptual models have been proposed (6): 1) pathway, 2) cumulative exposure and 3) critical/sensitivity period models. Heterogeneity of results from different study designs may be attributed to the research questions posed, and the time period for which the nutrient intake was observed or intervened upon. Observational studies provide unique opportunities to examine the long-term consequences of diet in early life (life-course epidemiology), which randomized trials cannot. In addition, we find that Dr. Ioannidis’s “definitive solution” is highly questionable and oversimplified. With the advancement of consumer knowledge, new foods and processing methods will continuously be introduced and new products will enter the market. As a result, our dietary recommendations will also need to change accordingly(7). For example, a large randomized trial of a high folate diet conducted in the 1980s or 1990s would likely give a different result than if conducted in a more contemporary setting where approximately 80 countries have a folate fortified food supply to reduce neural tube defects (8). It is likely that large long-term observational studies can document the association of dietary folate with chronic disease pre and post folate fortification.

Dr. Ioannidis stated that “Nutritional intake is notoriously difficult to capture with the questionnaire methods used by most studies” (1). The best, most accurate large prospective studies follow participants for many years with repeated measurement of dietary intake assessed through dietary questionnaires rigorously validated against detailed quantitative measures of intake and biochemical indicators. Questionnaires do have error, as do all ascertainment methods, but repeated measurements of long term diet reduce that error and also offer the unique opportunities to investigate changes in intake over time, detailed dose-response relationships, and assessment of critical exposure periods of etiologic effect. Our early studies of long term multivitamin use to reduce cancer risk suggested that the etiologic period of effect was 1-2 decades (9), a finding recently confirmed with a clinical trial (10). Measurement error correction methods have also been developed for both fixed covariates and time-varying covariates, to account for the inevitable error that occurs with assessing human behavior with any survey method (11-13).

Dr. Ioannidis concluded that in the future we would need pivotal mega-trials of comprehensive interventions (1). We would like to raise our concerns about “mega-trials” because bigger is not necessarily better. The counterfactual outcome of the “comprehensive interventions” is hard to define, and effect estimates of “comprehensive interventions” depend on the distribution of the different versions of these interventions as actually applied in a study population, which can be unknown (14). The MRFIT trial was just as he describes – a large Mega Trial that failed because both arms improved towards the comprehensive lifestyle intervention. Thus just creating a large trial does not guarantee anything more valid than an observational study or a small trial. Also assumptions are needed to infer the estimated effect of “comprehensive interventions” from one study population to another (14).

Dr. Ioannidis claimed that "food security, sustainability, social inequalities, famine, and impact of food production on climate change" are more important, but did not provide the relevant evidence, either observational or experimental. We wonder how he would conduct ethical and feasible pivotal mega-trials on these topics. These factors are upstream factors that determine individual food choices and dietary behaviors, which are most proximal to pathophysiology of diseases and health outcomes. Clearly, nutrition is a multi-dimensional and multi-level rather than a single-dimensional research field.

Despite the limitations related to observational studies, high-quality nutritional epidemiology research has provided new and valuable information for the public good. For many questions there is little alternative for human studies. Because many issues are directly related to the well being of individuals and populations, continued efforts to refine methods and data quality are warranted.

1. Ioannidis JP. Implausible results in human nutrition research. BMJ. 2013 Nov 14;347:f6698. doi: 10.1136/bmj.f6698. PubMed PMID: 24231028.
2. Willett WC. The WHI joins MRFIT: a revealing look beneath the covers. Am J Clin Nutr. 2010;91(4):829-30.
3. Hu FB, Willett WC. Optimal diets for prevention of coronary heart disease.JAMA. 2002;288(20):2569-78.
4. Siri-Tarino PW, Sun Q, Hu FB, Krauss RM. Saturated fatty acids and risk of coronary heart disease: modulation by replacement nutrients. Curr Atheroscler Rep. 2010;12(6):384-90.
5. Hu FB. Are refined carbohydrates worse than saturated fat? Am J Clin Nutr.2010;91(6):1541-2.
6. Colditz GA. Overview of the epidemiology methods and applications: strengths and limitations of observational study designs. Crit Rev Food Sci Nutr. 2010;50 Suppl 1:10-2. doi: 10.1080/10408398.2010.526838.
7. Willett WC, Stampfer MJ. Current evidence on Healthy Eating. Annu. Rev. Public Health 2013, 34:77-95.
8. Rimm EB, Stampfer MJ. Folate and cardiovascular disease: one size does not fit all. Lancet. 2011;378(9791):544-6.
9. Giovannucci E, Stampfer MJ, Colditz GA, Hunter DJ, Fuchs C, Rosner BA et al. Multivitamin use, folate, and colon cancer in women in the Nurses' Health Study. Ann Intern Med. 1998;129(7):517-24.
10. Gaziano JM, Sesso HD, Christen WG, Bubes V, Smith JP, MacFadyen J et al. Multivitamins in the prevention of cancer in men:the Physicians' Health Study II randomized controlled trial. JAMA. 2012;308(18):1871-80.
11. Rosner B. Willett WC, Spiegelman D. 1989. Correction of logistic regression relative risk estimates and confidence intervals for systematic within-person measurement error. Stat. Med. 8:1051-1069.
12. Rosner B. Gore R. 2001. Measurement error correction in nutritional epidemiology based on individual foods, with application to the relation of diet to breast cancer. Am. J. Epidemiol. 154:827-835.
13. Liao XM, Zucker DM, Li Y, Spiegelman D. “Survival Analysis with Error-Prone Time-Varying Covariates: A Risk Set Calibration Approach”. Biometrics, 2011 Mar;67(1):50-58.; PMCID: PMC2927810
14. Hernan MA, VanderWeele TJ. Compound treatments and transportability of causal inference. Epidemiology. 2011. Vol 22 No.3 Page 368

Competing interests: No competing interests

16 May 2014
Shanshan Li
Postdoctoral Research Fellow
Alan Flint, Jennifer Pai, John P. Forman, Frank B. Hu, Walter C. Willett, Kathryn M. Rexrode, Kenneth J. Mukamal, Eric B. Rimm
Harvard School of Public Health
677 Huntington Ave, Boston MA 02115