Implausible results in human nutrition research
BMJ 2013; 347 doi: https://doi.org/10.1136/bmj.f6698 (Published 14 November 2013) Cite this as: BMJ 2013;347:f6698
All rapid responses
Rapid responses are electronic comments to the editor. They enable our users to debate issues raised in articles published on bmj.com. A rapid response is first posted online. If you need the URL (web address) of an individual response, simply click on the response headline and copy the URL from the browser window. A proportion of responses will, after editing, be published online and in the print journal as letters, which are indexed in PubMed. Rapid responses are not indexed in PubMed and they are not journal articles. The BMJ reserves the right to remove responses which are being wilfully misrepresented as published articles or when it is brought to our attention that a response spreads misinformation.
From March 2022, the word limit for rapid responses will be 600 words not including references and author details. We will no longer post responses that exceed this limit.
The word limit for letters selected from posted responses remains 300 words.
We read with great interest Dr. Ioannidis’s paper on “Implausible results in human nutrition research: Definitive solutions won’t come from another million observational papers or small randomized trials” (1). However, we would like to point out a few statements in his article that need to be read with caution.
Randomized trials and observational studies are designed to ask slightly different research questions. An intention-to-treat analysis from randomized trials can give us an estimate of the health effects of the intervention, but will tend to underestimate the true effect due to imperfect long-term compliance. Observational studies examine the effect of those actually reporting the exposure, which is a different question and may be more relevant to the study of the health effects of food or nutrition on disease. Also, the findings from trials will usually be very imprecise because studies are typically stopped for ethical reasons once a result is statistically significant, meaning the confidence interval for the effect will range from near the null value to implausibly high.
Dr. Ioannidis used the drug paradigm to evaluate the effects of dietary factors, which is entirely inappropriate. While RCTs remain the gold standard for evaluating drug efficacy, it is not the right approach to evaluate the relationship between diet and risk of chronic diseases, which may take years or even decades to manifest and thus participants’ compliance to a certain diet regimen is typically very low (2). Furthermore, unlike drug trials, diet interventions often involve trade-offs of nutrients or foods (3). For example, when fat consumption is decreased, intake of carbohydrate often goes up to make up for the decreased calories (3, 4, 5). These dietary changes can only be achieved through changes in the overall dietary patterns. For nutrition and chronic diseases research, three main conceptual models have been proposed (6): 1) pathway, 2) cumulative exposure and 3) critical/sensitivity period models. Heterogeneity of results from different study designs may be attributed to the research questions posed, and the time period for which the nutrient intake was observed or intervened upon. Observational studies provide unique opportunities to examine the long-term consequences of diet in early life (life-course epidemiology), which randomized trials cannot. In addition, we find that Dr. Ioannidis’s “definitive solution” is highly questionable and oversimplified. With the advancement of consumer knowledge, new foods and processing methods will continuously be introduced and new products will enter the market. As a result, our dietary recommendations will also need to change accordingly(7). For example, a large randomized trial of a high folate diet conducted in the 1980s or 1990s would likely give a different result than if conducted in a more contemporary setting where approximately 80 countries have a folate fortified food supply to reduce neural tube defects (8). It is likely that large long-term observational studies can document the association of dietary folate with chronic disease pre and post folate fortification.
Dr. Ioannidis stated that “Nutritional intake is notoriously difficult to capture with the questionnaire methods used by most studies” (1). The best, most accurate large prospective studies follow participants for many years with repeated measurement of dietary intake assessed through dietary questionnaires rigorously validated against detailed quantitative measures of intake and biochemical indicators. Questionnaires do have error, as do all ascertainment methods, but repeated measurements of long term diet reduce that error and also offer the unique opportunities to investigate changes in intake over time, detailed dose-response relationships, and assessment of critical exposure periods of etiologic effect. Our early studies of long term multivitamin use to reduce cancer risk suggested that the etiologic period of effect was 1-2 decades (9), a finding recently confirmed with a clinical trial (10). Measurement error correction methods have also been developed for both fixed covariates and time-varying covariates, to account for the inevitable error that occurs with assessing human behavior with any survey method (11-13).
Dr. Ioannidis concluded that in the future we would need pivotal mega-trials of comprehensive interventions (1). We would like to raise our concerns about “mega-trials” because bigger is not necessarily better. The counterfactual outcome of the “comprehensive interventions” is hard to define, and effect estimates of “comprehensive interventions” depend on the distribution of the different versions of these interventions as actually applied in a study population, which can be unknown (14). The MRFIT trial was just as he describes – a large Mega Trial that failed because both arms improved towards the comprehensive lifestyle intervention. Thus just creating a large trial does not guarantee anything more valid than an observational study or a small trial. Also assumptions are needed to infer the estimated effect of “comprehensive interventions” from one study population to another (14).
Dr. Ioannidis claimed that "food security, sustainability, social inequalities, famine, and impact of food production on climate change" are more important, but did not provide the relevant evidence, either observational or experimental. We wonder how he would conduct ethical and feasible pivotal mega-trials on these topics. These factors are upstream factors that determine individual food choices and dietary behaviors, which are most proximal to pathophysiology of diseases and health outcomes. Clearly, nutrition is a multi-dimensional and multi-level rather than a single-dimensional research field.
Despite the limitations related to observational studies, high-quality nutritional epidemiology research has provided new and valuable information for the public good. For many questions there is little alternative for human studies. Because many issues are directly related to the well being of individuals and populations, continued efforts to refine methods and data quality are warranted.
1. Ioannidis JP. Implausible results in human nutrition research. BMJ. 2013 Nov 14;347:f6698. doi: 10.1136/bmj.f6698. PubMed PMID: 24231028.
2. Willett WC. The WHI joins MRFIT: a revealing look beneath the covers. Am J Clin Nutr. 2010;91(4):829-30.
3. Hu FB, Willett WC. Optimal diets for prevention of coronary heart disease.JAMA. 2002;288(20):2569-78.
4. Siri-Tarino PW, Sun Q, Hu FB, Krauss RM. Saturated fatty acids and risk of coronary heart disease: modulation by replacement nutrients. Curr Atheroscler Rep. 2010;12(6):384-90.
5. Hu FB. Are refined carbohydrates worse than saturated fat? Am J Clin Nutr.2010;91(6):1541-2.
6. Colditz GA. Overview of the epidemiology methods and applications: strengths and limitations of observational study designs. Crit Rev Food Sci Nutr. 2010;50 Suppl 1:10-2. doi: 10.1080/10408398.2010.526838.
7. Willett WC, Stampfer MJ. Current evidence on Healthy Eating. Annu. Rev. Public Health 2013, 34:77-95.
8. Rimm EB, Stampfer MJ. Folate and cardiovascular disease: one size does not fit all. Lancet. 2011;378(9791):544-6.
9. Giovannucci E, Stampfer MJ, Colditz GA, Hunter DJ, Fuchs C, Rosner BA et al. Multivitamin use, folate, and colon cancer in women in the Nurses' Health Study. Ann Intern Med. 1998;129(7):517-24.
10. Gaziano JM, Sesso HD, Christen WG, Bubes V, Smith JP, MacFadyen J et al. Multivitamins in the prevention of cancer in men:the Physicians' Health Study II randomized controlled trial. JAMA. 2012;308(18):1871-80.
11. Rosner B. Willett WC, Spiegelman D. 1989. Correction of logistic regression relative risk estimates and confidence intervals for systematic within-person measurement error. Stat. Med. 8:1051-1069.
12. Rosner B. Gore R. 2001. Measurement error correction in nutritional epidemiology based on individual foods, with application to the relation of diet to breast cancer. Am. J. Epidemiol. 154:827-835.
13. Liao XM, Zucker DM, Li Y, Spiegelman D. “Survival Analysis with Error-Prone Time-Varying Covariates: A Risk Set Calibration Approach”. Biometrics, 2011 Mar;67(1):50-58.; PMCID: PMC2927810
14. Hernan MA, VanderWeele TJ. Compound treatments and transportability of causal inference. Epidemiology. 2011. Vol 22 No.3 Page 368
Competing interests: No competing interests
We read with great interest Dr. Ioannidis’s paper on “Implausible results in human nutrition research: Definitive solutions won’t come from another million observational papers or small randomized trials” (1). While we can appreciate his editorial on the subject, this general discussion on the strengths and limitations of observational and experimental studies is not unique to nutrition and is commonplace in textbooks on epidemiological methods dating back to MacMahon and Pugh’s first text on the subject in 1970 (2). Strong commentary on the limitations of epidemiology in the study of nutrition can be traced back to Taubes’ article in Science in 1995 (3). A thorough rehashing of his arguments goes beyond the scope of this response, but in all areas of epidemiology (and other areas of scientific study), false positive and false negative results inevitably occur, and many studies lack replication. We discuss some of the main points below.
Dr. Ioannidis’s comment that “Many findings are entirely implausible” is not helpful as implausible finding occur by chance or other reasons in almost all areas of science, and conclusions should not be based on a single finding but rather the plausibility given all available evidence and on replication (1). In nutritional research, we usually evaluate the evidence from animal/mechanistic experiments, biological/metabolic findings, and epidemiological/intervention studies together to reach conclusions (4). No single study should stand on its own, especially now that decades of nutritional research are available. Similarly, no single editorial should completely dismiss the great value the last 50 years of research has provided. Dr. Ioannidis selectively picked a few estimates from the vast amount of nutrition literature without any context. More importantly, he did not include the 95% confidence intervals surrounding those estimates. The 95% CIs can provide important information on both certainty and uncertainty of the point estimates (5). In evidence-based nutrition, meta-analyses and systematic reviews instead of findings from a single study are used to develop recommendations or guidelines. Therefore, consistency and replications of the findings across different populations are extremely important.
If we use dietary fat and heart disease as a first example, the weak and inconsistent results mainly derive from early studies, which were not designed to investigate the fat-CHD association. However, they should not be interpreted as providing strong negative evidence (6). The limitations in early studies include small sample size; use of a single 24-hour recall for dietary assessment; lack of updated dietary measurement during follow-up; and no adjustment for total energy intake or other dietary factors. More recent large prospective cohort studies have addressed these limitations. Results from high quality observational studies correspond well with randomized controlled trials (Table 1). Interestingly, in a publication from 2005, Dr. Ioannidis made the similarly dramatic claim that “Most biomedical research findings are false for most research designs and for most fields” (7). Goodman and Greenland (and others) have provided detailed responses to this claim and identified the logical flaws of Dr. Ioannidis’ assertion (8-13).
Dr. Ioannidis concluded that in future we would need randomized controlled trials (RCTs) 10 times the sample size of the Prevención con Dieta Mediterránea (PREDIMED) study (1). Interestingly, the majority of nutrition trials, even the size of PREDIMED, would not have been proposed or conducted had there not been very compelling evidence of the benefits of the Mediterranean diet, olive oil, and nuts from short term trials with intermediate endpoint and observational studies with coronary heart disease - the exact studies Dr. Ioannidis argues against performing. Does Dr. Ioannidis suggest that we conduct randomized trials 10 times the size of PREDIMED based on no more than simple conjuring of hypothetical biological pathways connecting nutrition with long term health?
Contrary to common beliefs, RCTs are not immune to confounding and serious bias even if they are large, and they can provide seriously misleading answers (14). For example, RCT's of smoking cessation showed no benefit on mortality (15), very low birth weight, neonatal death or perinatal mortality (16). Also, studies such as the Women’s Health Initiative, with a $400 million arm to test the benefits of a very low fat diet, failed to test the fat hypothesis due to low compliance with the dietary intervention (17). The earlier Multiple Risk Factor Intervention Trial (MRFIT) also failed to test its original hypothesis due to poor adherence and found no significant effect of an intervention combining diet, smoking cessation and hypertension treatment on coronary disease (18). These issues will not simply disappear by increasing sample size and they will worsen with longer follow-up(14). Randomized trials are randomized and free of confounding only at baseline, and their interpretation is complicated by attrition and non-compliance that is subject to confounding and selection bias - the same issues we are concerned about in observational studies.
Finally, Dr. Ioannidis comment that “Definitive solutions won’t come from another million observational papers or small randomized trials” (1) over simplifies the conduct of nutrition research and shows a deeper misunderstanding of the field. Because of the complex nature of our field, conclusions need to be based on the broader totality of evidence from human observational and clinical studies. In the past decades, cardiovascular mortality has decreased dramatically in Finland, US, and other developed countries, and more than half of the decline is attributable to improvement in diet and lifestyle factors. The landmark North Karelia study (44), the Seven Countries Study (45), the Framingham Heart study (46), the Minnesota Heart Health Program (47), and many other studies have shown that his claim of implausible nutrition finding does not hold water. A recent example would be trans fat. Well conducted long term observational studies and small randomized trials provided consistent and convincing evidence of the adverse effects of trans fat on lipid levels, inflammation, endothelial dysfunction, and significantly increased risk of coronary heart disease (48, 49). Despite the lack of large, long-term trials (48, 49) important changes were made to nutritional labels and policies across the globe, and food companies have removed up to 80% of trans fat in foods in the US. This clear public health advance was made with just the evidence that Dr. Ioannidis scorned; millions of premature deaths will be prevented that would not have been had we waited to conduct a mega-trial of trans fat to prove harm, and which could easily provide misleading results if it had the same fate as the WHI or MRFIT trials.
Reference:
1. Ioannidis JP. Implausible results in human nutrition research. BMJ. 2013 Nov 14;347:f6698. doi: 10.1136/bmj.f6698.
2. MacMahon B, Pugh TF. Epidemiologic Methods (Little, Brown; 1960); reissued as Epidemiology: Principles and Methods (Little, Brown; 1970) (ISBN 0316542598)
3. Taubes, Gary; Mann, Charles C. Epidemiology faces its limits. Science; Jul 14, 1995;269, 5221. Pg. 164-169
4. Willett WC, Stampfer MJ. Current evidence on Healthy Eating. Annu. Rev. Public Health 2013, 34:77-95.
5. Altman DG. Why we need confidence intervals. World J Surg. 2005;29(5):554-6.
6. Willett WC. Dietary fats and coronary heart disease. J Intern Med. 2012 Jul;272(1):13-24. doi: 10.1111/j.1365-2796.2012.02553.x.
7. Ioannidis JP. 2005. Why most published research findings are false. PLoS Med 2:e123.
8. Goodman Steven, Greenland Sander. 2007. Assessing the unreliability of the medical literature: A response to “why most published research findings are false” (Feb 2007). Johns Hopkins University, Dept of biostatistics working papers. Working paper 125. http://biostats.bepress.com/jhubiostat/paper135
9. Goodman S, Greenland S. (2007) Why most published research findings are false: Problems in the analysis. PLoS Med 4(4):e168.doi:10.1371/journal.pmed/0040168
10. Wren JD. Truth, probability, and frameworks. PLoS Med. 2005 Nov;2(11):e361.
11. PLoS Medicine Editors (2005) Minimizing mistakes and embracing uncertainty. PLoS Med 2: e272. doi: 10.1371/journal.pmed.0020272.
12. Shrier I (2005) Power, reliability, and heterogeneous results. PLoS Med 2(11): e386.
13. Wren J (2005) Truth, probability, and frameworks. PLoS Med 2(11): e361.
14. Willett WC. The WHI joins MRFIT: a revealing look beneath the covers. Am J Clin Nutr. 2010;91(4):829-30.
15. Rose G, Hamilton PJ. A randomised controlled trial of the effect on middle-aged men of advice to stop smoking. J Epidemiol Community Health. 1978 Dec;32(4):275-81.
16. Lumley J, Chamberlain C, Dowswell T, Oliver S, Oakley L, Watson L. Interventions for promoting smoking cessation during pregnancy. Cochrane Database Syst Rev. 2009 Jul 8;(3):CD001055. doi: 10.1002/14651858.CD001055.pub3. Review.Update in: Cochrane Database Syst Rev. 2013;10:CD001055.
17. Howard BV, Van Horn L, Hsia J, Manson JE, Stefanick ML, Wassertheil-Smoller S et al. Low-fat dietary pattern and risk of cardiovascular disease: the Women's Health Initiative Randomized Controlled Dietary Modification Trial. JAMA. 2006 Feb 8;295(6):655-66.
18. Multiple Risk Factor Intervention Trial Research Group. Multiple Risk Factor Intervention Trial: risk factor changes and mortality results. JAMA1982;248:1465–77.
19. Martin LJ, Li Q, Melnichouk O, Greenberg C, Minkin S, Hislop G et al. A randomized trial of dietary intervention for breast cancer prevention. Cancer Res. 2011 Jan 1;71(1):123-33. doi: 10.1158/0008-5472.CAN-10-1436.
20. Prentice RL, Caan B, Chlebowski RT, Patterson R, Kuller LH, Ockene JK et al. Low-fat dietary pattern and risk of invasive breast cancer: the Women's Health Initiative Randomized Controlled Dietary Modification Trial. JAMA. 2006 Feb 8;295(6):629-42.
21. Byrne C, Rockett H, Holmes MD. Dietary fat, fat subtypes, and breast cancer risk: lack of an association among postmenopausal women with no history of benign breast disease. Cancer Epidemiol Biomarkers Prev. 2002 Mar;11(3):261-5.
22. Smith-Warner SA, Spiegelman D, Adami HO, Beeson WL, van den Brandt PA, Folsom AR et al.Types of dietary fat and breast cancer: a pooled analysis of cohort studies. Int J Cancer. 2001 Jun 1;92(5):767-74.
23. Sacks FM, Katan M. Randomized clinical trials on the effects of dietary fat and carbohydrate on plasma lipoproteins and cardiovascular disease. Am J Med.2002 Dec 30;113 Suppl 9B:13S-24S.
24. Mozaffarian D, Micha R, Wallace S. Effects on coronary heart disease of increasing polyunsaturated fat in place of saturated fat: a systematic review and meta-analysis of randomized controlled trials. PLoS Med. 2010;7(3):e1000252.
25. Jakobsen MU, O'Reilly EJ, Heitmann BL, Pereira MA, Bälter K, Fraser GE et al. Major types of dietary fat and risk of coronary heart disease: a pooled analysis of 11 cohort studies. Am J Clin Nutr. 2009 May;89(5):1425-32. doi: 10.3945/ajcn.2008.27124.
26. Estruch R, Ros E, Salas-Salvadó J, Covas MI, Corella D, Arós F et al; PREDIMED Study Investigators. Primary prevention of cardiovascular disease with a Mediterranean diet. N Engl J Med. 2013 Apr 4;368(14):1279-90. doi: 10.1056/NEJMoa1200303.
27. Sofi F, Abbate R, Gensini GF, Casini A. Accruing evidence on benefits of adherence to the Mediterranean diet on health: an updated systematic review and meta-analysis. Am J Clin Nutr. 2010 Nov;92(5):1189-96. doi: 10.3945/ajcn.2010.29673.
28. Shai I, Schwarzfuchs D, Henkin Y, Shahar DR, Witkow S, Greenberg I et al. Dietary Intervention Randomized Controlled Trial (DIRECT) Group. Weight loss with a low-carbohydrate, Mediterranean, or low-fat diet. N Engl J Med. 2008 Jul 17;359(3):229-41. doi: 10.1056/NEJMoa0708681.Erratum in: N Engl J Med. 2009 Dec 31;361(27):2681.
29. Esposito K, Kastorini CM, Panagiotakos DB, Giugliano D. Mediterranean diet and weight loss: meta-analysis of randomized controlled trials. Metab Syndr Relat Disord. 2011;9(1):1-12.
30. Martínez-González MA, García-Arellano A, Toledo E, Salas-Salvadó J, Buil-Cosiales P, Corella D et al; PREDIMED Study Investigators. A 14-item Mediterranean diet assessment tool and obesity indexes among high-risk subjects: the PREDIMED trial. PLoS One. 2012;7(8):e43134.
31. Malik VS, Pan A, Willett WC, Hu FB. Sugar-sweetened beverages and weight gain in children and adults: a systematic review and meta-analysis. Am J Clin Nutr.2013;98(4):1084-102.
32. Malik VS, Popkin BM, Bray GA, Després JP, Willett WC, Hu FB. Sugar-sweetened beverages and risk of metabolic syndrome and type 2 diabetes: a meta-analysis.Diabetes Care. 2010;33(11):2477-83.
33. Perez-Pozo SE, Schold J, Nakagawa T, Sánchez-Lozada LG, Johnson RJ, Lillo JL. Excessive fructose intake induces the features of metabolic syndrome in healthy adult men: role of uric acid in the hypertensive response. Int J Obes (Lond).2010;34(3):454-61.
34. Livesey G, Taylor R, Livesey H, Liu S. Is there a dose-response relation of dietary glycemic load to risk of type 2 diabetes? Meta-analysis of prospective cohort studies. Am J Clin Nutr. 2013;97(3):584-96.
35. Chiasson JL, Josse RG, Gomis R, Hanefeld M, Karasik A, Laakso M; STOP-NIDDM Trail Research Group. Acarbose for prevention of type 2 diabetes mellitus: the STOP-NIDDM randomised trial. Lancet. 2002 Jun 15;359(9323):2072-7.
36. Kelly S, Frost G, Whittaker V, Summerbell C. Low glycaemic index diets for coronary heart disease. Cochrane Database Syst Rev. 2004;(4):CD004467.
37. Mirrahimi A, de Souza RJ, Chiavaroli L, Sievenpiper JL, Beyene J, Hanley AJ et al. Associations of glycemic index and load with coronary heart disease events: a systematic review and meta-analysis of prospective cohorts. J Am Heart Assoc. 2012;1(5):e000752.doi:10.1161/JAHA.112.000752.
38. Gu D, Zhao Q, Chen J, Chen JC, Huang J, Bazzano LA et al. Reproducibility of blood pressure responses to dietary sodium and potassium interventions: the GenSalt study.Hypertension. 2013;62(3):499-505.
39. Sacks FM, Willett WC, Smith A, Brown LE, Rosner B, Moore TJ. Effect on blood pressure of potassium, calcium, and magnesium in women with low habitual intake. Hypertension. 1998 Jan;31(1):131-8.
40. Ascherio A, Hennekens C, Willett WC, Sacks F, Rosner B, Manson J et al. Prospective study of nutritional factors, blood pressure, and hypertension among US women. Hypertension. 1996 May;27(5):1065-72.
41. Aburto NJ, Hanson S, Gutierrez H, Hooper L, Elliott P, Cappuccio FP. Effect of increased potassium intake on cardiovascular risk factors and disease: systematic review and meta-analyses. BMJ. 2013;346:f1378. doi: 10.1136/bmj.f1378.
42. Giovannucci E, Stampfer MJ, Colditz GA, Hunter DJ, Fuchs C, Rosner BA et al. Multivitamin use, folate, and colon cancer in women in the Nurses' Health Study. Ann Intern Med. 1998;129(7):517-24.
43. Gaziano JM, Sesso HD, Christen WG, Bubes V, Smith JP, MacFadyen J et al. Multivitamins in the prevention of cancer in men:the Physicians' Health Study II randomized controlled trial. JAMA. 2012;308(18):1871-80.
44. Vartiainen E, Jousilahti P, Alfthan G, Sundvall J, Pietinen P, Puska P. Cardiovascular risk factor changes in Finland, 1972-1997. Int J Epidemiol. 2000;29(1):49-56.
45. Keys AC, Aravanis H, Blackburn R et al. Seven countries. A multivariate analysis of death and coronary heart disease. 1980. Cambridge: Harvard University Press.
46. Dawber TR, Meadors GF, Moore FE, Jr.: Epidemiological approaches to heart disease: the Framingham Study. Am J Public Health 1951; 41(3):279-286.
47. Luepker RV, Murray DM, Jacobs DR Jr, Mittelmark MB, Bracht N, Carlaw R et al. Community education for cardiovascular disease prevention: risk factor changes in the Minnesota Heart Health Program. Am J Public Health. 1994;84(9):1383-93.
48. Mozaffarian D, Katan MB, Ascherio A, Stampfer MJ, Willett WC (2006). "Trans Fatty Acids and Cardiovascular Disease". New England Journal of Medicine 354 (15): 1601–1613.
49. Mozaffarian D, Aro A, Willett WC. Health effects of trans-fatty acids:experimental and observational evidence. Eur J Clin Nutr. 2009 May;63 Suppl2:S5-21. doi: 10.1038/sj.ejcn.1602973.
Competing interests: No competing interests
I agree with Wiseman and Jackcson that the World Cancer Research Fund (WCRF)/American Institute for Cancer Research efforts are worthy initiatives. However, as they themselves acknowledge publication bias can distort the available information. For small effects, bias in a fragmented literature can wreak havoc even to the most comprehensive systematic efforts by the very best scientists. I don’t agree that it is much more difficult to interpret negative results in randomized controlled trials (RCTs) of nutrition than positive results. Given that the prior chances of single nutrients being highly effective are slim, a negative result in a well-designed RCT practically settles the matter. A positive result, conversely, may still be tenuous, especially when it comes from exploratory, secondary analyses, even in an otherwise well-designed RCT. Wiseman and Jackson apparently believe that observational epidemiology is so complex that the hypotheses that they generate cannot be refuted by RCTs. This surrounds observational epidemiology with some sort of sacrosanct aura. Conversely, I think that observational epidemiology is valuable and it can generate interesting hypotheses that are interesting precisely because they can be tested, validated, and refuted. If observational epidemiology created hypotheses that were untouchable, I would not be interested in it at all. I fully agree with Wiseman and Jackson that mechanistic evidence is very interesting and I applaud their efforts to standardize and systematize it. However, biases in this evidence are sometimes even stronger than those of observational epidemiology. Finally, I am extremely supportive of the need to tackle systematically the entire exposurome (1,2) rather than single nutrients. Eventually, I have no problem with making decisions and acting when we only know 1% of what we could know, but it is good to acknowledge what we don’t know.
Fraser suggests that tentative conclusions that important effects do exist for some diets (e.g. Mediterranean diet with virgin olive oil or nuts) seem more prudent than a dismissal. My intention was not to dismiss these effects. Conversely, I did say that I do believe that there is some beneficial effect and this could also have major public health implications. However, I do dismiss some very large effects that are occasionally claimed, simply because they don’t satisfy common sense criteria. E.g. if a serving of nuts per day decreased mortality by half, then I would expect to live for 140 years, simply because I do love nuts, but I don’t have such expectations. Fraser also mentions his experience with American Adventist vegetarians adding that “these results seem un-confounded by other non-dietary factors”. Personally, I cannot make this statement about lack of confounding for any observational study that I have been involved in. I cannot dismiss confounding when typically our studies in nutritional epidemiology capture only the minority of the risk variance, even when all measured variables are carefully considered. Fraser also lists a number of caveats and real-life difficulties about designing and running long-term RCTs. I don’t want to diminish the challenges in designing useful mega-trials, however, if as Fraser says “we do not know how to effectively intervene on such a complex behavior for such a long period”, then we should acknowledge that whatever we learn from observational nutritional epidemiology currently is useless, since we won’t be able to implement it in the long-term anyhow. I don’t share this viewpoint. Conversely, I believe that research should shift focus on understanding how to implement such long-term changes and interventions, rather than simply continuing with more of the same. Finally, I disagree with Fraser that consistent results across observational studies will certainly suggest causal effects. Consistent effects may indeed demonstrate causality in some cases. However, in other cases, they may just reflect a literature that is written, peer-reviewed, and edited by fervent believers who will not accept any other result other than what perpetuates their beliefs.
John P.A. Ioannidis, MD, DSc
Professor and Director, Stanford Prevention Research Center, Stanford, CA
E-mail: jioannid@stanford.edu
References:
1. Tzoulaki I, Patel CJ, Okamura T, Chan Q, Brown IJ, Miura K, Ueshima H, Zhao L, Van Horn L, Daviglus ML, Stamler J, Butte AJ, Ioannidis JP, Elliott P. A nutrient-wide association study on blood pressure. Circulation 2012;126(21):2456-64.
2. Ioannidis JP, Loy EY, Poulton R, Chia KS. Researching genetic versus nongenetic determinants of disease: a comparison and proposed unification. Sci Transl Med 2009;1:7ps8.
Competing interests: No competing interests
Dear Editors,
We read with great interest the article by Dr Ioannidis. He describes accurately many of the difficulties and disappointments of research results when trying to causally connect nutrition to risk of chronic diseases. Despite that, we believe that some of his conclusions may be overstated and premature. He may be correct that strong effects are improbable for many individual nutrients, a focus of much previous work. However, important effects for some foods/food groups are still entirely possible. On theoretical grounds, important effects on risk may be present and yet be very difficult to consistently detect with commonly used analytic approaches. Measurement errors are often large, and when statistical models have many covariates measured with error, residual confounding may often seriously bias effect estimates, usually toward to null.(1,2) Another issue is that care is needed with specific interpretation of effects from observational studies given that any exposure may be confounded by unmeasured variables that are tightly linked to it. Insufficient attention to these issues may largely explain the inconsistent results that Ioannidis highlights. We agree with him that large studies address only precision and not bias.
Nevertheless, observational studies have produced many results with sufficient consistency to allow consensus labels of “probable” referring to causal associations. For instance, there is in our opinion quite strong evidence from observational studies that red meats (especially processed meats) increase the risk of colorectal cancer, that nut consumption reduces risk of cardiovascular disease, and that some classes of fatty acids also affect risk of cardiovascular disease. The few intervention trials of nuts or Mediterranean diets may have produced some over-optimistic results, as stated by Ioannidis, yet these observational and trial results should not be dismissed so quickly. Given the largely consistent results from observational studies, despite the impediments mentioned above, and the supportive results from trials in selected populations, tentative conclusions that important effects do exist seem more prudent than a dismissal.
As another example of interest, our observational findings showing that American Adventist vegetarians have lower risk than their non-vegetarian counterparts for total mortality, cardiovascular disease, diabetes, and certain cancers are now consistent across two large American cohorts.(3-6) These results seem un-confounded by other non-dietary factors that we can identify. Although studies of dietary patterns do not identify specific dietary components that influence risk, they imply that such factors probably exist. It is a strong assumption that effects of dietary patterns result only from the sum of many very small and individually unidentifiable effects. In our view it is more likely that among these effects there will be some that stand out and be of greater interest. To conclude prematurely that diet has only small effects on risks of chronic disease would be a serious public health failure given the low cost of such preventive measures and the very low risk of side effects.
Although there are acknowledged problems with observational studies, in our view they are still our best hope to advance our understanding. Further efforts to minimize measurement errors and their effects (e.g. repeated dietary assessments perhaps using hand-held electronic technology, use of biomarkers in regression calibration etc) and the use of newer methods that improve control of confounding are ways forward. The mega-trials that Ioannidis proposes promise to be exceedingly expensive if used to investigate dietary effects on risk of cancer, for instance, and very likely infeasible. The intervention would need to last for a minimum of 10 years. We do not know how to effectively intervene on such a complex behavior for such a long period. Attempts to succeed would likely involve such highly selected subjects, perhaps with respect to socioeconomic status and baseline health, that broader application of results may be difficult. Even if it were possible to effectively intervene for at least a decade, the intervention would always be “unnatural” and the natural motivations for the diet and behavioral and sociologic associations with the diet, would be missing. Natural populations will often have subscribed to an approximately consistent dietary pattern for many decades, if not the whole of life. Another serious limitation of intervention studies is that only one (or perhaps two) interventions can be investigated, whereas observational studies can evaluate hypotheses about many potential causes.
In summary, it seems to us that observational studies continue to have great value. Although they will not individually produce “conclusive” results, mainly consistent results across such studies will certainly suggest causal effects. These should be followed by basic science investigations to see whether mechanisms can or cannot be found to support a causal hypothesis. Intervention trials may or may not be possible depending on the complexity of the necessary dietary intervention and its required duration. Trials would be more valuable if they are large enough to incorporate a naturally diverse population of subjects while retaining adequate power. At present this does not seem realistic.
References
1. Day NE, Wong MY, Bingham S, Khaw KT, Luben R, Michels KB et al. Correlated measurement error—Implications for nutritional epidemiology. Int J Epidemiol2004; 33:1373-81.
2. Schatzkin A, Kipnis V, Carroll RJ, Midthune D, Subar AF, Bingham S, Schoeller DA, Troiano RP, Freedman LS. A comparison of a food frequency questionnaire with a 24-hour recall for use in an epidemiological cohort study: results from the biomarker-based Observing Protein and Energy Nutrition (OPEN) study. Int J Epidemiol. 2003 Dec;32(6):1054-62.
3. Fraser GE. Diet, life expectancy and chronic disease. Studies of the health of vegetarians. Oxford University Press, New York, 2003.
4. Tonstad S, Stewart K, Oda K, Batech M, Herring RP, Fraser GE. Vegetarian diets and incidence of diabetes in the Adventist Health Study-2. Nutr Metab Cardiovasc Dis [Internet]. 2013 Apr;23(4):292–9. Available from: http://dx.doi.org/10.1016/j.numecd.2011.07.004
5. Tantamango-Bartley Y, Jaceldo-Siegl K, Fan J, Fraser GE. Vegetarian diets and the incidence of cancer in a low-risk population. Cancer Epidemiology Biomarkers & Prevention. 2013 Feb;22(2):286–94.
6. Orlich MJ, Singh PN, Sabaté J, Jaceldo-Siegl K, Fan J, Knutsen S, Beeson WL, Fraser GE. Vegetarian dietary patterns and mortality in Adventist Health Study 2. JAMA Intern Med. 2013 Jul 8;173(13):1230-8. doi: 10.1001/jamainternmed.2013.6473. PMID:23836264 [PubMed - indexed for MEDLINE]
Competing interests: No competing interests
INTERPRETING NUTRITION RESEARCH – MORE COMPLEX THAN APPARENT PLAUSIBILITY
RESPONSE TO IOANNIDIS
Martin J Wiseman FRCP FRCPath FAfN (Corresponding author)
Visiting Professor of Human Nutrition
Alan A Jackson CBE MD FRCP FRCPath FRCPH FAfN
Professor of Human Nutrition
NIHR Biomedical Research Centre in Nutrition
Southampton General Hospital (MP 113)
Tremona Road
Southampton SO16 6YD, UK
There is much to applaud in Ioannidis’s plea for a more rigorous interpretation of nutrition research (1). However there are four areas that he fails to address adequately. As in all medicine it is important to be able to take decisions in the face of limited evidence. Furthermore, while biomedical evidence is necessary, is unlikely to be sufficient to address the challenges posed by complex problems rooted in sociocultural experience and behaviour – such as nutrition.
First, it is easy to finds among the extensive literature single studies that report almost any finding. The question is whether those findings represent the body of evidence. The problem has been addressed by World Cancer Research Fund (WCRF)/ American Institute for Cancer Research 2007 Expert Report (2) and continues to be a matter for specific consideration within the Continuous Update Project (3). This is a thorough and exhaustive review of the world literature as it relates to diet, nutrition, physical activity and cancer, and is founded on the need to view all results in the context of a synthesis of all the evidence available. Even with such a comprehensive approach, publication bias can distort the available information.
Second, the nature of the epidemiology is to throw up several associations, only some of which are selected for reporting. Ioannidis draws attention to the findings for specific nutrients, and though he alludes to the issue, does not emphasise sufficiently that these reported findings can only be interpreted as markers of a much more complex set of exposures (the “exposome”) which include, but cannot be limited to diet (4). Equally the nature of nutrition is far more complex than the linear pseudo-pharmacological model implied by Ioannidis. The naïve interpretation of such findings as representing a simple causal pathway rather than a component of and marker for an aspect of a complex integrated system has led to randomised controlled trials (RCTs) testing nutrient supplements, out of context. Unsurprisingly these trials have mostly not proved positive (and some have shown adverse effects) (5); while positive results from RCTs can demonstrate efficacy (with causality needing to be inferred) it is much more difficult to interpret negative results. A RCT may be negative for a variety of reasons other than the overall lack of efficacy of the intervention. It may have been tested on an atypical population (high risk) using atypical exposures (nutrient supplements at high dose) at the wrong time (late in disease aetiology) or for an insufficient period (weeks or months rather than years or decades).
This difficulty with RCTs highlights the third area, where Ioannidis implies that hypotheses generated from observational studies can be refuted by negative RCTs. For the reasons stated above this is fallacious. The hypotheses generated by observational studies are more complex, and difficult to address, than is possible in RCTs. Therefore RCTs tend to address hypotheses that are simplifications of those posed by the epidemiology and selected as being amenable to an RCT. All types of evidence have advantages and disadvantages, and the results always need to be incorporated into an overall synthesis which recognises the broader issues.
Finally it is important to understand the mechanistic underpinning of any observed associations. It is easy to find a mechanistic study that reports results that can be used to describe a plausible biological explanation, but even more than for epidemiology, the preclinical literature presents challenges of interpretation and is prone to publication bias. Interpretation of the mechanistic literature is particularly challenging because the experimental conditions for studies using animal models or in vitro systems are rarely identical to those already reported, so that the reproducibility of particular findings is often speculative. Further, the validity of specific animal models or in vitro systems to human metabolism is also often unclear. These issues (6) have been the reason for WCRF International to commission work to enable a systematic review of the mechanistic evidence linking nutritional exposures to cancer.
A major driver, or purpose, of nutrition research should be to improve health or management of disease. Ioannidis goes some way to helping that but only offers an incomplete interpretation. In clinical practice, we always have uncertainties in how we manage our patients but despite this, are not paralysed but use the art of professional practice to act in their best interests. This approach is equally valid in the field of public health. Bradford Hill in 1965 (7) set out his framework for inferring causality from observational studies. This was not a dry academic exercise, as it is sometimes portrayed, but had at its heart the need – when presented with a problem where RCTs were not plausible to conduct – to decide what action to take, on the best evidence available, however flawed. It may be that the current literature appears to overstate the impact of single nutrients, but the ecological variations in geographic distribution of disease and their plasticity with time suggest that the whole ”exposome” of nutrition and other environmental determinants is in fact not over- but underestimated by a naïve interpretation of the published literature.
Competing interests:
All authors have completed the Unified Competing Interest form, and have no interests to declare.
MJW is Medical and Scientific Adviser to WCRF International and was Project Director for the WCRF/AICR 2007 Expert Report.
AAJ was a member of the independent Expert Panel for the 2007 Expert Report and is Chair of the independent Expert Panel for the Continuous Update Project
References
1. Ioannidis J. Implausible results in human nutrition research. BMJ 2013;347:f6698
2. WCRF/AICR. Food, Nutrition, Physical Activity, and the Prevention of Cancer: a Global Perspective. Washington DC: AICR, 2007
3. WCRF. 3ww.wcrf.org/cancer_research/cup/ (accessed 3 December 2013)
4. Wild C P. The exposome: from concept to utility. Int J Epi 2012;41:24–32
5. Virtamo J, Pietinen P, Huttunen J K et al. Incidence of cancer and mortality following alpha-tocopherol and beta-carotene supplementation: a postintervention follow-up. JAMA 2003;290:476-85
6. Begley C G and Ellis L M. Raise standards for preclinical cancer research. Nature 2012;483:531-2
7. Hill A B. The environment and disease: association or causation? Proc R Soc Med 1965;58:295-300
Competing interests: No competing interests
John Ioannidis wishes that all observational and small trial claims from nutritional research would stop. Were they to do so, the Daily Express would lose about one quarter of its front page headlines: Two portions of veg wards off Alzheimer's, Secret of avoiding heart disease, One glass of wine prolongs life. If those headlines have not actually appeared, I'm sure they will one day.
Competing interests: No competing interests
The massive problem of implausible results is not limited to nutritional epidemiology [1]. It concerns the whole field of observational epidemiology [2].
One overlooked reason is that epidemiology has not yet moved beyond correlation. Indeed, our increasing ability to gather large amount of data and our growing confidence to conduct highly complex (multi-variable, multi-level, etc) analyses have let us think that we had solved issues of selection bias or of confounding in observational studies [3].
Having an explicit causal reasoning, notably thanks to recent developments in causal thinking [4, 5], may help observational epidemiology move beyond correlation.
1. Ioannidis JPA. Implausible results in human nutrition research. BMJ 2013; 347:f6698
2. Taubes G. Epidemiolgy faces its limits. Science 1995; 269:164-9.
3. Chiolero A. Big data in epidemiology: too big to fail? Epidemiology 2013; 24(6):938-9.
4. Hernán MA, Hernández-Díaz S, Werler MM, Mitchell AA. Causal knowledge as a prerequisite for confounding evaluation: an application to birth defects epidemiology. Am J Epidemiol 2002; 155(2):176-84.
5. Rubin DB. The design versus the analysis of observational studies for causal effects: parallels with the design of randomized trials. Stat Med 2007; 26(1):20-36.
Competing interests: No competing interests
Re: Implausible results in human nutrition research
I greatly appreciate the constructive criticism of Li et al. which unfortunately came to my attention only recently. Their team comprises top thought leaders in this field and their comments reflect very nicely some of the major challenges and endless frustrations that we face in nutritional epidemiology.
Randomized trials do have several short-comings and I have repeatedly highlighted several of them (1-4), but this does not mean that they are not necessary to study effects of interventions. Li et al. mention two caveats which are not necessarily the major ones that I would be concerned of for randomized trials on nutrition. Imperfect long-term compliance may indeed be an issue in long-term trials. However, for a pragmatic trial this is not a disadvantage. What we are interested to find out is whether a nutrition-based or nutrition-related intervention works. If people cannot adhere to it for whatever reason, then it is not worth it. As an extreme example, obesity would disappear if caloric intake could decrease to zero and kept at zero, but of course this is impossible. Trials should ask for what is possible, what makes sense and what can have an impact on people rather than abstract etiological speculations and theories. Imprecise results due to early stopping because of significant results is also a concern. The solution again is pragmatic, i.e. performing large trials that are not stopped because of early significant results. Pragmatism includes the notion that one seeks a result that can be translated for real people, i.e. the size of the benefit must be significant for public health purposes, not just statistically significant.
The Harvard team very often uses the term “drug paradigm” to counter the arguments of people like myself who believe that there is more room for randomized trials in nutrition. This is an inaccurate characterization. The vast majority of drug trials are not pragmatic randomized trials (5). They are small, short-term, surrogate endpoint, efficacy trials. In fact, a review found only 9 pragmatic trials funded by the industry over 15 years (6) – as opposed to hundreds of thousands of “drug trials”. Drug trial-like designs could be used also for nutrition - with focused populations and short-term follow-up, compliance and trial retention is likely to be as good as that seen in the respective drug trials. But eventually the most informative trials in nutrition are likely to be long-term, pragmatic trials of strategies or complex interventions, and such trials are very rare in the drug trials world.
Li et al. mention a long-list of extra challenges in studying nutrients and their effects on health: trade-offs of nutrients or foods, difficulty to arrive at the optimal explanatory model (pathway, cumulative exposure, critical/sensitivity period models), heterogeneity due to study designs. I fully agree. But I think that these are reasons that should make people pose on whether exploratory observational analyses can ever reach a definitive answer on, given all these complexities. Randomized trials at least can tell us whether, allowing for all these complexities, there is something we can do that will benefit people.
The introduction of new food, new processing, new guidelines and new recommendations is yet another challenge that creates problems both for randomized trials and for observational epidemiology. Li et al. seem to assume that observational studies will do just fine and will be able to track changes in the strength and even direction of associations, as these superimposed factors are accumulating. At a minimum, these effects and their changes would have to be very large in order to emerge clearly from such complex, convoluted tracks. This may be the case on some occasions, but for most effects and interacting variables where even main effects are likely to be small (based on what we have seen so far), observational epidemiology may be operating at a range where it cannot separate effects from noise. If effects are tiny, randomized trials are not necessary and observational data dredging may be a waste of time. If they are small but meaningful for public health, randomized trials again would give us the best sense of whether they are worth doing something about.
I sympathize with the view of Li et al. that questionnaire methods and related methods to measure intake have improved over time. But even Li et al. agree that “questionnaires do have error”. While these tools may be good enough for studying large trends, I have concerns about their ability to study modest and small effects and/or replace experimental design safeguards. Better measurement tools are needed for any type of research, randomized or not, and the Harvard team should be congratulated for being a leader in this effort, for recognizing the deficiencies, and trying to find ways to correct them. However, in the meanwhile, for investigating small effect sizes (the majority of the effects seen for single nutrients), memory-based questionnaires can safely be considered a largely suboptimal, if not failed, tool.
I also agree that when it comes to “mega-trials”, bigger is not necessarily better. Sample size does not guarantee quality. However, if the effects that we want to capture are small, we need very large sample sizes. I do not share the interpretation by the Harvard team in this editorial or other papers (7) that large trials such as MRFIT or WHI (7) have been failures. Again, the real question is not whether one can study an association where everybody is 100% compliant and perfect. Mega-trials can study real effects. Conversely to the belief that observational datasets offer real-world effects, this is not necessarily the case, when the contrasts used are such that cannot be achieved by intervening on real people’s behaviors. From this perspective, observational associations can be practically totally unreal, as they try to extrapolate between-individual observations to within-individual effects.
I still believe that food security, sustainability, social inequalities, famine, and impact of food production on climate change (indicative references are (8-10)) may be more important problems that are worth tackling instead of adding more papers testing single nutrients or diets at an observational association level. How to address these problems is not always straightforward, but some of the effect sizes involved are very large and thus we could get reasonably reliable evidence to act on even with observational designs, or even with uncontrolled surveys sometimes. Then randomized trials may be needed to address focused, tailored questions, as needed. I would be thrilled to see the Harvard team use their leadership in the nutrition field in order to argue in favor of shifting more scientific effort and funding to these areas instead of defending a paradigm of nutrient-disease association that is offering very limited, barely incremental progress at this point. I fully agree with Li et al. that “nutrition is a multi-dimensional and multi-level rather than a single-dimensional research field” and that “many issues are directly related to the well-being of individuals and populations.”
1. Ioannidis JP. Clinical trials: what a waste. BMJ. 2014 Dec 10;349:g7089.
2. Ioannidis JP. Adverse events in randomized trials: neglected, restricted, distorted, and silenced. Arch Intern Med. 2009 Oct 26;169(19):1737-9.
3. Lathyris DN, Patsopoulos NA, Salanti G, Ioannidis JP. Industry sponsorship and selection of comparators in randomized clinical trials. Eur J Clin Invest. 2010 Feb;40(2):172-82.
4. Ioannidis JP. Some main problems eroding the credibility and relevance of randomized trials. Bull NYU Hosp Jt Dis. 2008;66(2):135-9.
5. Chalkidou K, Tunis S, Whicher D, Fowler R, Zwarenstein M. The role for pragmatic randomized controlled trials (pRCTs) in comparative effectiveness research. Clin Trials. 2012 Aug;9(4):436-46.
6. Buesching DP, Luce BR, Berger ML. The role of private industry in pragmatic comparative effectiveness trials. J Comp Eff Res. 2012 Mar;1(2):147-56.
7. Satija A, Yu E, Willett WC, Hu FB. Understanding nutritional epidemiology and its role in policy. Adv Nutr. 2015 Jan 15;6(1):5-18.
8. Godfray HC, Beddington JR, Crute IR, Haddad L, Lawrence D, Muir JF, Pretty J, Robinson S, Thomas SM, Toulmin C. Food security: the challenge of feeding 9 billion people. Science. 2010 Feb 12;327(5967):812-8.
9. Marmot M. Social determinants of health inequalities. Lancet. 2005 Mar 19-25;365(9464):1099-104.
10. McMichael AJ, Powles JW, Butler CD, Uauy R. Food, livestock production, energy, climate change, and health. Lancet. 2007 Oct 6;370(9594):1253-63.
Competing interests: No competing interests