Implausible results in human nutrition research
BMJ 2013; 347 doi: https://doi.org/10.1136/bmj.f6698 (Published 14 November 2013) Cite this as: BMJ 2013;347:f6698- John P A Ioannidis, professor of medicine, health research and policy, and statistics
- jioannid{at}stanford.edu
Research into human nutrition has been criticized on numerous occasions. Critics have focused on the poor track record of observational claims when tested in subsequent randomized trials (0/52 success rate in one review) and perpetuated fallacies.1 2 3 In contrast to major nutritional deficiencies and extreme cases, the effects of modest differences in nutrient intake have been difficult to study reliably at the population level. Nonetheless, some results, even of randomized trials, have been extremely promising.4 5 However, to establish a less controversial legacy for this important field, we should avoid past traps and be explicit about reasonable expectations. Implausible results that are “too good to be true” still threaten nutritional research on many fronts, including survey measurements, observational associations, treatment effects in randomized trials, and estimates of the impact on populations.
Nutritional intake is notoriously difficult to capture with the questionnaire methods used by most studies. A recent analysis showed that in the National Health and Nutrition Examination Survey, an otherwise superb study, for two thirds of the participants the energy intake measures inferred from the questionnaire are incompatible with life.6 More sophisticated measurements based on biochemical, web, camera, mobile, or sensor tools may not necessarily reduce bias.7 Caution about the reliability of measurements should extend to inferences that depend on them.
Almost every single nutrient imaginable has peer reviewed publications associating it with almost any outcome.8 On 25 October 2013, PubMed listed 34 291 papers with the keywords “coffee OR caffeine” and 12 741 with “soy,” many of which referred to associations. In this literature of epidemic proportions, how many results are correct?
Many findings are entirely implausible. Relative risks that suggest we can halve the burden of cancer with just a couple of servings a day of a single nutrient still circulate widely in peer reviewed journals.8 However, on the basis of dozens of randomized trials, single nutrients are unlikely to have relative risks less than 0.90 for major clinical outcomes when extreme tertiles of population intake are compared—most are greater than 0.95.9 For overall mortality, relative risks are typically greater than 0.995, if not entirely null. The respective absolute risk differences would be trivial. Observational studies and even randomized trials of single nutrients seem hopeless, with rare exceptions. Even minimal confounding or other biases create noise that exceeds any genuine effect. Big datasets just confer spurious precision status to noise.
Larger effect sizes are more plausible for complex dietary patterns that sum the effects of multiple nutrients and behaviors. Indeed, some randomized trials have shown interesting results. The Lyon Diet Heart study and recently the Primary Prevention of Cardiovascular Disease with a Mediterranean Diet (PREDIMED) trial showed 70% and 30% relative risk reductions, respectively,4 5 in composite clinical outcomes with Mediterranean diets. These effect sizes are probably greatly exaggerated. The early termination of these trials owing to statistically significant interim analyses inflates estimates of treatment effects.10 Other reasons for inflated effects include the selection of high risk populations (patients with heart disease and metabolic syndrome, respectively) and invalid comparator diets in control arms (in PREDIMED, 37% of energy came from fat in the “low fat” control arm, whereas low fat is defined as <10%). Inflated effects can also be caused by arm imbalances despite randomization and unavoidable unmasked designs that may affect ascertainment of clinical outcomes, such as stroke. PREDIMED data are also supporting a rapidly growing factory of secondary publications, many of which present grossly implausible observational claims—for example, eating more than three servings of nuts a week decreases overall mortality by 39%.11
Despite the hype, these randomized trials represent a major step forward. They offer hope that in the future we could identify nutrition related interventions that produce a 5-10% relative risk reduction in overall mortality in the general population, not just in high risk patients. However, such studies would require more than 10 times the sample size of PREDIMED (n=7447 participants and 348 recorded deaths), long term follow-up, linkage to death registries, and careful efforts to maximize adherence. Interventions may consider not only nutritional patterns, but also behavioral modifiers and other socioeconomic and built environment factors that affect lifestyle and adherence. Trial sponsoring and conduct should be free of conflicts that favor nutritional products or diets. Given that fanatical opinions abound in nutrition, allegiance bias should also be minimized.
According to the latest burden of disease study,12 26% of deaths and 14% of disability adjusted life years in the United States are attributed to dietary risk factors, even without counting the impact of obesity. No other risk factor comes anywhere close to diet in these calculations (not even tobacco and physical inactivity). I suspect this is yet another implausible result. It builds on risk estimates from the same data of largely implausible nutritional studies discussed above. Moreover, socioeconomic factors are not considered at all, although they may be at the root of health problems. Poor diet may partly be a correlate or one of several paths through which social factors operate on health.
Even if the impact of dietary risks is one tenth of that suggested by the burden of disease study,12 it still deserves attention. Definitive solutions will not come from another million observational papers or a few small randomized trials. Randomized trials are needed mainly to inform the design of pivotal mega-trials of comprehensive interventions. We should also continue to explore other aspects of food and nutrition—such as food security, sustainability, social inequalities, famine, and impact of food production on climate change—that may also affect human societies and wellbeing through multiple pathways. Food and nutrition may well make a major difference, but perhaps for reasons other than those that are usually touted, debated about, and contradicted.
Notes
Cite this as: BMJ 2013;347:f6698
Footnotes
Competing interests: I have read and understood the BMJ Group policy on declaration of interests and declare the following interests: I am a co-investigator in a randomized trial of a low carbohydrate versus low fat diet that is funded by the US National Institutes of Health and the non-profit Nutrition Science Initiative.
Provenance and peer review: Commissioned; not externally peer reviewed.