Effects of a low carbohydrate diet on energy expenditure during weight loss maintenance: randomized trialBMJ 2018; 363 doi: https://doi.org/10.1136/bmj.k4583 (Published 14 November 2018) Cite this as: BMJ 2018;363:k4583
All rapid responses
In the previous response, Mr. Moulthrop claims that he has identified a serious error in Dr. Hall's analysis. He suggests that because Dr. Ludwig's study "gives a highly significant result" and because it was the primary outcome … "random chance is very unlikely" and that Hall's reasoning for the result being random chance is not as compelling as Ludwig's interpretation.
There are several misconceptions here regarding P-values and hypothesis testing. The P-value is the probability of getting a test statistic at least as extreme as what was observed if every model assumption is correct . Some key assumptions include:
- Randomization (sampling, assignment)
- No uncontrolled sources of systematic error in the results
- That the test model (often a null model) is correct
When we make these assumptions (that are often not correct) we are engaging in a thought experiment, so that all results that deviate from what we expect under the model are thought to be a result of pure random error. These are assumptions, not truths. Again, by this logic, all results under these assumptions are via random error.
A highly significant result indicates a few things.
- A dimensional violation specified by the test hypothesis has been detected by the test.
- The data are not very compatible with the test model and test hypothesis.
- In a Neyman-Pearson framework, we are allowed to reject the test hypothesis based on a prespecified alpha.
What a highly significant result does not indicate:
- That the results are meaningful
- That random chance is unlikely
- That the test hypothesis is most likely to be false (significance can indicate a violation of assumptions, it doesn't tell us *which* assumption has been violated)
Furthermore, an inflated familywise error rate (which is important in a Neyman-Pearson framework) does not simply occur as a result of multiple comparisons and is not solved by specifying the primary endpoint in advance. There are several other factors that can inflate this error rate, where it is far more likely in the long run to produce a significant result that is a false positive [2,3].
Rather than fixate only on P-values and whether or not they are "highly significant", we can interpret observed p (the realization of the random variable P), as a continuous measure of compatibility and look to the width of the compatibility (confidence) interval to see what effects are compatible with the test model and its assumptions. Producing a likelihood function  and a P-value function [6,7] where every compatibility interval is graphed, will foster a much more nuanced interpretation of the data.
Trying to produce explanations as to why results gave "highly significant results" is highly misguided. Statistical significance and hypothesis testing have utility in certain areas where one is trying to automate decisions and control errors in the long run . This may be useful in areas where replication is feasible, such as psychology . In nutrition, where experiments similar to this are not always feasible, it may not hold as much utility.
The fixation on trying to explain highly significant results from a test of the null hypothesis truly is reflective of “cargo-cult statistics” .
1. Greenland S, Senn SJ, Rothman KJ, et al. Statistical tests, P values, confidence intervals, and power: a guide to misinterpretations. Eur J Epidemiol 2016;31:337–50. doi:10.1007/s10654-016-0149-3
2. Simmons JP, Nelson LD, Simonsohn U. False-positive psychology: undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychol Sci 2011;22:1359–66. doi:10.1177/0956797611417632
3. Wicherts JM, Veldkamp CLS, Augusteijn HEM, et al. Degrees of Freedom in Planning, Running, Analyzing, and Reporting Psychological Studies: A Checklist to Avoid p-Hacking. Front Psychol 2016;7:1832. doi:10.3389/fpsyg.2016.01832
4. Greenland S. Valid P-values behave exactly as they should: Some misleading criticisms of P-values and their resolution with S-values. Am Stat 2018;18.
5. Royall R. Statistical evidence: a likelihood paradigm. Routledge 2017. https://www.taylorfrancis.com/books/9781351414562
6. Poole C. Beyond the confidence interval. Am J Public Health 1987;77:195–9.https://www.ncbi.nlm.nih.gov/pubmed/3799860
7. Rothman KJ, Greenland S, Lash TL, et al. Modern epidemiology. Published Online First: 2008.https://www.annemergmed.com/article/S0196-0644(08)01394-2/abstract
8. Lehmann EL, Romano JP. Testing Statistical Hypotheses. Springer Science & Business Media 2006. https://market.android.com/details?id=book-K6t5qn-SEp8C
9. Lakens D, Adolfi FG, Albers CJ, et al. Justify your alpha. Nature Human Behaviour 2018;2:168–71. doi:10.1038/s41562-018-0311-x
10. Stark PB, Saltelli A. Cargo-cult statistics and scientific crisis. Significance 2018;15:40–3. doi:10.1111/j.1740-9713.2018.01174.x
Competing interests: No competing interests
Re: Effects of a low carbohydrate diet on energy expenditure during weight loss maintenance: randomized trial
I have read the responses of Hall et al. and the replies of Ludwig et al. Admittedly I have no special expertise in these subjects, but I believe I have identified a serious flaw in Dr. Hall’s arguments.
He says that his analysis is correct, while Dr. Ludwig’s is incorrect. But Ludwig’s analysis gives a highly significant result. Dr. Hall, if he is right, should be able to explain why Ludwig’s analysis gives a highly significant result. The only explanation implied by Hall’s arguments is that it is random chance. We know “p-hacking” can occur if there are many possible analyses. Then one can choose the analysis that gives a significant result by random chance. This is clearly not the situation here, since this is the primary endpoint, and there only a small number of possible analyses.
Hence, since random chance is very unlikely, there should be a reason that Dr. Ludwig’s analysis gives a significant result. Dr. Ludwig has a compelling explanation for this. Dr. Hall, if he is to be convincing, should give an explanation for the significant result. The non-significant result of Hall is explained plausibly by both Hall and Ludwig, so that result does not distinguish who is correct. The significant result is explained much more plausibly by Dr. Ludwig, so it is compelling evidence that Ludwig is correct and Hall is wrong.
Competing interests: No competing interests
We would like to clarify procedures related to data acquisition and maintenance of the study group assignment masking (blind).
The final analysis plan, with a change in specification of the baseline for total energy expenditure (TEE, the primary outcome) was approved by the institutional review board and posted in September 2017, prior to unmasking diet group assignment, as stated in the main manuscript.
At that time, the final outcome data for the third and main cohort (comprising 50% of participants) had not yet been received by the Boston investigators. However, data for the prior cohorts had been received.
Therefore, the relevant sentence in the online Supplement (page 16) would more accurately include the word “complete” as follows (1): “Submitted Final Data Analysis Plan (version 2017.06.14) to IRB and obtained approval prior to receiving the complete primary outcome data and breaking the randomization blind."
Dr. William Wong in Houston sent raw isotopic data periodically in batches to Boston during the study. Data cleaning and modeling (i.e., curve fitting to allow for calculation of the primary outcome) were performed in late 2017 after posting the final analysis plan. The statistician broke the blind and delivered the initial results to the principal investigators in January 2018. The statistician maintained the only depository for the primary outcome until posting the full database with publication of the manuscript in November 2018.
1. Supplemental Information (online): https://www.bmj.com/content/bmj/suppl/2018/11/13/bmj.k4583.DC1/ebbc04619...
Competing interests: As detailed in manuscript
Hall and Guo posted a reanalysis of our study using the pre-weight loss rather than the post-weight loss measure of total energy expenditure, finding a non-significant effect (1). They propose that the pre-weight loss baseline is more “correct” and raise concern about the timing of our change to the clinical trials registry.
This alternative analysis is not new. We performed this calculation during manuscript revision, as publicly available in the Peer Review material (2). The BMJ editors and statistician had full access to these data and we deferred to them regarding whether to include this analysis in the manuscript. We did not “refuse” to do so.
The reasons for using the post-weight loss baseline were extensively discussed in the Peer Review material and in the manuscript. (For reader convenience, we include the Peer Review response at bottom.) We were transparent about this a priori change, and included a detailed timeline in the Supplemental materials (3).
We further addressed this issue in a prior Rapid Response (4), showing that Hall’s previously stated concerns about changing body weight (5) were unfounded, and that statistical models including pre-weight loss baseline did not materially change the result.
There is a straightforward explanation for why the registry was corrected as the “7[th] of 8 versions of the protocol.” We planned to post a final analysis plan, with comprehensive detail of our statistical methods, prior to receiving the final data from our collaborator in Houston and breaking the blind. In preparing that plan under guidance of our statistician, we identified the misspecification – an inadvertent holdover from a prior cross-over study (our present study is parallel design, in which the weight-loss phase would introduce inter-individual variability) (6). Earlier protocol revisions addressed logistical and other issues in study conduct; we had no reason to review the analysis plan and identify the misspecification prior to that time. We are puzzled as to the relevance and implication of this point about protocol revision number.
Finally, we note that our study went through a highly rigorous review, involving 6 independent experts, including more than 120 points of criticism and discussion. We exceeded professional standards of transparency, by making the full database publicly available immediately upon publication – perhaps for the first time for a nutrition study of this magnitude and complexity.
1. Hall KD, Guo J. 28 November 2018. No significant effect of dietary carbohydrate versus fat on the reduction in total energy expenditure during Maintenance of lost weight. https://www.bmj.com/content/363/bmj.k4583/rr-16
2. Peer Review, Author Response, 18 September 2018, Point #1 pages 2-4: https://www.bmj.com/sites/default/files/attachments/bmj-article/pre-pub-...
3. Supplemental information, page 16: https://www.bmj.com/content/bmj/suppl/2018/11/13/bmj.k4583.DC1/ebbc04619...
4. Ludwig DS, Ebbeling, CB, Feldman HA. 20 November 2018. Choice of baseline for primary endpoint. https://www.bmj.com/content/363/bmj.k4583/rr-11
5. Belluz J, Vox, updated November 21, 2018. Does cutting carbs really help keep weight off? The big new diet study, explained. https://www.vox.com/science-and-health/2018/11/16/18096633/keto-low-carb...
6. Ebbeling CB, Swain JF, Feldman HA, Wong WW, Hachey DL, Garcia-Lago E, Ludwig DS. Effects of dietary composition on energy expenditure during weight-loss maintenance. JAMA. 2012, 307:2627-34 https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3564212/
POINT #1 FROM PEER REVIEW, 18 SEPTEMBER 2018:
We closely adhered to the a priori clinical trials and protocol plans, also as detailed in our methods paper and Open Science Framework registry (https://osf.io/rvbuy/).
The only discrepancy involves a change from initial specification in the anchor used to calculate change in the primary outcome. In our original analysis plan of 2014, we had indicated the preweight loss (i.e., pre-Run-In, BSL in Figure 1) measurement as the anchor for determining the diet effect on total energy expenditure (TEE), but this was an error on our part. We corrected this error in the Clinical Trials registry and used an analysis for our manuscript with the post-weight loss (i.e., post-Run-In, PWL in Figure 1) measurement as the anchor. For the reasons explained below, we request an exception to your general rule of including both analyses in the Results section, and instead have provided further clarification of this issue in Methods.
1. The initial listing was clearly an error:
A) As a general rule, anchor data should be collected as close to randomization as possible, to decrease error introduced by time-varying covariates. The pre-Run-In measurement involves a 3- to 4-month delay prior to initiation of the Test diets.
B) In addition to this delay, the pre-Run-In measurement is strongly confounded by weight loss, whereas the specific aim of the study is to examine TEE during weight maintenance. (Indeed, the title of the study in the registry is: Dietary Composition and Energy Expenditure During Weight-Loss Maintenance.)
C) The expressed purpose of the Run-In was to produce 12% weight loss, changing biological state (i.e., creating a predisposition to weight regain) to test the study hypotheses. Thus, it would be inconsistent with study aims and methodologically inappropriate to use the pre-Run-In time point to establish a precise and accurate anchor for determining how the Test diets change TEE. Doing so would necessitate a substantially larger number of participants (and cost) to account for the additional imprecision, with no scientific benefit.
2. Study power (and thus participant number) was determined with use of post-Run-In measurement as the anchor.
A) Our a priori power calculations defined the primary outcome as “change in total energy expenditure at week 20 of the test phase compared to week 0 (post-weight loss).”
B) We did not take into account the variability between pre-Run-In and post-Run-In (Week 0) measurements, which in our case had r-values on the order of only 0.3 for the unadjusted model and 0.5 for the adjusted model.
C) Not surprisingly, doing the analysis with the pre-Run-In measurement as the anchor yielded a mean estimate in the same direction, but with substantial loss of precision and a statistically non-significant overall effect in the ITT. For example, TEE in the unadjusted model of Low vs High Carbohydrate diets was + 141 kcal/day (p=0.08, overall p=0.2).
3. The error was recognized and corrected a priori. We obtained IRB approval for our final analysis plan on 06 Sept 2017, before the blind was broken (and indeed, before measurement of the primary outcome had been completed by our collaborator Bill Wong in Houston). Similarly, we corrected the Clinical Trials registry prior to breaking the blind. We provide documentation of this timeline, and additional detail, in the Supplement Protocol section.
Although we agree with the general policy of including multiple analyses where discrepancies exist, we think an exception would be warranted in this situation. For reasons suggested above, we believe that we have fulfilled the letter and spirit of an a priori analysis plan. Furthermore, we are concerned that the additional analysis would provide no meaningful biological insights – that is, no useful information about the nature of the relationship between dietary composition and energy expenditure. Rather, inclusion of the additional analysis would tend to elevate and give undue attention to an error, and therefore potentially cause confusion.
To place our study in the context of other diet trials, I reviewed the Clinical Trials registry of diet and weight loss trials published in one of the JAMA journals since 2015 (to obtain a collection of cross-specialty examples). Of the 13 trials identified, 8 had significant changes in the primary outcome since initial posting. In several additional trials, the level of detail for the primary outcome was insufficient to exclude multiple statistical treatments [NB, see online Peer Review for references to these studies].
Regarding this last point, our pre-analysis plan provided a comparatively great level of detail. For contrast, the study by Hall et al of 2016 (cited in the Endocrine Society Scientific Statement as major evidence against the Carbohydrate-Insulin Model) included only a 1- paragraph statistical plan (https://osf.io/9q8cu/ beginning bottom of page 23), thus providing freedom to analyze outcomes in many different ways (e.g., post hoc exclusion of outliers).
My intent in providing this context isn’t to justify suboptimal practices, but rather to clarify that error or lack of specificity in an initial registry of major diet trials (vs industry-sponsored drug trials which have much larger budgets, infrastructural support and standardization) is more the rule than the exception. However, we believe that our overall rigor is comparatively very high, and the proposed course of action entails no risk to the integrity of the data analyses.
To address your reasonable concerns on this point and to maintain maximum transparency, we have clarified this situation in the Methods (page 12, para 2) and provided additional detail in the Supplement Protocol section. Please also note that we have committed to post the complete data set on a publicly available server upon publication of the manuscript, so that anyone can perform additional exploratory analyses, including this one. Nevertheless, we will defer to you, and include the additional analysis, if you disagree with our proposed solution.
Competing interests: As detailed in the manuscript
No Significant Effect of Dietary Carbohydrate versus Fat on the Reduction in Total Energy Expenditure During Maintenance of Lost Weight
The question of whether the ratio of dietary carbohydrates to fat substantially impacts total energy expenditure (TEE) or body fat has been investigated for decades, with most studies pointing to no clinically meaningful effect 1. However, a recent study by Ebbeling et al. reported substantial differences in TEE between diets varying in their ratio of carbohydrate to fat 2.
The original pre-registered statistical analysis plan for the primary study outcome of Ebbeling et al. addressed the question of whether the reduction in TEE during weight loss maintenance compared to the pre-weight loss baseline depended on the dietary carbohydrate to fat ratio – a design similar to a previous study by many of the same authors 3. However, the final analysis plan was modified to make the diet comparisons with the TEE measurements collected in the immediate post-weight loss period rather than at the pre-weight loss baseline. As fully described in a manuscript available on the bioRxiv pre-print server (https://www.biorxiv.org/content/early/2018/11/28/476655), reanalyzing the data according to the original analysis plan of Ebbeling et al. found that the TEE differences were no longer statistically significant between the diet groups and the nominal diet differences of ~100 kcal/d were much smaller than the ~250 kcal/d differences reported in the final publication. In other words, when conducting the analysis originally planned by the authors we found that the significant increases in TEE with the low carbohydrate diet that were reported by Ebbeling et al. disappeared. Furthermore, the significant TEE effect modification by baseline insulin secretion also disappeared.
As justification for our reanalysis, we note that most of the history of the study (7 of 8 versions of the protocol spanning from 2014-2016) the planned primary outcome calculations used the pre-weight loss TEE baseline as the anchor point for the subsequent diet comparisons during weight loss maintenance. Prior to unmasking the randomization blind, but after all cohorts had completed the trial, the final protocol amendment in 2017 altered the previously planned statistical analysis to use the post-weight loss TEE measurement as the anchor point to make the subsequent diet comparisons.
The reasons for the change in the analysis plan were not provided in the protocol amendment or the final statistical analysis plan, but the Supplemental Materials in the final publication provided three reasons. First, the post-weight loss TEE measurement was chosen as the new anchor point because it occurred closer to the point of diet randomization. Second, the pre-weight loss TEE measurement would be “strongly confounded by weight loss”. How this might happen and why the post weight loss measure would not be similarly affected is difficult to imagine. Finally, Ebbeling et al. argued that the pre-weight loss baseline would have been inappropriate because the TEE measurements were insufficiently accurate or precise and therefore the study would be under-powered. However, the power calculations in the protocol were based on TEE data from a pilot study using the pre-weight loss TEE measurements as the basis for comparing how different diets affected the absolute reduction in TEE during weight loss maintenance 3. The pilot study did not measure TEE in the period immediately post-weight loss and therefore could not have been used to power the recent study in question.
Despite a request by the BMJ Editors to report the results of their original analysis plan, Ebbeling et al. refused because they were “concerned that the additional analysis would provide no meaningful biological insights – that is, no useful information about the nature of the relationship between dietary composition and energy expenditure. Rather, inclusion of the additional analysis would tend to elevate and give undue attention to an error, and therefore potentially cause confusion.”
We believe that the revised analysis plan has caused confusion and that the original statistical analysis plan that used pre-weight loss TEE as the anchor point is preferable for several reasons. First, it specifically addresses the question of whether the typical reduction in TEE that accompanies maintenance of lost weight depends on the carbohydrate to fat ratio of the weight loss maintenance diet. Second, the revised plan is potentially confounded by the substantial adaptive thermogenesis that occurs immediately post-weight loss that typically becomes less severe after a period of energy balance and weight loss maintenance 4 5. Finally, the pre-weight loss baseline TEE measurements were obtained in the situation where the doubly labeled water method is routinely employed: free-living people maintaining their habitual weight. Ideally, a post-weight loss TEE measurement should have first stabilized subjects at the lower body weight for several weeks prior to dosing with doubly labeled water 6. In contrast, the post-weight loss TEE measurements conducted by Ebbeling et al. were obtained during the same 2-week weight stabilization period when diet calories were being progressively increased at a rate determined by each individual subject’s recent rate of weight loss. While the doubly labeled water method generally provides a robust and valid estimate of TEE, this situation of simultaneous refeeding immediately post-weight loss potentially introduces uncertainty into the conversion of CO2 production into TEE. For example, the daily respiratory quotient during this period was clearly not equal to the food quotient as was assumed by Ebbeling et al. While such an effect can be appropriately modeled 7, this was not done in their TEE calculations.
In conclusion, when analyzed using the original statistical plan that was not confounded by the immediate post-weight loss period, the data of Ebbeling et al. do not support the conclusion that the ratio of dietary carbohydrate to fat affects the reduction in TEE during weight loss maintenance. While there are many reasons people could benefit from consuming healthy low carbohydrate diets 8, such diets are unlikely to help offset the usual reduction in TEE during maintenance of lost weight.
1. Hall KD, Guo J. Obesity Energetics: Body Weight Regulation and the Effects of Diet Composition. Gastroenterology 2017;152(7):1718-27 e3. doi: 10.1053/j.gastro.2017.01.052 [published Online First: 2017/02/15]
2. Ebbeling CB, Feldman HA, Klein GL, et al. Effects of a low carbohydrate diet on energy expenditure during weight loss maintenance: randomized trial. Bmj 2018;363:k4583. doi: 10.1136/bmj.k4583 [published Online First: 2018/11/16]
3. Ebbeling CB, Swain JF, Feldman HA, et al. Effects of dietary composition on energy expenditure during weight-loss maintenance. Jama 2012;307(24):2627-34.
4. Hall KD. Computational Modeling of Energy Metabolism and Body Composition Dynamics. In: Krentz AW, Heinemann L, Hompesch M, eds. Translational Research Methods for Diabetes, Obesity and Cardiometabolic Drug Development. London: Springer-Verlag 2015:265-82.
5. Weinsier RL, Nagy TR, Hunter GR, et al. Do adaptive changes in metabolic rate favor weight regain in weight-reduced individuals? An examination of the set-point theory. Am J Clin Nutr 2000;72(5):1088-94.
6. Bhutani S, Racine N, Shriver T, et al. Special Considerations for Measuring Energy Expenditure with Doubly Labeled Water under Atypical Conditions. Journal of obesity & weight loss therapy 2015;5(Suppl 5) doi: 10.4172/2165-7904.S5-002 [published Online First: 2016/03/11]
7. Hall KD, Guo J, Chen KY, et al. Methodologic Issues in Doubly Labeled Water Measurements of Energy Expenditure During Very Low-Carbohydrate Diets. bioRxiv 2018 doi: 10.1101/403931
8. Hall KD, Chung ST. Low-carbohydrate diets for the treatment of obesity and type 2 diabetes. Curr Opin Clin Nutr Metab Care 2018;21(4):308-12. doi: 10.1097/mco.0000000000000470 [published Online First: 2018/04/21]
Competing interests: KDH has participated in a series of debates with Dr. David S. Ludwig, the senior author of the main study in question, regarding the merits and demerits of the carbohydrate-insulin model of obesity as well as the physiological response of the human body to isocaloric diets varying in the ratio of carbohydrates to fat.
Scientific debates in nutrition may sometimes proceed along predictable lines, seemingly influenced by personal ethical views. Advocates of vegan diets tend to emphasize studies showing the benefits of consuming carbohydrate over fat, perhaps because virtually all dietary carbohydrates come from plants. (In a previous post, we responded to criticism of our study by 2 individuals with leadership roles in Physicians Committee for Responsible Medicine, a group that discourages use of animals for food and medical research (1).)
Conversely, advocates of low-carbohydrate diets commonly highlight the advantages of animal products, perhaps because meat, eggs and dairy products were historically primary sources of dietary fat.
However, these implicit biases do not reflect the reality of the modern food environment. One can easily consume a low-fat, animal-based diet with foods such as chicken breast, lean beef, egg whites, and low-fat cheese.
Alternatively, a low-carbohydrate diet can be based on high-fat plant foods including nuts, seeds, olive oil, avocado, coconut oil and dark chocolate – as exemplified by several recent popular books and a large social media group (2) that recommend a vegan ketogenic diet.
To find common ground, we must carefully distinguish scientific from ethical considerations. But with the abundant choice of foods now available for most people in developed countries, all sides of the diet debates can take comfort that scientific truth and the pursuit of an ethical lifestyle need not be in conflict.
1. Authors' response to Kahleova, Katz and Barnard. https://www.bmj.com/content/363/bmj.k4583/rr-12
2. Facebook Group, Vegan Keto Made Simple. https://www.facebook.com/groups/320351758396552/
Competing interests: As detailed in the manuscript
The details of the diet given in the methods paper:
make it clear that while this study has stuck to the official guidelines and recommendations for nutrient composition, there is still a problem here that should be addressed by all researchers in this field. According to the article:
"Total fiber content was consistent with recommendations from the Institute of Medicine (1) and reflected a gradient across the 3 diets (17.5, 15, and 12.5 g/1000 kcal with the high-, moderate-, and low-carbohydrate diets)."
Now, the argument for a low-fat high carb diet has its roots in observational studies done on indigenous populations, e.g. (2), but the sorts of diets that people in such studies ate, are planted based whole food diets containing small amounts of meat and fish that are very low in refined oils. Such diets have a fiber content of a factor of 2 to 3 higher than the current recommendation. Also such diets will contain much more magnesium, which is an element that plays an important role in metabolism, also about a factor of 2 to 3 more than current guidelines.
We can easily verify this by considering the fiber and magnesium contents of a list of whole foods and computing the amounts of these nutrients per Kcal of energy. E.g. potatoes contain 2.9 *10^(-2) g/Kcal of fiber and 0.3 mg/Kcal of magnesium. Looking at a large list of many such energy rich whole foods reveals that a 2500 Kcal carb based whole food diet should contain at least 0.7 grams of magnesium and at least 70 grams of fiber. Here we note that sources of dietary fat cannot be refined oils but can be foods like walnuts, almonds, linseed etc. which are also high in magnesium and fiber. Also vegetables have a much higher fiber to energy ratio than then carb-rich foods.
One can formulate this in the opposite way by asking what kind of whole food diet of a 2500 kcal would actually yield the RDAs of 400 mg of magnesium and 40 grams of fiber. One then finds that this is not possible, unless one includes large amounts of olives which contain an atypically low amount of fiber and magnesium per unit energy.
Then given that the human body should be assumed to have adapted itself to a diet that based on unrefined foods that can be obtained from Nature, we should have some doubts about the healthfulness of the much lower official recommendations for fiber and magnesium compared to that of a diet based on whole foods.
1. Institute of Medicine Dietary Reference Intakes for energy, carbohydrate, fiber, fat, fatty acids, cholesterol, protein, and amino acids. Washington (DC): National Academies Press; 2002.
2. Shaper AG, Jones KW. Serum-cholesterol, diet, and coronary heart-disease in Africans and Asians in Uganda. The Lancet, 534–537 (1959).
Competing interests: No competing interests
Many of the concerns raised by Kahleova, Katz and Barnard were addressed in the manuscript, but we will elaborate here.
1. We aimed to control as many factors as possible between treatment groups within the context of real-life diets (not liquid research formulas) over the course of a full academic year, including: the proportion of sugar to total carbohydrate, the proportion of saturated fat to total fat, the protein source, and non-starchy vegetables. While some dietary factors differed, as acknowledged in the Discussion section, the resulting diets reflect healthful representations of their respective macronutrient compositions. The high carbohydrate diet was in fact substantially higher in fiber than the lower carbohydrate diet, and exceeded prevailing recommendations on whole grain consumption. The diet effect on biomarkers, such as triglycerides, reflects the observed associations with total carbohydrate intake among a general population. Furthermore, the glycemic index of the high carbohydrate diet was quite low for a diet that must, in practice, include significant amounts of grains. Indeed, we avoided extreme manipulations of all groups – such as using an uncharacteristically low amount of saturated fat in the low carbohydrate group. A methods paper provides comprehensive details of dietary design, including our approach to foster treatment differentiation with conservative food substitutions (1).
2. The adjustment of energy intake was done to maintain weight stability after weight loss to test the primary hypothesis, as extensively discussed in our methods. Failure to control weight loss, as Kahleova et al seem to prefer, would severely confound our primary outcome, total energy expenditure.
3. We agree that spontaneous increases in physical activity level may mediate, to some degree, the diet effect, as hypothesized by the carbohydrate-insulin model (2, 3). (We provided no instruction to change physical activity level.) With the greater observed availability of metabolic fuels (4), participants may have felt more energetic. Interestingly, evidence suggests that physical activity level, like hunger, is under biological control (5).
4. We disagree that fat restriction is more effective for weight control, as demonstrated by multiple meta-analyses (6, 7, 8, 9, 10). We disagree that a low fat diet has benefit for longevity, as shown in a prospective 3-cohort study (11) and elsewhere. High fat foods like nuts, avocado, olive oil, full fat yogurt, fatty fish and dark chocolate are associated with protection against diabetes, heart disease and perhaps neurodegenerative diseases – but that debate is beyond the scope of this venue. We address the issue of translation to public health in the manuscript.
1. Julia MW Wong, Lisa Bielak, Ralph G Eddy, Lauren Stone, Paul R Lakin, Megan Sandman, Courtenay Devlin, Linda Seger-Shippee, Dina Wiroll, Patricia K Luoto, Gloria L Klein, David S Ludwig, Cara B Ebbeling. An academia-industry partnership for planning and executing a community-based feeding study. Curr Dev Nutr 2018; 2(9): nzy060.
2. Ludwig DS, Ebbeling CB. The Carbohydrate-Insulin Model of Obesity: Beyond “Calories In, Calories Out”. JAMA Intern Med 2018;178:1098-103
3. Ludwig DS, Friedman MI. Increasing adiposity: consequence or cause of overeating? JAMA 2014;311:2167-8
4. Walsh CO, Ebbeling CB, Swain JF, Markowitz RL, Feldman HA, Ludwig DS. Effects of diet composition on postprandial energy availability during weight loss maintenance. PLoS One 2013;8(3):e58172
5. Richmond RC, Davey Smith G, Ness AR, den Hoed M, McMahon G, Timpson NJ. Assessing causality in the association between child adiposity and physical activity levels: a Mendelian randomization analysis. PLoS Med 2014;18;11:e1001618.
6. Bueno NB, de Melo IS, de Oliveira SL, da Rocha Ataide T. Very-low-carbohydrate ketogenic diet v. low-fat diet for long-term weight loss: a meta-analysis of randomised controlled trials. Br J Nutr 2013;110:1178-87
7. Mancini JG, Filion KB, Atallah R, Eisenberg MJ. Systematic review of the mediterranean diet for long-term weight loss. Am J Med 2016;129:407-415
8. Mansoor N, Vinknes KJ, Veierød MB, Retterstøl K. Effects of low-carbohydrate diets v. low-fat diets on body weight and cardiovascular risk factors: a meta-analysis of randomised controlled trials. Br J Nutr 2016;115:466-79
9. Sackner-Bernstein J, Kanter D, Kaul S. Dietary intervention for overweight and obese adults: comparison of low-carbohydrate and low-fat diets. A meta-analysis. PLoS One 2015;10:e0139817
10. Tobias DK, Chen M, Manson JE, Ludwig DS, Willett W, Hu FB. Effect of low-fat diet interventions versus other diet interventions on long-term weight change in adults: a systematic review and meta-analysis. Lancet Diabetes Endocrinol 2015;3:968-79
11. Dong D. Wang, Yanping Li, Stephanie E. Chiuve, Meir J. Stampfer, JoAnn E. Manson, Eric B. Rimm, Walter C. Willett, Frank B. Hu. Association of specific dietary fats with total and cause-specific mortality. JAMA Intern Med 2016;176:1134-45
Competing interests: As detailed in the manuscript
We write to clarify selection of the baseline to determine the diet effect on total energy expenditure and other outcomes. As we discussed in the Supplement, we chose to measure baseline in the post-weight loss (not pre-weight loss) assessment period for several reasons:
1) As a general rule, anchor data should be collected as close to the time of randomization as possible, to decrease error introduced by any time-varying confounder. The pre-weight loss measurement was obtained 3 to 4 months prior to initiation of the test diets.
2) The pre-weight loss measurement is strongly confounded by weight loss, whereas our specific aim is to examine total energy expenditure during weight-loss maintenance (consistent with the registered title of the protocol, Dietary Composition and Energy Expenditure During Weight-Loss Maintenance).
3) The stated purpose of the Run-In phase is to produce 12% weight loss, changing biological state (i.e., creating a predisposition to weight regain) to test the study hypotheses. Thus, it would be inconsistent with study aims and methodologically inappropriate to use the pre-weight loss measurement as a precise and accurate anchor for evaluating change in total energy expenditure in response to the test diets. Doing so would necessitate a substantially larger number of participants (and increased cost) to account for the additional imprecision, with no scientific benefit.
Some have recently argued that there might be ongoing weight change during the post-weight loss period that could inflate our results (1). However, this measurement was done before randomization, so any individual variations in weight or other factors could not confound study findings. Although error in baseline measurement could reduce the precision of our effect estimates – a non-issue here, as we had abundant power to test the primary hypothesis – it would not create bias in favor of any one diet.
Furthermore, as can be determined from the publicly available database, weight change during the 15-day assessment period was very small, averaging 23 grams per day, with no significant difference between individuals who would be assigned to the 3 diet groups (p=0.2). This value compares favorably with Redman et al (2), who state that “an accepted criterion for weight stability in controlled feeding studies is fluctuations in weight <250 g/d.”
Finally, our fully adjusted model included “pre-weight loss values for BMI, percentage lean mass, and total energy expenditure; and weight loss from pre-weight loss to start of trial.” The results of this model were not materially different from those of the minimally adjusted model, indicating that these pre-randomization factors were unimportant.
1. Belluz J. Does cutting carbs really help keep weight off? The big new diet study, explained. Vox, 16 Nov 2018. https://www.vox.com/science-and-health/2018/11/16/18096633/keto-low-carb...
2. Redman LM, Kraus WE, Bhapkar M, Das SK, Racette SB, Martin CK, Fontana L, Wong WW, Roberts SB, Ravussin E; CALERIE Study Group. Energy requirements in nonobese men and women: results from CALERIE. Am J Clin Nutr 2014;99:71-8
Competing interests: As detailed in the manuscript
We would like to clarify two methodological issues in our study, relating to an inclusion criterion and creation of the Per Protocol group.
First, the weight loss inclusion criterion was 12 ± 2% of pre-weight loss body weight. Once a participant achieved 10 to 14% weight loss on two days during the Run-In phase (based on daily weights obtained using Wi-Fi scales), we considered her or him eligible for the randomized trial. At that point, we aimed to stabilize weight prior to beginning “Start” of trial assessments. The Start weight reported in the manuscript was based on average weight measurements during the doubly-labeled water protocol. There was some variation in weight between the time of formal inclusion and the Start assessment period. We reported the Start mean weight loss of 10.5% (SD 1.7). The numerical distribution was as follows:
≤ 8%: 8
> 8 to 9%: 24
>9 to <10%: 30
10 to 14%: 100
>14 to 15%: 0
>15 to 16%: 1
>16 %: 1
As with any baseline factor, randomization will protect against selective bias to study outcomes related to this variation. That is, pre-randomization differences in weight loss might decrease precision and thereby lower statistical power (which was not of concern for our primary outcome) but would not artifactually inflate the observed diet effect. Moreover, our fully adjusted model included Run-In weight loss as a covariate.
Second, our Per Protocol group comprised 120 participants whose weight did not deviate by more than 2 kg of individual Start weight at any post-randomization assessment period. Some of these individuals dropped out or were missing some outcome data for other reasons, but all available data for this group were included in the statistical analyses. Due to space limitations, the published Figure 2 did not provide full descriptive detail, but the corresponding figure in our final submitted manuscript makes this issue clearer (1).
1. Peer review content: First revised article. https://www.bmj.com/sites/default/files/attachments/bmj-article/pre-pub-...
Competing interests: As detailed in the manuscript