Author Response to Hall and Guo Regarding Data Reanalysis and Other Criticisms
Hall and Guo posted a reanalysis of our study using the pre-weight loss rather than the post-weight loss measure of total energy expenditure, finding a non-significant effect (1). They propose that the pre-weight loss baseline is more “correct” and raise concern about the timing of our change to the clinical trials registry.
This alternative analysis is not new. We performed this calculation during manuscript revision, as publicly available in the Peer Review material (2). The BMJ editors and statistician had full access to these data and we deferred to them regarding whether to include this analysis in the manuscript. We did not “refuse” to do so.
The reasons for using the post-weight loss baseline were extensively discussed in the Peer Review material and in the manuscript. (For reader convenience, we include the Peer Review response at bottom.) We were transparent about this a priori change, and included a detailed timeline in the Supplemental materials (3).
We further addressed this issue in a prior Rapid Response (4), showing that Hall’s previously stated concerns about changing body weight (5) were unfounded, and that statistical models including pre-weight loss baseline did not materially change the result.
There is a straightforward explanation for why the registry was corrected as the “7[th] of 8 versions of the protocol.” We planned to post a final analysis plan, with comprehensive detail of our statistical methods, prior to receiving the final data from our collaborator in Houston and breaking the blind. In preparing that plan under guidance of our statistician, we identified the misspecification – an inadvertent holdover from a prior cross-over study (our present study is parallel design, in which the weight-loss phase would introduce inter-individual variability) (6). Earlier protocol revisions addressed logistical and other issues in study conduct; we had no reason to review the analysis plan and identify the misspecification prior to that time. We are puzzled as to the relevance and implication of this point about protocol revision number.
Finally, we note that our study went through a highly rigorous review, involving 6 independent experts, including more than 120 points of criticism and discussion. We exceeded professional standards of transparency, by making the full database publicly available immediately upon publication – perhaps for the first time for a nutrition study of this magnitude and complexity.
CITATIONS
1. Hall KD, Guo J. 28 November 2018. No significant effect of dietary carbohydrate versus fat on the reduction in total energy expenditure during Maintenance of lost weight. https://www.bmj.com/content/363/bmj.k4583/rr-16
6. Ebbeling CB, Swain JF, Feldman HA, Wong WW, Hachey DL, Garcia-Lago E, Ludwig DS. Effects of dietary composition on energy expenditure during weight-loss maintenance. JAMA. 2012, 307:2627-34 https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3564212/
POINT #1 FROM PEER REVIEW, 18 SEPTEMBER 2018:
We closely adhered to the a priori clinical trials and protocol plans, also as detailed in our methods paper and Open Science Framework registry (https://osf.io/rvbuy/).
The only discrepancy involves a change from initial specification in the anchor used to calculate change in the primary outcome. In our original analysis plan of 2014, we had indicated the preweight loss (i.e., pre-Run-In, BSL in Figure 1) measurement as the anchor for determining the diet effect on total energy expenditure (TEE), but this was an error on our part. We corrected this error in the Clinical Trials registry and used an analysis for our manuscript with the post-weight loss (i.e., post-Run-In, PWL in Figure 1) measurement as the anchor. For the reasons explained below, we request an exception to your general rule of including both analyses in the Results section, and instead have provided further clarification of this issue in Methods.
1. The initial listing was clearly an error:
A) As a general rule, anchor data should be collected as close to randomization as possible, to decrease error introduced by time-varying covariates. The pre-Run-In measurement involves a 3- to 4-month delay prior to initiation of the Test diets.
B) In addition to this delay, the pre-Run-In measurement is strongly confounded by weight loss, whereas the specific aim of the study is to examine TEE during weight maintenance. (Indeed, the title of the study in the registry is: Dietary Composition and Energy Expenditure During Weight-Loss Maintenance.)
C) The expressed purpose of the Run-In was to produce 12% weight loss, changing biological state (i.e., creating a predisposition to weight regain) to test the study hypotheses. Thus, it would be inconsistent with study aims and methodologically inappropriate to use the pre-Run-In time point to establish a precise and accurate anchor for determining how the Test diets change TEE. Doing so would necessitate a substantially larger number of participants (and cost) to account for the additional imprecision, with no scientific benefit.
2. Study power (and thus participant number) was determined with use of post-Run-In measurement as the anchor.
A) Our a priori power calculations defined the primary outcome as “change in total energy expenditure at week 20 of the test phase compared to week 0 (post-weight loss).”
B) We did not take into account the variability between pre-Run-In and post-Run-In (Week 0) measurements, which in our case had r-values on the order of only 0.3 for the unadjusted model and 0.5 for the adjusted model.
C) Not surprisingly, doing the analysis with the pre-Run-In measurement as the anchor yielded a mean estimate in the same direction, but with substantial loss of precision and a statistically non-significant overall effect in the ITT. For example, TEE in the unadjusted model of Low vs High Carbohydrate diets was + 141 kcal/day (p=0.08, overall p=0.2).
3. The error was recognized and corrected a priori. We obtained IRB approval for our final analysis plan on 06 Sept 2017, before the blind was broken (and indeed, before measurement of the primary outcome had been completed by our collaborator Bill Wong in Houston). Similarly, we corrected the Clinical Trials registry prior to breaking the blind. We provide documentation of this timeline, and additional detail, in the Supplement Protocol section.
Although we agree with the general policy of including multiple analyses where discrepancies exist, we think an exception would be warranted in this situation. For reasons suggested above, we believe that we have fulfilled the letter and spirit of an a priori analysis plan. Furthermore, we are concerned that the additional analysis would provide no meaningful biological insights – that is, no useful information about the nature of the relationship between dietary composition and energy expenditure. Rather, inclusion of the additional analysis would tend to elevate and give undue attention to an error, and therefore potentially cause confusion.
To place our study in the context of other diet trials, I reviewed the Clinical Trials registry of diet and weight loss trials published in one of the JAMA journals since 2015 (to obtain a collection of cross-specialty examples). Of the 13 trials identified, 8 had significant changes in the primary outcome since initial posting. In several additional trials, the level of detail for the primary outcome was insufficient to exclude multiple statistical treatments [NB, see online Peer Review for references to these studies].
Regarding this last point, our pre-analysis plan provided a comparatively great level of detail. For contrast, the study by Hall et al of 2016 (cited in the Endocrine Society Scientific Statement as major evidence against the Carbohydrate-Insulin Model) included only a 1- paragraph statistical plan (https://osf.io/9q8cu/ beginning bottom of page 23), thus providing freedom to analyze outcomes in many different ways (e.g., post hoc exclusion of outliers).
My intent in providing this context isn’t to justify suboptimal practices, but rather to clarify that error or lack of specificity in an initial registry of major diet trials (vs industry-sponsored drug trials which have much larger budgets, infrastructural support and standardization) is more the rule than the exception. However, we believe that our overall rigor is comparatively very high, and the proposed course of action entails no risk to the integrity of the data analyses.
To address your reasonable concerns on this point and to maintain maximum transparency, we have clarified this situation in the Methods (page 12, para 2) and provided additional detail in the Supplement Protocol section. Please also note that we have committed to post the complete data set on a publicly available server upon publication of the manuscript, so that anyone can perform additional exploratory analyses, including this one. Nevertheless, we will defer to you, and include the additional analysis, if you disagree with our proposed solution.
Competing interests:
As detailed in the manuscript
Rapid Response:
Author Response to Hall and Guo Regarding Data Reanalysis and Other Criticisms
Hall and Guo posted a reanalysis of our study using the pre-weight loss rather than the post-weight loss measure of total energy expenditure, finding a non-significant effect (1). They propose that the pre-weight loss baseline is more “correct” and raise concern about the timing of our change to the clinical trials registry.
This alternative analysis is not new. We performed this calculation during manuscript revision, as publicly available in the Peer Review material (2). The BMJ editors and statistician had full access to these data and we deferred to them regarding whether to include this analysis in the manuscript. We did not “refuse” to do so.
The reasons for using the post-weight loss baseline were extensively discussed in the Peer Review material and in the manuscript. (For reader convenience, we include the Peer Review response at bottom.) We were transparent about this a priori change, and included a detailed timeline in the Supplemental materials (3).
We further addressed this issue in a prior Rapid Response (4), showing that Hall’s previously stated concerns about changing body weight (5) were unfounded, and that statistical models including pre-weight loss baseline did not materially change the result.
There is a straightforward explanation for why the registry was corrected as the “7[th] of 8 versions of the protocol.” We planned to post a final analysis plan, with comprehensive detail of our statistical methods, prior to receiving the final data from our collaborator in Houston and breaking the blind. In preparing that plan under guidance of our statistician, we identified the misspecification – an inadvertent holdover from a prior cross-over study (our present study is parallel design, in which the weight-loss phase would introduce inter-individual variability) (6). Earlier protocol revisions addressed logistical and other issues in study conduct; we had no reason to review the analysis plan and identify the misspecification prior to that time. We are puzzled as to the relevance and implication of this point about protocol revision number.
Finally, we note that our study went through a highly rigorous review, involving 6 independent experts, including more than 120 points of criticism and discussion. We exceeded professional standards of transparency, by making the full database publicly available immediately upon publication – perhaps for the first time for a nutrition study of this magnitude and complexity.
CITATIONS
1. Hall KD, Guo J. 28 November 2018. No significant effect of dietary carbohydrate versus fat on the reduction in total energy expenditure during Maintenance of lost weight. https://www.bmj.com/content/363/bmj.k4583/rr-16
2. Peer Review, Author Response, 18 September 2018, Point #1 pages 2-4: https://www.bmj.com/sites/default/files/attachments/bmj-article/pre-pub-...
3. Supplemental information, page 16: https://www.bmj.com/content/bmj/suppl/2018/11/13/bmj.k4583.DC1/ebbc04619...
4. Ludwig DS, Ebbeling, CB, Feldman HA. 20 November 2018. Choice of baseline for primary endpoint. https://www.bmj.com/content/363/bmj.k4583/rr-11
5. Belluz J, Vox, updated November 21, 2018. Does cutting carbs really help keep weight off? The big new diet study, explained. https://www.vox.com/science-and-health/2018/11/16/18096633/keto-low-carb...
6. Ebbeling CB, Swain JF, Feldman HA, Wong WW, Hachey DL, Garcia-Lago E, Ludwig DS. Effects of dietary composition on energy expenditure during weight-loss maintenance. JAMA. 2012, 307:2627-34 https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3564212/
POINT #1 FROM PEER REVIEW, 18 SEPTEMBER 2018:
We closely adhered to the a priori clinical trials and protocol plans, also as detailed in our methods paper and Open Science Framework registry (https://osf.io/rvbuy/).
The only discrepancy involves a change from initial specification in the anchor used to calculate change in the primary outcome. In our original analysis plan of 2014, we had indicated the preweight loss (i.e., pre-Run-In, BSL in Figure 1) measurement as the anchor for determining the diet effect on total energy expenditure (TEE), but this was an error on our part. We corrected this error in the Clinical Trials registry and used an analysis for our manuscript with the post-weight loss (i.e., post-Run-In, PWL in Figure 1) measurement as the anchor. For the reasons explained below, we request an exception to your general rule of including both analyses in the Results section, and instead have provided further clarification of this issue in Methods.
1. The initial listing was clearly an error:
A) As a general rule, anchor data should be collected as close to randomization as possible, to decrease error introduced by time-varying covariates. The pre-Run-In measurement involves a 3- to 4-month delay prior to initiation of the Test diets.
B) In addition to this delay, the pre-Run-In measurement is strongly confounded by weight loss, whereas the specific aim of the study is to examine TEE during weight maintenance. (Indeed, the title of the study in the registry is: Dietary Composition and Energy Expenditure During Weight-Loss Maintenance.)
C) The expressed purpose of the Run-In was to produce 12% weight loss, changing biological state (i.e., creating a predisposition to weight regain) to test the study hypotheses. Thus, it would be inconsistent with study aims and methodologically inappropriate to use the pre-Run-In time point to establish a precise and accurate anchor for determining how the Test diets change TEE. Doing so would necessitate a substantially larger number of participants (and cost) to account for the additional imprecision, with no scientific benefit.
2. Study power (and thus participant number) was determined with use of post-Run-In measurement as the anchor.
A) Our a priori power calculations defined the primary outcome as “change in total energy expenditure at week 20 of the test phase compared to week 0 (post-weight loss).”
B) We did not take into account the variability between pre-Run-In and post-Run-In (Week 0) measurements, which in our case had r-values on the order of only 0.3 for the unadjusted model and 0.5 for the adjusted model.
C) Not surprisingly, doing the analysis with the pre-Run-In measurement as the anchor yielded a mean estimate in the same direction, but with substantial loss of precision and a statistically non-significant overall effect in the ITT. For example, TEE in the unadjusted model of Low vs High Carbohydrate diets was + 141 kcal/day (p=0.08, overall p=0.2).
3. The error was recognized and corrected a priori. We obtained IRB approval for our final analysis plan on 06 Sept 2017, before the blind was broken (and indeed, before measurement of the primary outcome had been completed by our collaborator Bill Wong in Houston). Similarly, we corrected the Clinical Trials registry prior to breaking the blind. We provide documentation of this timeline, and additional detail, in the Supplement Protocol section.
Although we agree with the general policy of including multiple analyses where discrepancies exist, we think an exception would be warranted in this situation. For reasons suggested above, we believe that we have fulfilled the letter and spirit of an a priori analysis plan. Furthermore, we are concerned that the additional analysis would provide no meaningful biological insights – that is, no useful information about the nature of the relationship between dietary composition and energy expenditure. Rather, inclusion of the additional analysis would tend to elevate and give undue attention to an error, and therefore potentially cause confusion.
To place our study in the context of other diet trials, I reviewed the Clinical Trials registry of diet and weight loss trials published in one of the JAMA journals since 2015 (to obtain a collection of cross-specialty examples). Of the 13 trials identified, 8 had significant changes in the primary outcome since initial posting. In several additional trials, the level of detail for the primary outcome was insufficient to exclude multiple statistical treatments [NB, see online Peer Review for references to these studies].
Regarding this last point, our pre-analysis plan provided a comparatively great level of detail. For contrast, the study by Hall et al of 2016 (cited in the Endocrine Society Scientific Statement as major evidence against the Carbohydrate-Insulin Model) included only a 1- paragraph statistical plan (https://osf.io/9q8cu/ beginning bottom of page 23), thus providing freedom to analyze outcomes in many different ways (e.g., post hoc exclusion of outliers).
My intent in providing this context isn’t to justify suboptimal practices, but rather to clarify that error or lack of specificity in an initial registry of major diet trials (vs industry-sponsored drug trials which have much larger budgets, infrastructural support and standardization) is more the rule than the exception. However, we believe that our overall rigor is comparatively very high, and the proposed course of action entails no risk to the integrity of the data analyses.
To address your reasonable concerns on this point and to maintain maximum transparency, we have clarified this situation in the Methods (page 12, para 2) and provided additional detail in the Supplement Protocol section. Please also note that we have committed to post the complete data set on a publicly available server upon publication of the manuscript, so that anyone can perform additional exploratory analyses, including this one. Nevertheless, we will defer to you, and include the additional analysis, if you disagree with our proposed solution.
Competing interests: As detailed in the manuscript