Post-submission changes to prespecified statistical analysis plans
BMJ 2022; 378 doi: https://doi.org/10.1136/bmj.o2244 (Published 21 September 2022) Cite this as: BMJ 2022;378:o2244
- Nazrul Islam, research editor and associate professor1 2,
- Tim J Cole, statistical editor and professor1 3,
- Joseph S Ross, research editor and professor1 4,
- Timothy Feeney, research editor and doctoral student1 5,
- Elizabeth Loder, head of research and professor1 6
- 1The BMJ
- 2Faculty of Medicine, University of Southampton, Southampton, UK
- 3UCL Great Ormond Street Institute of Child Health, University College London, London, UK
- 4Yale School of Medicine, Yale University, New Haven, CT, USA
- 5Gillings School of Global Public Health, University of North Carolina-Chapel Hill, Chapel Hill, NC, USA
- 6Department of Neurology, Brigham and Women’s Hospital, Harvard Medical School, Boston, MA
- Correspondence to: E Loder eloder{at}bmj.com
Transparency and reproducibility are two of the fundamental principles of evidence generation and dissemination of research knowledge.12 Researchers ask questions, challenge answers, replicate results, and scrutinise findings—this is how we accumulate knowledge and help develop evidence based policy. Scientific integrity, reliability, reproducibility, and transparency are paramount if scientific studies are to serve as the basis for policies that can have a big effect on human wellbeing.3
Detailed guidelines on writing a prespecified research protocol and statistical analysis plan have been developed to improve the transparency and reproducibility of research.456 Prespecified plans have a critical role in minimising and detecting selective analysis and reporting.78The BMJ requires authors of clinical trials to upload their protocol and statistical analysis plans during submission, and encourages it for other study types. The protocols and analysis plans of all accepted studies are published alongside their findings.9 This is to reassure readers that the analysis was honest, did not involve post-hoc preferential analysis or selective reporting of favourable results, and that no additional analysis was done before publication to bias the findings.
Challenges to the statistical analysis plan
During the peer review process, editors, reviewers, and statistical advisers check that authors have adhered to the major components of the prespecified plan, especially with clinical trials.9 However, analyses occasionally deviate from those specified. The authors may have discovered an error in their statistical analysis plan, identified something unexpected in the data, found a better way of conducting an analysis, or perhaps learnt something from the data that was worth exploring. In these cases, The BMJ requires authors to document any changes clearly and provide a sound rationale for doing so, to ensure full transparency.
The review process may also identify a flaw or limitation in the plan, leading editors to ask authors to revisit their methods or analyses. Examples include, but are not limited to, authors being asked to use ANCOVA rather than ANOVA for modelling continuous outcomes; use multiple imputation rather than the missing indicator method; avoid dichotomising continuous variables; use additional statistical techniques to test the robustness of their results; select potential confounders using directed acyclic graphs; or adjust for variables used for stratification in a randomised controlled trial.
As part of The BMJ’s editorial team, we occasionally encounter resistance from authors when we ask for revisions that would modify prespecified analyses. Although there is a strong case for adhering to a study’s analysis plan, the methods and analysis must also be acceptable to the journal considering that study for publication. If authors feel obliged to do alternative analyses to satisfy a journal, does this undermine the whole purpose of prespecified plans?
One key aim of the prespecified plan is to prevent researchers from “fishing” for expected or hoped for findings that could bias a study’s results and potentially invalidate the conclusions. If authors adhere to their plan when conducting a study, however, it has served its purpose before the study is submitted to a journal—that particular source of bias has been ruled out. Journal editors, reviewers, and advisers suggest changes to help strengthen research findings and improve reporting. Their suggestions are independent, made without any knowledge of how suggested changes might influence findings, and—it must be assumed—without competing interests. Therefore, post-submission changes that modify the statistical analysis plan do not conflict with its core purpose.
Nevertheless, major deviations should be kept to a minimum. Editors have a responsibility to protect authors and research integrity by ensuring that any suggested changes to the planned analysis adhere to a trial’s original objectives and hypotheses. Any deviations, whether initiated by authors or journals, must be documented clearly in the final version of the manuscript under a separate subheading “protocol deviations.” The authors can also clarify here that a particular analysis was conducted following suggestions from the peer reviewers.
The BMJ shares the academic and professional identities of reviewers with authors, and vice versa. Both authors and reviewers are asked to declare any conflicts of interest. In addition, comments made by editors and reviewers, along with authors’ responses and revisions, are published with the accepted paper. This open peer review makes the process transparent, accountable, and iterative.
Suggestions from editors, reviewers, or statistical advisers are not always implemented — authors have the scholarly independence to challenge them with appropriate academic argument. We encourage such discussion since it provides an excellent opportunity for us to learn from each other. We do not, however, accept a refusal to follow editors’ or reviewers’ recommendations based solely on the argument that they would violate the statistical analysis plan. Post-submission changes to an analysis that go beyond the plan and improve reporting can only be a good thing—the plan should not be considered immutable.
Acknowledgments
We thank the other BMJ editors, especially Kamran Abbasi and Richard Riley, for their review and comments.
Footnotes
Competing interests: We have read and understood BMJ policy on declaration of interests and declare that all authors are involved in assessing research submitted to The BMJ for publication. JSR receives research support through Yale University from Johnson and Johnson to develop methods of clinical trial data sharing, from the Medical Device Innovation Consortium as part of the National Evaluation System for Health Technology (NEST), from the Food and Drug Administration for the Yale-Mayo Clinic CERSI program (U01FD005938), from the Agency for Healthcare Research and Quality (R01HS022882), from the National Heart, Lung and Blood Institute of the National Institutes of Health (NIH) (R01HS025164, R01HL144644), and from Arnold Ventures. He is an expert witness at the request of relator’s attorneys, the Greene Law Firm, in a qui tam suit alleging violations of the False Claims Act and Anti-Kickback Statute against Biogen. TF is funded by the National Institute of Allergy and Infectious Diseases and has received funds from unrelated consulting work for Takeda, Pfizer, Acadia and iHeed.
Provenance and peer review: Commissioned; not externally peer reviewed.