Data sharing and reanalysis of randomized controlled trials in leading biomedical journals with a full data sharing policy: survey of studies published in The BMJ and PLOS MedicineBMJ 2018; 360 doi: https://doi.org/10.1136/bmj.k400 (Published 13 February 2018) Cite this as: BMJ 2018;360:k400
- Florian Naudet, postdoctoral fellow1,
- Charlotte Sakarovitch, senior statistician2,
- Perrine Janiaud, postdoctoral fellow1,
- Ioana Cristea, visiting scholar1 3,
- Daniele Fanelli, senior scientist1 4,
- David Moher, visiting scholar1 5,
- John P A Ioannidis, professor1 6
- 1Meta-Research Innovation Center at Stanford (METRICS), Stanford University, Stanford, California, USA
- 2Quantitative Sciences Unit, Division of Biomedical Informatics Research, Department of Medicine, Stanford University, Stanford, CA, USA
- 3Department of Clinical Psychology and Psychotherapy, Babes-Bolyai University, Romania
- 4Department of Methodology, London School of Economics and Political Science, UK
- 5Centre for Journalology, Clinical Epidemiology Program, Ottawa Hospital Research Institute, Ottawa, ON, Canada
- 6Departments of Medicine, of Health Research and Policy, of Biomedical Data Science, and of Statistics, Stanford University, Stanford, California, USA
- Correspondence to: J P A Ioannidis
- Accepted 8 January 2018
Objectives To explore the effectiveness of data sharing by randomized controlled trials (RCTs) in journals with a full data sharing policy and to describe potential difficulties encountered in the process of performing reanalyses of the primary outcomes.
Design Survey of published RCTs.
Eligibility criteria RCTs that had been submitted and published by The BMJ and PLOS Medicine subsequent to the adoption of data sharing policies by these journals.
Main outcome measure The primary outcome was data availability, defined as the eventual receipt of complete data with clear labelling. Primary outcomes were reanalyzed to assess to what extent studies were reproduced. Difficulties encountered were described.
Results 37 RCTs (21 from The BMJ and 16 from PLOS Medicine) published between 2013 and 2016 met the eligibility criteria. 17/37 (46%, 95% confidence interval 30% to 62%) satisfied the definition of data availability and 14 of the 17 (82%, 59% to 94%) were fully reproduced on all their primary outcomes. Of the remaining RCTs, errors were identified in two but reached similar conclusions and one paper did not provide enough information in the Methods section to reproduce the analyses. Difficulties identified included problems in contacting corresponding authors and lack of resources on their behalf in preparing the datasets. In addition, there was a range of different data sharing practices across study groups.
Conclusions Data availability was not optimal in two journals with a strong policy for data sharing. When investigators shared data, most reanalyses largely reproduced the original results. Data sharing practices need to become more widespread and streamlined to allow meaningful reanalyses and reuse of data.
Trial registration Open Science Framework osf.io/c4zke.
Contributors: FN, DF, and JI conceived and designed the experiments. FN, PJ, CS, IC, and JI performed the experiments. FN and CS analyzed the data. FN, CS, PJ, IC, DF, DM, and JI interpreted the results. FN wrote the first draft of the manuscript. CS, PJ, IC, DF, DM, and JI contributed to the writing of the manuscript. FN, CS, PJ, IC, DF, DM, and JI agreed with the results and conclusions of the manuscript. All authors have read, and confirm that they meet, ICMJE criteria for authorship. All authors had full access to all of the data (including statistical reports and tables) in the study and can take responsibility for the integrity of the data and the accuracy of the data analysis. FN is the guarantor.
Competing interests: All authors have completed the ICMJE uniform disclosure form at http://www.icmje.org/coi_disclosure.pdf (available on request from the corresponding author) and declare that (1) No authors have support from any company for the submitted work; (2) FN has relationships (travel/accommodations expenses covered/reimbursed) with Servier, BMS, Lundbeck, and Janssen who might have an interest in the work submitted in the previous three years. In the past three years PJ received a fellowship/grant from GSK for her PhD as part of a public-private collaboration. CS, IC, DF, DM, and JPAI have no relationship with any company that might have an interest in the work submitted; (3) no author’s spouse, partner, or children have any financial relationships that could be relevant to the submitted work; and (4) none of the authors has any non-financial interests that could be relevant to the submitted work.
Funding: METRICS has been fundedby Laura and John Arnold Foundation but there was no direct funding for this study. FN received grants from La Fondation Pierre Deniker, Rennes University Hospital, France (CORECT: COmité de la Recherche Clinique et Translationelle) and Agence Nationale de la Recherche (ANR), PJ is supported by a postdoctoral fellowship from the Laura and John Arnold Foundation, IC was supported by the Laura and John Arnold Foundation and the Romanian National Authority for Scientific Research and Innovation, CNCS–UEFISCDI, project number PN-II-RU-TE-2014-4-1316 (awarded to IC), and the work of JI is supported by an unrestricted gift from Sue and Bob O’Donnell. The sponsors had no role concerning preparation, review, or approval of the manuscript.
Ethical approval: Not required.
Data sharing: The code is shared on the Open Science Framework (https://osf.io/jgsw3/). All datasets that were used are retrievable following the instruction of the original papers.
Transparency: The guarantor (FN) affirms that the manuscript is a honest, accurate, and transparent account of the study bring reported; that no important aspects of the study have been omitted; and that any discrepancies from the study as planned (and, if relevant, registered) have been explained.
This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.