Avoidable waste of research related to inadequate methods in clinical trialsBMJ 2015; 350 doi: https://doi.org/10.1136/bmj.h809 (Published 24 March 2015) Cite this as: BMJ 2015;350:h809
- Youri Yordanov, physician and PhD student12,
- Agnes Dechartres, researcher134,
- Raphaël Porcher, associate professor134,
- Isabelle Boutron, professor1345,
- Douglas G Altman, professor and director6,
- Philippe Ravaud, professor and director13457
- 1Centre de Recherche Epidémiologie et Statistique, INSERM U1153, Paris, France
- 2Service des Urgences, Hôpital Saint-Antoine, Assistance Publique-Hôpitaux de Paris, Paris, France
- 3Centre d’Epidémiologie Clinique, Hôpital Hôtel-Dieu, Assistance Publique-Hôpitaux de Paris, Paris, France
- 4Faculté de Médecine, Université Paris Descartes, Sorbonne Paris Cité, Paris, France
- 5French Cochrane Centre, Paris, France
- 6Centre for Statistics in Medicine, University of Oxford, Oxford, UK
- 7Department of Epidemiology, Mailman School of Public Health, Columbia University New York, USA
- Correspondence to: A Dechartres, Centre de Recherche Epidémiologie et Statistique, INSERM U1153, Centre d’Epidémiologie Clinique, Hôpital Hôtel-Dieu, 1 place du Parvis Notre Dame, 75004 Paris, France
- Accepted 20 January 2015
Objective To assess the waste of research related to inadequate methods in trials included in Cochrane reviews and to examine to what extent this waste could be avoided. A secondary objective was to perform a simulation study to re-estimate this avoidable waste if all trials were adequately reported.
Design Methodological review and simulation study.
Data sources Trials included in the meta-analysis of the primary outcome of Cochrane reviews published between April 2012 and March 2013.
Data extraction and synthesis We collected the risk of bias assessment made by the review authors for each trial. For a random sample of 200 trials with at least one domain at high risk of bias, we re-assessed risk of bias and identified all related methodological problems. For each problem, possible adjustments were proposed that were then validated by an expert panel also evaluating their feasibility (easy or not) and cost. Avoidable waste was defined as trials with at least one domain at high risk of bias for which easy adjustments with no or minor cost could change all domains to low risk. In the simulation study, after extrapolating our re-assessment of risk of bias to all trials, we considered each domain rated as unclear risk of bias as missing data and used multiple imputations to determine whether they were at high or low risk.
Results Of 1286 trials from 205 meta-analyses, 556 (43%) had at least one domain at high risk of bias. Among the sample of 200 of these trials, 142 were confirmed as high risk; in these, we identified 25 types of methodological problem. Adjustments were possible in 136 trials (96%). Easy adjustments with no or minor cost could be applied in 71 trials (50%), resulting in 17 trials (12%) changing to low risk for all domains. So the avoidable waste represented 12% (95% CI 7% to 18%) of trials with at least one domain at high risk. After correcting for incomplete reporting, avoidable waste due to inadequate methods was estimated at 42% (95% CI 36% to 49%).
Conclusions An important burden of wasted research is related to inadequate methods. This waste could be partly avoided by simple and inexpensive adjustments.
We thank Carolina Riveros for help with the risk of bias assessment, Sally Hopewell for participating to the expert panel, and Ali Al Khafaji for help with data extraction. We also thank Annick Tibi, pharmacist in charge of the pharmaceutical services for all clinical trials sponsored by Assistance Publique-Hôpitaux de Paris; Patrick Boyer, orthopaedic surgeon at Bichat Hospital, Paris; and Patrick Rozenberg, obstetrician at Poissy hospital for discussing feasibility and cost of methodological adjustments for blinding in pharmacological and non-pharmacological trials, respectively.
Contributors: YY was involved in the study conception, selection of trials, data extraction, data analysis, interpretation of results, and drafting the manuscript; AD was involved in the study conception, selection of trials, data extraction, data analysis, interpretation of results, and drafting the manuscript; RP was involved in the data analysis, interpretation of results, and drafting the manuscript; IB was involved in the study conception, interpretation of results, and drafting the manuscript; DA was involved in the interpretation of results and drafting the manuscript; PR was involved in the study conception, interpretation of results, and drafting the manuscript. AD is the guarantor.
Funding: This study did not receive any specific funding. The team Centre d’Epidemiologie Clinique is supported by an academic grant from the programme “Equipe espoir de la Recherche”, Fondation pour la Recherche Médicale (FRM), Paris, France (No DEQ20101221475). AD is funded by the Institut National de la Santé et de la Recherche Médicale (INSERM). DGA is supported by Cancer Research UK (C5529). These funders had no role in the design and conduct of the study, collection, management, analysis, interpretation of the data, and preparation, review, or approval of the manuscript.
Competing interests: All authors have completed the ICMJE uniform disclosure form at www.icmje.org/coi_disclosure.pdf (available on request from the corresponding author) and declare: no support from any organisation for the submitted work; no financial relationships with any organisations that might have an interest in the submitted work in the previous three years; no other relationships or activities that could appear to have influenced the submitted work.
Ethical approval: Not needed.
Transparency: AD affirms that the manuscript is an honest, accurate, and transparent account of the study being reported; that no important aspects of the study have been omitted; and that any discrepancies from the study as planned have been explained.
Data sharing: Data available upon request for academic researchers.
This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.