Intended for healthcare professionals

CCBY Open access
Research

Frequency of discrepancies in retracted clinical trial reports versus unretracted reports: blinded case-control study

BMJ 2015; 351 doi: https://doi.org/10.1136/bmj.h4708 (Published 20 September 2015) Cite this as: BMJ 2015;351:h4708
  1. Graham D Cole, research fellow1,
  2. Alexandra N Nowbar, academic foundation doctor1,
  3. Michael Mielewczik, research fellow1,
  4. Matthew J Shun-Shin, research fellow1,
  5. Darrel P Francis, professor1
  1. 1International Centre for Circulatory Health, National Heart and Lung Institute, Imperial College London W2 1LA, UK
  1. Correspondence to: G D Cole g.cole{at}imperial.ac.uk
  • Accepted 14 August 2015

Abstract

Objectives To compare the frequency of discrepancies in retracted reports of clinical trials with those in adjacent unretracted reports in the same journal.

Design Blinded case-control study.

Setting Journals in PubMed.

Population 50 manuscripts, classified on PubMed as retracted clinical trials, paired with 50 adjacent unretracted manuscripts from the same journals. Reports were randomly selected from PubMed in December 2012, with no restriction on publication date. Controls were the preceding unretracted clinical trial published in the same journal. All traces of retraction were removed. Three scientists, blinded to the retraction status of individual reports, reviewed all 100 trial reports for discrepancies. Discrepancies were pooled and cross checked before being counted into prespecified categories. Only then was the retraction status unblinded for analysis.

Main outcome measure Total number of discrepancies (defined as mathematically or logically contradictory statements) in each clinical trial report.

Results Of 479 discrepancies found in the 100 trial reports, 348 were in the 50 retracted reports and 131 in the 50 unretracted reports. On average, individual retracted reports had a greater number of discrepancies than unretracted reports (median 4 (interquartile range 2-8.75) v 0 (0-5); P<0.001). Papers with a discrepancy were significantly more likely to be retracted than those without a discrepancy (odds ratio 5.7 (95% confidence interval 2.2 to 14.5); P<0.001). In particular, three types of discrepancy arose significantly more frequently in retracted than unretracted reports: factual discrepancies (P=0.002), arithmetical errors (P=0.01), and missed P values (P=0.02). Results from a retrospective analysis indicated that citations and journal impact factor were unlikely to affect the result.

Conclusions Discrepancies in published trial reports should no longer be assumed to be unimportant. Scientists, blinded to retraction status and with no specialist skill in the field, identify significantly more discrepancies in retracted than unretracted reports of clinical trials. Discrepancies could be an early and accessible signal of unreliability in clinical trial reports.

Footnotes

  • We thank the infrastructural support provided by the National Institute for Health Research Biomedical Research Centre, based at Imperial College Healthcare NHS Trust and Imperial College London.

  • Contributors: GDC designed the study, examined the trials, cross checked the discrepancies, analysed the data, and drafted and revised the paper. ANN and MM examined the trials, cross checked the discrepancies, and drafted and revised the paper. MJS-S analysed the data and revised the manuscript. DPF cross checked the discrepancies, analysed the data, and drafted and revised the manuscript. DPF is guarantor.

  • Funding: This study was not funded by any external organisation. All authors are associated with Imperial College London. GDC and MJS-S are clinical research training fellows at the British Heart Foundation (FS/12/12/29294 and FS/14/27/30752, respectively). DPF is a senior clinical research fellow at the British Heart Foundation (FS/10/038). MM is supported by a junior research fellowship at Imperial College London. Neither the institution nor any funder had any role in devising, conducting, analysing, or reporting this study.

  • Competing interests: All authors have completed the ICMJE uniform disclosure form at www.icmje.org/coi_disclosure.pdf and declare: no support from any organisation for the submitted work; no financial relationships with any organisations that might have an interest in the submitted work in the previous three years; no other relationships or activities that could appear to have influenced the submitted work.

  • Ethical approval: Not required.

  • Data sharing: The complete list of identified discrepancies is shown in web appendix 3. These data are freely available from the corresponding author in editable form on request. We welcome and will make public any corrections or updates from readers. We also provide the raw data in web appendix 6 to permit reanalysis of our data, and the code we used for our analysis in web appendix 1.

  • The guarantor affirms that this manuscript is an honest, accurate, and transparent account of the study being reported; that no important aspects of the study have been omitted; and that any discrepancies from the study as planned (and, if relevant, registered) have been explained.

This is an Open Access article distributed in accordance with the terms of the Creative Commons Attribution (CC BY 4.0) license, which permits others to distribute, remix, adapt and build upon this work, for commercial use, provided the original work is properly cited. See: http://creativecommons.org/licenses/by/4.0/.

View Full Text