Research

Discrepancies in sample size calculations and data analyses reported in randomised trials: comparison of publications with protocols

BMJ 2008; 337 doi: 10.1136/bmj.a2299 (Published 4 December 2008)
Cite this as: BMJ 2008;337:a2299
  1. An-Wen Chan, clinical fellow1,
  2. Asbjørn Hróbjartsson, senior researcher2,
  3. Karsten J Jørgensen, researcher2,
  4. Peter C Gøtzsche, director2,
  5. Douglas G Altman, director3
  1. 1Mayo Clinic, Rochester, USA
  2. 2Nordic Cochrane Centre, Copenhagen, Denmark
  3. 3Centre for Statistics in Medicine, University of Oxford, Oxford
  1. Correspondence to: A-W Chan chan.anwen{at}mayo.edu
  • Accepted 10 September 2008

Abstract

Objective To evaluate how often sample size calculations and methods of statistical analysis are pre-specified or changed in randomised trials.

Design Retrospective cohort study.

Data source Protocols and journal publications of published randomised parallel group trials initially approved in 1994-5 by the scientific-ethics committees for Copenhagen and Frederiksberg, Denmark (n=70).

Main outcome measure Proportion of protocols and publications that did not provide key information about sample size calculations and statistical methods; proportion of trials with discrepancies between information presented in the protocol and the publication.

Results Only 11/62 trials described existing sample size calculations fully and consistently in both the protocol and the publication. The method of handling protocol deviations was described in 37 protocols and 43 publications. The method of handling missing data was described in 16 protocols and 49 publications. 39/49 protocols and 42/43 publications reported the statistical test used to analyse primary outcome measures. Unacknowledged discrepancies between protocols and publications were found for sample size calculations (18/34 trials), methods of handling protocol deviations (19/43) and missing data (39/49), primary outcome analyses (25/42), subgroup analyses (25/25), and adjusted analyses (23/28). Interim analyses were described in 13 protocols but mentioned in only five corresponding publications.

Conclusion When reported in publications, sample size calculations and statistical methods were often explicitly discrepant with the protocol or not pre-specified. Such amendments were rarely acknowledged in the trial publication. The reliability of trial reports cannot be assessed without having access to the full protocols.

Footnotes

  • Contributors: All authors contributed to the study design, data collection, data interpretation, and drafting of the manuscript. A-WC also did the statistical analyses and is the guarantor.

  • Funding sources: None.

  • Competing interests: A-WC chairs and AH, DGA, and PCG are members of the steering committee for the SPIRIT (standard protocol items for randomised trials) initiative, a project that aims to define key protocol content for randomised trials.

  • Ethical approval: The scientific-ethics committees for Copenhagen and Frederiksberg granted unrestricted access to trial protocols. The research did not undergo the usual ethical review process, as patients’ data were not reviewed.

  • Provenance and peer review: Not commissioned; externally peer reviewed.

This is an open-access article distributed under the terms of the Creative Commons Attribution Non-commercial License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Rapid responses

THIS WEEK'S POLL

Read related article

See previous polls

BMJ most popular