Frequency and reasons for outcome reporting bias in clinical trials: interviews with trialistsBMJ 2011; 342 doi: https://doi.org/10.1136/bmj.c7153 (Published 06 January 2011) Cite this as: BMJ 2011;342:c7153
- R M D Smyth, research associate12,
- J J Kirkham, research associate1,
- A Jacoby, professor of medical sociology2,
- D G Altman, professor of statistics in medicine3,
- C Gamble, senior lecturer1,
- P R Williamson, professor of medical statistics1
- 1Centre for Medical Statistics and Health Evaluation, University of Liverpool, Liverpool, UK
- 2Division of Public Health, University of Liverpool, Liverpool, UK
- 3Centre for Statistics in Medicine, University of Oxford, Oxford, UK
- Correspondence to: P R Williamson
- Accepted 5 October 2010
Objectives To provide information on the frequency and reasons for outcome reporting bias in clinical trials.
Design Trial protocols were compared with subsequent publication(s) to identify any discrepancies in the outcomes reported, and telephone interviews were conducted with the respective trialists to investigate more extensively the reporting of the research and the issue of unreported outcomes.
Participants Chief investigators, or lead or coauthors of trials, were identified from two sources: trials published since 2002 covered in Cochrane systematic reviews where at least one trial analysed was suspected of being at risk of outcome reporting bias (issue 4, 2006; issue 1, 2007, and issue 2, 2007 of the Cochrane library); and a random sample of trial reports indexed on PubMed between August 2007 and July 2008.
Setting Australia, Canada, Germany, the Netherlands, New Zealand, the United Kingdom, and the United States.
Main outcome measures Frequency of incomplete outcome reporting—signified by outcomes that were specified in a trial’s protocol but not fully reported in subsequent publications—and trialists’ reasons for incomplete reporting of outcomes.
Results 268 trials were identified for inclusion (183 from the cohort of Cochrane systematic reviews and 85 from PubMed). Initially, 161 respective investigators responded to our requests for interview, 130 (81%) of whom agreed to be interviewed. However, failure to achieve subsequent contact, obtain a copy of the study protocol, or both meant that final interviews were conducted with 59 (37%) of the 161 trialists. Sixteen trial investigators failed to report analysed outcomes at the time of the primary publication, 17 trialists collected outcome data that were subsequently not analysed, and five trialists did not measure a prespecified outcome over the course of the trial. In almost all trials in which prespecified outcomes had been analysed but not reported (15/16, 94%), this under-reporting resulted in bias. In nearly a quarter of trials in which prespecified outcomes had been measured but not analysed (4/17, 24%), the “direction” of the main findings influenced the investigators’ decision not to analyse the remaining data collected. In 14 (67%) of the 21 randomly selected PubMed trials, there was at least one unreported efficacy or harm outcome. More than a quarter (6/21, 29%) of these trials were found to have displayed outcome reporting bias.
Conclusion The prevalence of incomplete outcome reporting is high. Trialists seemed generally unaware of the implications for the evidence base of not reporting all outcomes and protocol changes. A general lack of consensus regarding the choice of outcomes in particular clinical settings was evident and affects trial design, conduct, analysis, and reporting.
Contributors: The study was conceived by PRW, DGA, CG, and AJ. PRW, AJ, DGA, CG, and RMDS designed the study protocol. RMDS contacted the trialists to conduct the interviews. A comparison of trial protocols with trial reports was carried out by RMDS and JJK. Interviews were performed by RMDS. Analysis was performed by RMDS and JJK, with input and supervision from PRW and AJ. RMDS prepared the initial manuscript; PRW, AJ, DGA, and CG were involved in revisions of this manuscript. All authors commented on the final manuscript before submission. PRW is the guarantor for the project.
Funding: The ORBIT (Outcome Reporting Bias In Trials) project was funded by the Medical Research Council (grant number G0500952). The funders had no role in the study design, data collection and analysis, decision to publish, or preparation of this manuscript. DGA is supported by Cancer Research UK.
Competing interests: All authors have completed the Unified Competing Interest form at www.icmje.org/coi_disclosure.pdf (available on request from the corresponding author) and declare: no support from any organisation for the submitted work; no financial relationships with any organisations that might have an interest in the submitted work in the previous three years; and no other relationships or activities that could appear to have influenced the submitted work.
Data sharing: No additional data available.
This is an open-access article distributed under the terms of the Creative Commons Attribution Non-commercial License, which permits use, distribution, and reproduction in any medium, provided the original work is properly cited, the use is non commercial and is otherwise in compliance with the license. See: http://creativecommons.org/licenses/by-nc/2.0/ and http://creativecommons.org/licenses/by-nc/2.0/legalcode.