Intended for healthcare professionals

CCBYNC Open access

Rapid response to:

Research

Impact of study outcome on submission and acceptance metrics for peer reviewed medical journals: six year retrospective review of all completed GlaxoSmithKline human drug research studies

BMJ 2017; 357 doi: https://doi.org/10.1136/bmj.j1726 (Published 21 April 2017) Cite this as: BMJ 2017;357:j1726

Rapid Response:

Solving the problems of publication and submission bias is an important step, but the focus must now turn to outcome reporting bias.

“Publication Bias” is perhaps the most famous member of a group of research integrity issues that go under the name of “Reporting Bias”. The phrase “Publication Bias” refers to the widely-documented observation that studies involving positive outcomes are published in the academic literature more frequently than those with negative outcomes. As the pharmaceutical industry is often considered one of the main perpetrators of publication bias, this paper by Evoniuk et al. represents a positive and highly encouraging contribution to the debate, with the authors now reporting a higher level of publication in journals for their studies with negative outcomes (77%) in comparison to those with positive outcomes (66%).

One documented cause of publication bias is a higher level of rejection by academic journals. This can mask a related form of reporting bias called “submission bias”. Submission bias occurs when researchers are selective about which research they actually submit to journals in the first place. Whereas publication bias can still occur even when research teams genuinely attempt to publish their results (and thus is often considered a problem of the science publishing system as a whole), submission bias can be considered more of a research misconduct issue as it is caused specially by the researchers themselves. Again, the paper by Evoniuk et al. reports some encouraging news in this regard. The authors show that 85% of the GSK studies in the declared timeframe had been submitted for publication (a further 9% were still under journal review, and 7% had been published as conference abstracts), with 71% having been successfully published. These results imply that the cause of the 14% of studies not being published was due to journal rejection rather than submission bias. Furthermore, of the studies that were published, 10% required three or more submissions, demonstrating a significant commitment by researchers to persevere until their studies were published. Consequently, the authors concluded that GSK has successfully implemented its policy whereby “all human research studies of its drug products (whether investigational or marketed) are submitted for journal publication”.

This is all very encouraging, but, consider the reason why the community wants the results of studies to be published in the first place: the aim must be to ensure that the wider medical (and patient) community have access to the evidence base used to justify clinical decision making. Solving the problems of submission and publication bias by ensuring that work is submitted to journals is obviously an important step in the right direction, but this is a distraction from the real reason to submit studies to academic journals. Indeed, if the only aim is to record that studies occurred, research registries alone can adequately fulfil this function. Rather, the role of the academic literature is to provide a thorough peer reviewed analysis of trials, their design, and critically whether the primary and secondary objectives were met in a statistically significant way. Any bias or lack of transparency in this more detailed type or report moves beyond the problems of submission or publication bias and into a third type of reporting bias called “outcome reporting bias”.

Again, laudably Evoniuk et al. account for the most common form of outcome reporting bias (switching primary and secondary outcomes based on statistical significance) by only using the originally stated primary outcome to define whether a trial could be considered positive or not. They also plainly acknowledged that they did not try to detect whether “selective reporting” had occurred. By this they presumably mean detecting whether or not all original outcomes (both primary and secondary) had been fully reported. Within the context of their particular piece of work this was perhaps appropriate as the task of detecting selective reporting is not trivial [1]. However, whilst solving the problems of submission and publication bias is certainly a step in the right direction, the issue of selective outcome reporting is the real “elephant” in the transparency room. Without knowing whether all relevant information from trials is being made available it is difficult to determine whether the publications analysed here are adequately reported each trial, or whether they have missed out critical information.

One potential and achievable solution to the problem of outcome reporting bias is to ensure that trials declare all their proposed outcomes (both primary and secondary) prior to collecting any data. Such information should be obtained from ethics committee records as these contain the full details of authorised trials [2]. This way, when a publication subsequently arrives, there will be a way for interested and independent parties (clinicians, Cochrane reviewers, other researchers etc.) to check whether the trial has been reported and discussed fully and appropriately. The issues of submission and publication bias are certainly important, and GSK is to be lauded for their significant steps leading the industry towards transparency, but clinical trials are a complex business and appropriate transparency will not be fully achieved until the medical and research community get the opportunity to know about, and scrutinise, all outcomes of all trials in detail.

1. Begum R, Kolstoe S. Can UK NHS research ethics committees effectively monitor publication and outcome reporting bias? BMC Med Ethics 2015;356:51. doi:10.1186/ s12910-015-0042-8 pmid:26206479
2. Kolstoe, S.E. Research Ethics committees/boards should routinely publish primary and secondary outcomes of all ethically approved research. BMJ 2017;356:j1501 doi: https://doi.org/10.1136/bmj.j1501

Competing interests: The author received funding as part of a wider academic collaboration with GlaxoSmithKline between 2009 and 2012.

03 May 2017
Simon E Kolstoe
Senior Lecturer& University Ethics Advisor
University of Portsmouth
Institute of Biomedical and Biomolecular Science, University of Portsmouth, Portsmouth, UK