Impact of study outcome on submission and acceptance metrics for peer reviewed medical journals: six year retrospective review of all completed GlaxoSmithKline human drug research studiesBMJ 2017; 357 doi: https://doi.org/10.1136/bmj.j1726 (Published 21 April 2017) Cite this as: BMJ 2017;357:j1726
- Gary Evoniuk, director, publications and disclosure (retired)1,
- Bernadette Mansi, head, publications and disclosure2,
- Barbara DeCastro, director, publications operations2,
- Jennie Sykes, vice president, global medical platforms3
- 1GlaxoSmithKline Research and Development, Research Triangle Park, NC 27709, USA
- 2GlaxoSmithKline Research and Development, Collegeville, PA, USA
- 3GlaxoSmithKline Research and Development, London, UK
- Correspondence to: G Evoniuk
- Accepted 27 March 2017
Objectives To determine whether the outcome of drug studies influenced submission and/or acceptance rates for publication in peer reviewed medical journals.
Design A six year retrospective review of publication status by study outcome for all human drug research studies conducted by a single industry sponsor (GlaxoSmithKline) that completed from 1 January 2009 to 30 June 2014 and were therefore due for manuscript submission (per the sponsor’s policy) to peer reviewed journals within 18 months of study completion—that is, 31 December 2015. In addition, manuscripts from studies completing after 30 June 2014 were included irrespective of outcome if they were submitted before 31 December 2015.
Setting Studies conducted by a single industry sponsor (GlaxoSmithKline)
Studies reviewed 1064 human drug research studies.
Main outcome measures All studies were assigned a publication status at 26 February 2016 including (as applicable): study completion date, date of first primary manuscript submission, number of submissions, journal decision(s), and publication date. All studies were also classified with assessors blinded to publication status as “positive” (perceived favorable outcome for the drug under study), “negative” (perceived unfavorable outcome for the drug under study), mixed, or non-comparative based on the presence and outcome of the primary outcome measure(s) for each study. “Negative” studies included safety studies in which the primary outcome was achieved but was adverse for the drug under study. For the total cohort and each of the four study outcomes, measures included descriptive statistics for study phase, time from study completion to submission and publication, and number and outcome (accepted/rejected) of publication submissions.
Results Of the 1064 studies (phase I-IV, interventional and non-interventional) included, 321 had study outcomes classified as positive, 155 as negative, 52 as mixed, and 536 as non-comparative. At the time of publication cut-off date (26 February 2016), 904 (85%) studies had been submitted for publication as full manuscripts and 751 (71%) had been successfully published or accepted, with 100 (9%) still under journal review. An additional 77 (7%) studies were conference abstracts and were not included in submission or publication rates. Submission rates by study outcome were 79% for the 321 studies with positive outcomes, 92% for the 155 with negative outcomes, 94% for the 52 with mixed outcomes, and 85% for the 536 non-comparative studies; while rates of publication at the cut-off date were 66%, 77%, 77%, and 71%, respectively. Median time from study completion to submission was 537 days (interquartile range 396-638 days) and 823 days (650-1063 days) from completion to publication, with similar times observed across study outcomes. First time acceptance rates were 56% for studies with positive outcomes and 48% for studies with negative outcomes. Over 10% of studies across all categories required three or more submissions to achieve successful publication. At the time of analysis, 83 studies had not been submitted for publication, including 49 bioequivalence studies with positive outcomes and 33 non-comparative studies. Most studies (98%, 1041/1064) had results posted to one or more public registers, including all studies subject to FDAAA (Food and Drug Administration Amendments Act) requirements for posting to www.clinicaltrials.gov.
Conclusions Over the period studied, there was no evidence of submission or publication bias: 92% of studies with negative outcomes were submitted for publication by the cut-off date versus 79% of those with positive outcomes. Publication rates were slightly higher for studies with a negative (that is, unfavorable) outcome compared with a positive outcome, despite a slightly lower rate of acceptance at first submission. Many studies required multiple submission attempts before they were accepted for publication. Analyses focusing solely on publication rates do not take into account unsuccessful efforts to publish. Sponsors and journal editors should share similar information to contribute to better understanding of issues and barriers to full transparency.
Industry and, to a lesser extent, academia have been criticized for failing to submit clinical trial data for publication, especially when the results are perceived to be “negative”—that is, unfavorable to the drug under study—leading to publication bias. Despite a commitment by many industry sponsors to publish all research results, failure to publish regardless of outcome adversely impacts the credibility of all industry sponsored research.12
Transparent reporting of medical research irrespective of outcome fulfills an ethical obligation to the patients taking part in a trial, advances scientific understanding, and could inform treatment decisions. Although public posting of summary results is now a legal requirement for many types of studies, publication in peer reviewed journals is still considered the ideal form of disclosure as it provides critical context that aids interpretation. Literature surveys carried out since 2010 suggest that 56-85% of trials with study protocols registered on www.clinicaltrials.gov and other public sites were eventually published.3456789 There has been wide variance across and within sponsor categories (such as industry, academia, government) and by study outcomes (presence versus absence of significance) in both publication rates as well as time to publication, with a general temporal trend toward increased disclosure rates.346789101112 Studies lacking statistically significant outcomes were less likely to be published and, when they were published, took a longer time from study completion to publication.1011
Most literaturesurveys rely on information on public websites (such as www.clinicaltrials.gov) that lack the information necessary to take into account the effect of unsuccessful attempts to publish (that is, journal rejection) on publication rates and time to publish. We analyzed study and publication outcomes using a cohort of all drug trials completed from January 2009 to June 2014 from a single pharmaceutical sponsor, GlaxoSmithKline (GSK). Since 2009, GSK policy requires that all human research studies of its drug products (whether investigational or marketed) are submitted for journal publication within 18 months of study completion unless exempted. For this analysis, we classified all studies completing during this period (including those not submitted or published) by outcome and hypothesized that there would be no difference in submission rates by study outcomes (per sponsor policy) but that studies with negative outcomes would have lower acceptance rates and require a greater number of submission attempts to be published in peer reviewed medical journals.
From the sponsor’s clinical trial management system, we compiled a comprehensive list of all human drug research studies sponsored by GSK Pharmaceuticals that completed (last participant’s final study visit, or completion of statistical analysis for observational studies) from 1 January 2009 to 30 June 2014. We also extracted and cross checked other study characteristics (study type, phase, end date) against www.clinicaltrials.gov and www.gsk-clinicalstudyregister.com. We selected 1 January 2009 as the start as this was the date from which GSK policy required all human drug research studies to be submitted for publication within 18 months of study completion. We selected 30 June 2014 as the cut off to ensure capture of the cohort of studies expected by this policy to have been submitted by 31 December 2015. To capture all manuscript submissions from 2010-15, we also included studies that completed after 30 June 2014, irrespective of outcome, if they had a manuscript submitted by 31 December 2015. We excluded studies of vaccines or consumer products, with no drug administration, with a change in sponsor, or with termination before completion of sufficient patient enrollment to conduct the primary statistical analyses, and studies submitted for publication outside 2010-15 (see fig A in appendix).
Positive: significant difference (P<0.05 or non-overlapping 95% confidence intervals) on the primary outcome measure prespecified in the protocol in favor of the experimental drug or, for safety studies, a lack of adverse safety findings (such as corrected QT studies/non-inferiority) or all formulations tested were within bioequivalence/non-inferiority limits. A subcategory of “pure positive” studies was also tracked for those studies meeting only the first criterion listed above
Negative: lack of significant difference (as defined above) on primary outcome measure prespecified in the protocol or appearance of an adverse safety finding or lack of bioequivalence. A subcategory of “pure negative” studies was also tracked for those studies meeting only the first criterion listed above
Mixed: both significant and non-significant results on studies with more than one primary endpoint prespecified in the protocol
Non-comparative: studies that did not meet the above criteria, including those with only descriptive statistics—that is, no preplanned formal comparisons or prospective identification of a primary endpoint (typically phase I and pharmacokinetic studies)—or studies that were terminated early without the planned statistical analysis of the primary outcome.
In classifying study outcomes we considered only the outcomes of the primary endpoint and did not consider secondary endpoints. Outcomes were assessed by an external party (Tata Consultancy Services medical writing team) and verified by one of the authors (GE), both of whom were blinded to publication status. Concordance of initial assessments was >80% with discrepancies resolved by consensus discussion.
Publication status was determined from a snapshot review of information within the sponsor’s publication tracking system as of 26 February 2016, which provided the following information for studies included in this cohort: number of submissions, outcome of each submission, and dates of submission and publication (when available). We included only submissions of full manuscripts to peer reviewed journals in the main analysis. Posting of study outcomes to public registries such as www.clinicaltrials.gov or www.gsk-clinicalstudyregister.com did not qualify as submission for publication. Disclosure through conference abstracts was also tracked but not included in the main analysis. Studies were grouped into published (including those accepted but awaiting publication), those currently under journal review, and those not published (either rejected or not submitted). The number of submissions for each study was also recorded. Resubmission to the same journal was not counted as a separate submission, but resubmission to a separate journal within or outside of a given publisher’s journal “family” was counted. Although more than one study could be combined into a single publication submission, for the purpose of this analysis we considered each study as a separate attempt to publish, given the interest in the comparison of study outcomes to publication status.
After separate compilation of study outcomes and publication status, we merged the data and generated descriptive statistics for publication status and number of submission attempts. No other formal statistical comparisons were planned but we used Fisher’s exact test post hoc for proportion of submissions, acceptance, and first time acceptance for studies with positive versus negative outcomes.
No patients were involved in setting the research question or the outcome measures, nor were they involved in developing plans for implementation of the study. No patients were asked to advise on interpretation or writing up of results. There are no plans to disseminate the results of the research to study participants or the relevant patient community.
The search identified 1003 studies that completed from 1 January 2009 and 30 June 2014 and were therefore expected to have a primary manuscript submitted within 18 months of study completion (that is, by the end of 2015) and an additional 61 more recently completed studies that had a manuscript submitted by the end of 2015, making a total of 1064 studies in the cohort. These studies were then classified according to study outcome (table 1⇓); 45% (n=476) of all studies and 50% (386/779) of interventional studies could be classified as having positive or negative outcomes, including 69% of all phase IIb and phase III studies. In contrast, 65% (n=250/385) of phase I studies were classified as non-comparative and comprised nearly half of the studies within this outcome category (see table A in appendix for additional information on characteristics of non-comparative studies).
Table 2⇓ summarizes publication status by study phase: 904 (85%) of all studies had been submitted for publication as full manuscripts, 751 (71%) had been accepted and/or published as full manuscripts, and 100 (9%) were submitted but still awaiting a journal decision. Some 133 studies were combined into a total of 65 submissions to increase their scientific interest, and the 771 remaining were submitted as standalone publications. Full manuscript publication rates were highest for phase II and III studies (78-88%), whereas phase I studies had the lowest publication rate (57%, 219/385), though an additional 18% (69) were disclosed via conference abstracts.
Of the 83studies that were not submitted for publication in any form at the time of analysis cut off, 49 were positive bioequivalence studies that showed no differences between formulations; 24 did not examine the safety or efficacy of a drug; nine were terminated early such that the primary outcome measure was not analyzed; and one was excluded because of confounding by indication. Of these, 100% (81/81) that were subject to FDAAA (Food and Drug Administration Amendments Act) or GSK requirements for public disclosure had results posted to www.clinicaltrials.gov or www.gsk-clinicalstudyregister.com (for the full cohort of 1064 studies, 1041 had results posted).
Table 3⇓ summarizes publication status by study outcome. Rates of publication were similar across all categories of study outcome: 66-77% of all categories were published as full manuscripts by the cut-off date with a further 5-8% disclosed as abstracts only. Submission rates were lower for studies with positive outcomes than for studies with negative outcomes (79% v 92%, Fisher’s exact P<0.001), as were acceptance rates (66% v 77%, P=0.019). Rates of non-publication were similar across all categories of study outcome, whether because of journal rejections (3-10%) or lack of manuscript submission during the analysis period (6-16%). Overall, the median time from study completion to submission was 537 days (interquartile range 396-638 days) and from completion to publication was 823 days (650-1063 days). These times were broadly similar across study outcomes, though for studies with negative outcomes, median times to submission and publication were 31 and 102 days longer, respectively, than for studies with positive outcomes.
Figure 1⇓ summarizes acceptance rates for studies with positive and negative outcomes. First time acceptance rates were 56% for positive studies and 48% for negative studies (P=0.17). These rates were also similar when we performed the same analysis using the pure positive and pure negative categories described in methods (50% v 49%). About three quarters of studies were accepted after one to two submissions for both study outcome categories (78% for positive v 73% for negative studies).
In this systematic review of more than 1000 studies from a single industry sponsor, rates of submission for publication in peer reviewed journals were broadly similar across study types and outcomes. Over the study period, a greater proportion of studies with negative outcomes than positive outcomes were submitted (92% v 79%) and accepted (77% v 66%) for publication in a journal, indicating a lack of bias against either submission or publication of negative studies by sponsor and journals, respectively. Given the proximity of the analysis cut-off point to the end of the six year review period, journal decisions were still pending for 9% of manuscripts submitted. An additional 7% of studies were disclosed through conference abstracts only (not included in publication totals). In total, 83 studies (8%) had not been submitted for publication in any form at the time of analysis, of which 49 were bioequivalence studies with positive outcomes and 33 were non-comparative studies. Most (98%) studies had results posted to one or more public registers, including all studies subject to FDAAA requirements for posting on www.clinicaltrials.gov.
While the topic of industry bias in reporting results of negative studies had been recently and widely debated, to our knowledge no previous study has systematically covered the issue of publication bias according to study outcome.16710111213 In our cohort, overall publication rates were broadly in line with recent estimates based solely on data obtained from www.clinicaltrials.gov.567 Median times to publication were also consistent with previously published figures.6789 When we evaluated the impact of study outcome, first time acceptance rates for studies with positive outcomes were numerically, but not significantly, higher than for those with negative outcomes (56% v 48%). Median time from study completion to submission was 31 days longer for negative than for positive studies and time to publication was 102 days longer, suggesting that greater effort and care (such as journal selection) are required for publication of negative data. Even so, 10% of all studies and 13% of those with negative outcomes required three or more submissions to achieve journal publication. These data suggest that rates of publication discussed in the scientific literature might substantially underestimate the effort that sponsors make to publish their results as they do not capture the number of submissions and rates of journal rejection.
Our analysis also shows that studies with negative outcomes can be and have been successfully published in the scientific literature and provides no evidence of a systematic bias against their acceptance by journals. Most (92%) studies in our cohort that had an unfavorable outcome for the drug under evaluation were submitted, and 77% had been published by the analysis cut-off date. The fact that slightly more than half of studies included in our analysis could not be readily classified into those with positive or negative outcomes, however, also suggests that such a dichotomous scheme represents an oversimplification of study outcomes, particularly when early phase studies are included. The current transparency debate and efforts might be better served by focusing on ensuring full publication of all studies, irrespective of outcome.
Strengths and limitations
Although our study has several strengths—including a large systematic cohort, inclusion of all studies (irrespective of phase or outcome), rigorous methods, and metrics on submissions as well as final publication rates—it also has important limitations. Clarity in specifying endpoints and classification of study outcomes could be inexact and subject to debate. For applicable studies, we deliberately chose to classify studies as having a positive or negative outcome for the drug, rather than on the basis of significance alone, because for some types of studies (such as corrected QT safety studies) lack of significance is a favorable outcome for the drug being tested. Conversely, significant negative safety findings could preclude further development of an investigational drug. We did not examine selective outcome reporting for the 751 studies that were published, although a unified process for reporting endpoints in study summaries and reports is followed by the sponsor to ensure consistent reporting. Most importantly, the data reported here are from a single industry sponsor. Without further data from other research sponsors, it is not possible to determine whether these results generalize to other industry sponsors or to other types of sponsors (academia, government). Despite this limitation, these data provide a clear signal that submission and publication bias against studies with negative outcomes might be less widespread than has generally been assumed and should not deter efforts to publish them.
In summary, the conduct and publication of analyses of submissions and successful publication according to study outcome are potentially important actions needed to assess and improve actual practice, and when appropriate, to correct misperceptions regarding publication bias that adversely impact the credibility of drug research. We encourage other sponsors and journal editors to share similar information in efforts to contribute to better understanding of issues and barriers to full transparency.
What is already known on this topic
Literature surveys carried out since 2010 suggest that 15-44% of studies registered on www.clinicaltrials.gov and other public sites had not been published in medical journals
Studies lacking significant outcomes were less likely to be published and, when published, took a longer time from study completion to publication
Failure of sponsors to publish all research results, regardless of outcome, adversely impacts the credibility of medical research
What this study adds
Rates of publication previously discussed in the scientific literature might have substantially underestimated the effort that sponsors made to publish their results as they did not capture sponsor submission and journal rejection rates
Submission and publication bias against studies with negative outcomes might be less widespread than has generally been assumed and should not deter efforts to publish them
Greater sharing of submission and rejection metrics by sponsors and journals could contribute to better understanding of issues and barriers to full transparency
We thank the many thousands of patients who participated in the clinical trials that comprised our dataset. We thank the medical writing team from Tata Consultancy Services (TCS) for help in classifying study outcomes and WNS Global Services Knowledge Centre for verification and formatting of publication data, both of which were paid for by GSK. TCS provided various services to GSK but had neither direct involvement in the conduct of studies included in this cohort, nor any knowledge of their publication status.
Contributors: GE conceived the project and its design, supervised the collection and analysis of data, and prepared the first draft of this manuscript and led its critical revisions. BM and BDeC participated in the collection and analysis and interpretation of data and contributed critical revisions to the manuscript. JS contributed to data analysis and interpretation and critical revisions of the manuscript. All four authors assume accountability as individuals for the final manuscript and its contents. GE is guarantor.
Funding: All studies described in this analysis were funded by GSK.
Competing interests: All authors have completed the ICMJE uniform disclosure form. All four authors were employees and shareholders in GSK at the time this research was conducted. GSK sponsored the >1000 individual studies described in this analysis, paid the salaries of the four named authors, and funded the services acknowledged above, but was not otherwise involved in the project conception or execution. BM and BDeC are members of the Medical Publishing Insights and Practices Steering Committee, which is chaired by BM.
Ethical approval: As a meta-analysis of previously completed (and registered) human research, the current analysis did not require study registration, ethics committee approval, or informed consent (the latter were obtained for the individual studies contained in the analysis).
Data sharing: Data underlying tables and figures are available on request from the corresponding author.
Transparency: The lead author affirms that this manuscript is an honest, accurate and transparent account of the research being reported, that no important aspects have been omitted, and that any discrepancies from the study as planned have been explained.
This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.