Accessibility of trial reports for drugs stalling in development: a systematic assessment of registered trials
BMJ 2015; 350 doi: https://doi.org/10.1136/bmj.h1116 (Published 09 March 2015) Cite this as: BMJ 2015;350:h1116
- Amanda Hakala, master’s student1,
- Jonathan Kimmelman, associate professor1,
- Benjamin Carlisle, doctoral student1,
- Georgina Freeman, research assistant1,
- Dean Fergusson, senior scientist2
- 1Studies of Translation, Ethics, and Medicine (STREAM), Biomedical Ethics Unit, Department of Social Studies of Medicine, McGill University, H3A 1X1, Montreal, Canada
- 2Centre for Practice Changing Research, Ottawa Hospital Research Institute, University of Ottawa, K1H 8L6, Ottawa, Canada
- Correspondence to: J Kimmelman jonathan.kimmelman{at}mcgill.ca
- Accepted 20 January 2014
Abstract
Objective To quantify the proportion of trials for unsuccessfully licensed drugs that are not published.
Design A systematic assessment of the availability of published research reports for publicly registered trials testing drugs stalling in clinical development (“stalled drugs”) and drugs receiving regulatory licensure in the same time period (“licensed drugs”).
Data sources Searches of clinicaltrials.gov, Google Scholar, PubMed, Embase, and electronic query of contacts in registries to identify trials and assess publication status.
Eligibility criteria The cohort of licensed drugs consisted of all disease modifying drugs in cancer, cardiovascular disease, or neurological disorders that received Food and Drug Administration licensure during 2005 to 2009 inclusive. The cohort of stalled drugs included unlicensed drugs in the same disease areas that had at least one completed phase III trial before 2009 and no evidence of any clinical trial activity after 31 December 2009. Inclusion criteria for registered trials advanced into publication searching were an intervention tested in a disease modifying, phase II, III, or IV trial registered on clinicaltrials.gov, with enrolment of at least one patient, that reported primary outcome collection end date between 1 January 2006 and 31 December 2008.
Results The unadjusted publication proportion for registered trials of licensed drugs was 75% (72/96) versus 37% (30/81) for stalled drugs. The adjusted hazard ratio for publication was 2.7 (95% confidence interval 1.7 to 4.3) in favour of licensed drug trials. Higher publication rates for licensed drug trials were observed regardless of disease type, sponsorship, trial phase, and geography. The rate of non-publication of stalled drug trials was significantly higher for studies that did not complete enrolment compared with licensed drug trials. A total of 20 135 patients participated in trials of stalled drugs that were never published.
Conclusions Much of the information collected in unsuccessful drug trials is inaccessible to the broader research and practice communities. These findings provide an evidence base and rationale for policy reforms aimed at promoting transparency, ethics, and accountability in clinical research.
Introduction
Many commentators describe the contemporary era as the “age of information,” and in medicine, the information most relevant for guiding clinical and policy decision making is trial evidence. Yet, despite a vigorous research enterprise, a large fraction of evidence generated in trials is not published or poorly reported. For instance, Bero et al observed that only 78% of efficacy trials submitted to the Food and Drug Administration (FDA) for approved new drug applications were published, of which one third omitted the primary outcome.1 Of the outcomes that did not favour the test drug, 47% were not published. Such underreporting frustrates evaluation of trial methods, the balanced appraisal of an intervention’s clinical utility,2 and timely recognition of safety issues.3 Non-publication also fails to honour the sacrifice of patients who participate in such studies.
Discussion of non-publication and reporting has centred on trials of interventions used in practice. Policy makers and others have devised various measures to promote deposition of trial information.4 However, the vast majority of drugs entered into clinical development are never licensed.5 The process of clinical translation typically involves a series of trials, starting with phase I (which test safety and dose in a small population), continuing to phase II (aimed at detecting potential for efficacy), and then phase III (aimed at providing sufficient evidence of clinical utility to justify regulatory approval). Existing policy measures that promote publication have little traction on such trials if products are not ultimately licensed.
Trials in unsuccessful translation trajectories contain a wealth of information for research planning. For instance, they provide an indication of whether pathophysiological theories driving drug development efforts are valid. Where drugs are members of a class that has licensed products, they also provide valuable information about safety and pharmacology. Moreover, publication of unlicensed drug trials helps ensure that risks of exposure to unsafe or ineffective drugs—and the social expenditures associated with such trials—are redeemed by a gain of scientific knowledge.
Recently, the US Health and Human Services proposed policy revisions that would oblige drug developers to deposit trial findings in clinicaltrials.gov regardless of the licensure status of test products.6 7 In this study, we provide evidence to support such policies by systematically assessing the availability of research reports for drugs that stalled in clinical development.
Methods
Our primary objective was to evaluate publication success for registered trials of recent “stalled” drugs—defined as those that reached late stage clinical testing (phase III) but were discontinued from further development. We defined such stalled drugs by the absence of registered or ongoing clinical trials after 31 December 2009 to allow at least four years of development inactivity. As a basis of primary comparison, we also assessed publication success for trials involving drugs that ultimately received regulatory licensure in the same time period (“licensed drugs”). Our primary outcome, publication success, was defined as a full journal publication.
Study sample
To establish a basis for building our sample, we focused on drugs for which the primary indication categories driving translation efforts were cancer, cardiovascular disease, or neurological disorders. Our licensed drug cohort consisted of all new molecular entities (NMEs) licensed by the FDA in the three domains (neurological, cardiovascular, and cancer) between 2005 and 2009 inclusive. These drugs were identified on the basis of FDA approval documents and had phase III trials that took place between 1998 and 2009. This cohort was used as an anchor for constructing the cohort of stalled drugs.
Our cohort of stalled drugs was created to capture unsuccessful NMEs that were contemporaneously pursued. It consisted of investigational new drugs that reached phase III trials by 2009 but stalled between 2005 and 2009, allowing for at least 4.5 years of trial inactivity before publication searches were conducted. The NMEs were obtained via an automated search of clinicaltrials.gov for studies with keywords for neurological disease, cardiovascular disease, and cancer; intervention names were exported to an Excel file (see supplemental methods for details of our protocol for populating cohorts). Drug names were consolidated and licensed drugs excluded. Clinicaltrials.gov was subsequently searched manually to identify the completion date of drug development (defined as the primary completion date of the last study, or, in order of priority, the study completion date, the completion date, the anticipated primary completion date, or the anticipated study completion date of the last trial). Withdrawn trials were not counted; instead the completion date of the last trial of any other status (completed, terminated, unknown, or active and not recruiting) was used for completion date of drug development identification purposes. NMEs with completion dates falling between 1 January 2005 and 31 December 2009 were advanced to publication searching. Lastly, we performed a search of Factiva and Google Scholar for press releases documenting the discontinuation of NMEs (BC, GF). When a drug was confirmed as discontinued, it was classified as a “gold standard” stalled drug. In populating the stalled drug cohort to parallel the licensed drug cohort, we pooled all stalled drugs, as the gold standard drug trials alone were insufficient to populate our cohort. Using this protocol, we captured 34 unlicensed drugs that had at least one completed phase III trial before 2009 and no evidence of any trial activity as of 31 December 2009. All drugs must have tested for disease modification in our three indication categories (for example, we excluded from our cancer cohort antiemetic drugs to treat chemotherapy-induced nausea).
We next searched clinicaltrials.gov for all trials involving each drug in our cohorts. Some drugs had an unusually large volume of registered trials. To ensure our findings addressed general publication patterns for drugs rather than those for a small group of extensively trialled drugs, we reduced the number of studies captured in our search by capping trials per drug in our sample at six for neurological and cardiovascular drugs and four for cancer drugs (which were more numerous). If the number of eligible trials for a drug exceeded the cap, we selected from eligible trials using a random number generator.
Inclusion criteria for registered trials that were to be assessed for publication status were (a) an intervention tested in a disease modifying trial, (b) a phase II, III, or IV trial registered on clinicaltrials.gov, (c) enrolment of at least one patient, and (d) the reported end date for primary outcome collection was between 1 January 2006 and 31 December 2008 (affording at least 5.5 years between trial end date and our literature search).
We extracted the following information from registry entries: start date, end date (defined as final data collection for the final primary outcome measure), location, indication, sponsor, projected accrual, actual patient accrual, phase, completion status, and whether results were deposited on clinicaltrials.gov. Since authors infrequently update records on clinicaltrials.gov,8 our sample included studies registered as “completed,” “terminated,” “unknown,” and “active, not recruiting.”
Publication status
We performed several searches to assess publication status. First, we checked clinicaltrials.gov records for links to publication. Second, we searched Google Scholar using the unique clinicaltrials.gov identifier assigned to each trial. Third, we searched PubMed and Embase using drug name (and variants), indication, subindication, study design (that is, randomised, double blind), and comparator. We then confirmed publication identity by comparing sample size, trial arms, doses and administration schedules, comparators, and primary investigator names to the registration record. A trial was considered published only if the study had appeared as a full publication in a journal, though we also recorded abstract-only publication. Lastly, we queried investigators named in registration records. Four electronic queries were sent at 1-2 week intervals if we were able to find an email address. The final literature search was performed by AH in June 2014. For quality control, GF independently verified the publication status of 10% of the trials in our sample; there were no discrepancies. Our query to investigators received ethical approval by the McGill University Institutional Review Board.
To determine whether trials of stalled drugs were published with greater delay than those for licensed drugs, we examined time elapsed between the registered end date (last primary outcome data collection) and publication date (excluding advanced online publication). We also tested in an exploratory manner whether non-publication of stalled drug trials was more common for studies that were solely privately funded, earlier phase, and that did not complete enrolment. We followed the terminology defined on clinicaltrials.gov and considered all entities listed under the “sponsors” and “collaborators” headings in the trial record as funders of the trial. We subsequently categorised trials by whether they were solely funded by private, for profit companies. We defined incomplete enrolment, or under-accrual, as <85% of projected enrolment at trial registration.
Statistics
Unadjusted analysis of publication success in each cohort (stalled versus licensed) was conducted simply as a comparison of the proportion of trials with a full journal publication. To appropriately adjust for time to publication or non-publication, potential confounding risk factors, and clustering at the drug and disease level, we performed an adjusted analysis. Specifically, the effect of licensed versus stalled drug trials on time to publication was analysed using a Cox proportional hazards model accounting for intra-cluster dependency at the drug level and covariates such as sample size, indication, sponsor, trial phase, and geographic location. The analysis was done using SAS (9.3) (SAS Institute, Cary, NC, USA). We tested the effect of publication success between licensed and stalled drug trials within a number of a priori subgroups including funding status, trial phase, condition, actual versus expected accrual, and study location. We defined significance as P≤0.05 and did not correct for multiplicities.
In approximating sample size for our study, we assumed an α of 0.05, a β of 0.20, and hypothesised that 65% of trials for licensed drugs are published versus 45% of trials for stalled drugs. With these assumptions unadjusted for potential confounding and time to publication, we required 96 trials in each cohort to demonstrate a statistically significant decrement of publication for stalled drugs.
Results
Descriptive
We identified a total of 177 eligible trials for our sample, which comprised 25 licensed and 34 stalled drugs (see supplementary table S1 in online appendix). Our trial sample captured a total of 266 licensed drug trials, from which we randomly sampled 96 trials for publication searching (see supplementary fig S2), and 104 stalled drug trials, from which we randomly selected 81 for publication searching. In general, the volume of registered trials was far greater for licensed drugs than unlicensed ones, and for cancer drugs versus non-cancer drugs. On average, four eligible trials were registered for each licensed drug and 2.4 eligible trials were registered for each stalled drug. The table⇓ shows the characteristics of the trials advanced into publication searches. The mean time between trial closure and our search was 80.9 months for licensed drugs (range 66–101 months) and 84.7 months for stalled drugs (range 66–101 months).
Characteristics of clinical trials included in comparison of availability of published research reports for drugs that stalled in clinical development and drugs that were licensed for use. Values are numbers (percentages) unless stated otherwise
Analysis of publication success
Unadjusted publication success for registered trials of licensed drugs was 75% (72/96) compared with 37% (30/81) for trials of stalled drugs (risk ratio 1.55 (95% confidence interval 1.09 to 2.19)). An additional four licensed drug trials and five stalled drug trials were published as abstracts only (risk ratio 1.46 (1.06 to 2.02)). None of the stalled drug trial results that were unpublished were deposited on clinicaltrials.gov. Results of 15% (14/96) of licensed drug trials and 57% (46/81) of stalled drug trials were not publically accessible in any form (full publication, abstract, or results deposited in clinicaltrials.gov). Based on registration records, the total number of patients participating in trials in our two cohorts was 110 168. The proportion of patients that participated in trials that were not reported in full publication was 11% for licensed drugs (5543 participants) and 34% for stalled drugs (21 135 participants). The mean time elapsed between the registered end date (last primary outcome data collection) and date of publication (excluding advanced online publication) was 29.0 months (range 0–75 months) for approved drugs and 28.6 months (range 5–57 months) for unlicensed drugs. This analysis excluded interim reports or studies published before the primary completion date listed on clinicaltrials.gov.
Our adjusted estimate of publication success demonstrated a 2.7-fold greater rate of publication success for trials of licensed drug compared with stalled drug trials (hazard ratio 2.71 (1.72 to 4.26)) (figure⇓).
Fig Rates of publication of trials of licensed drugs compared with trials of stalled drug: overall and by major subgroups
Attempts to reach contacts in trial registries to confirm non-publication were mostly unsuccessful. Of the 75 unpublished trials across both cohorts, we found contact information for 42 trials (11 for licensed drugs, 31 for stalled drugs). We received nine responses to our 42 queries (four for licensed drugs, five for unlicensed drugs); no publications not captured in our searches were identified, and respondents stated no intent to publish. We were unable to obtain any publications for trials sampled from two licensed drugs (8%, 2/25) and 15 stalled drugs (44%, 15/34). Of these 15 stalled drugs, seven had less than five published trials on PubMed when we searched all registered trials; many of these were phase I trials or pharmacokinetic/pharmacodynamics studies. For four stalled drugs, we were unable to identify any published trials by searching PubMed using the name of the drug used in registration records.
Subgroup analyses
Publication rates in favour of licensed trials were consistent across all disease types although a non-significant increase was seen for stalled neurological drugs (figure⇑). The rate of publication success was higher in licensed trials regardless of funding status, although the rate was comparatively higher for trials with private for profit funding only compared with others. Publication rates were higher for licensed drug trials regardless of trial phase (II or III) and geography (North America or other).
Licensed drug trials that under-accrued (<85% of projected patient enrolment) had a higher publication rate than stalled trials (hazard ratio 3.9 (95% confidence interval 1.3 to 11.8)). This analysis excluded the 17 licensed drug trials and eight stalled drug trials for which the registry information was insufficient to calculate a percentage of target accrual. However, 11 (64.7%) and two (25%) of these excluded studies, respectively, were published in a full report.
Discussion
We use a sample of new cancer, neurological, and cardiovascular drugs to show that rate of publication success for registered trials of licensed drugs was nearly three times higher than that for trials of stalled drugs. Indeed, for 44% of stalled drugs in our study, no full reports were available at all for the sampled trials, and for four stalled drugs, no published trials at all were accessible on PubMed. These figures are lower than previous estimates of publication for trials reporting non-significant results of FDA licensed products (66%).9 The very low publication rate, coupled with the low response rate of lead investigators, suggests that much of the information collected in unsuccessful drug translation trajectories is inaccessible to the broader research community.
This has troubling policy implications. Trials are ethical only insofar as risks are outweighed by benefits to subjects, if any, and to society in the form of generalisable knowledge. Information that is not accessible through publication cannot be absorbed into a broader body of scientific knowledge. This failure to take even the initial steps at redeeming burdens of over 20 000 patients in our cohort alone runs contrary to major codes of ethics,10 11 and is troubling in the context of drugs where trial participation is likely to have involved exposure to unsafe or ineffective drugs. Moreover, non-publication deprives research systems and caregivers of state of the art evidence.
Value of information from trials of stalled drugs
It might be argued that information from unsuccessful drug translation trajectories has little value for research and healthcare systems, given that products are not available for clinical application. That this information is highly valued, however, is suggested by the fact that the European Medicines Agency (EMA) frequently receives queries for trial results of unlicensed drugs.12 There are at least four ways that evidence from unsuccessful translation trajectories informs both research and care. First, discordance between prior findings and trial outcomes provides feedback on experimental systems such as preclinical models or surrogate outcomes.13
Second, evidence from new drugs informs the planning of other trials, including those for other drugs in the same class. With the rising costs of drug development, many drug developers are turning to literature based discovery to identify novel indications for abandoned drugs,14 and such analysis requires liberal access to trial information. Discovery of class effects can inform the planning of drug development and design of trials—thereby better protecting patients in later trials.
Third, some stalled drugs are later repositioned for different indications,15 16 combined with a different delivery method,17 or used as a chemical platform for further drug development. Safety and pharmacodynamics findings are vital for the planning of such repositioning.
Fourth, evidence from unsuccessful translation trajectories has applications in clinical medicine. Drugs not approved in the US are sometimes approved in other jurisdictions. Stalled drugs also share properties with products that are licensed. Safety information informs pharmacovigilance or medical countermeasures; pharmacodynamics can inform a better understanding of how to maximise the clinical utility of a drug in the same class. Trials of abandoned drugs also contain components that are already in clinical application. Such studies might involve active comparators, the use of clinically validated diagnostic methods, or a combination of a stalled drug with an established modality such as surgery. The data on established components within stalled drug trials are of value for healthcare systems as well. For many sorts of questions, data from unlicensed translation trajectories can and should be incorporated into systematic reviews.
Study limitations
Our findings should be interpreted in light of several limitations. First, we sampled only drugs in three disease areas. However, these diseases areas are broad, diverse, and major foci of drug development—and hence provide reasonable proxies for generalised trends in drug development. Second, it is also conceivable that, with time, more trials of stalled drugs in our sample will be published. Third, our sample size was limited by the number of approved and stalled drugs in our time window. However, our selection of time window was constrained by the need for a current sample that nevertheless allowed sufficient time for follow up of publication. Fourth, our study was hampered by limitations in trial registries and low response rates. Drugs are not linked by synonyms, necessitating extensive searching—including searches for misspelt drug names. Contact information on registration records is also very limited. These limitations point to the continuing need for publication, not registration, as a primary means of reporting trial results.
Implications for policy
Several initiatives are aimed at promoting accessibility of trial evidence for unlicensed drugs,18 including research partnerships between pharmaceutical companies and academic institutions.19 20 Current FDA policies do not mandate publication or disclosure of results from trials for unlicensed drugs; European Medicines Agency (EMA) policy likewise does not require publication of clinical trial results. As of 1 January 2015, the EMA has implemented a new clinical trial transparency policy that will involve publication of clinical study reports submitted to the EMA in support of a marketing authorisation application. However, this regulation does not extend to the publication of trials of drugs that stall earlier in development before marketing application. Likewise, the new Clinical Trials Regulation EU No 536/2014 passed by the European Parliament and scheduled to take effect in mid-2016 will only mandate the publication of clinical study reports for trials used in marketing authorisation applications. However, detailed and lay summaries will be required for every trial with a location in the European Union.21
The US is currently considering policy that would, for the first time, oblige drug developers to deposit in public databases essential findings from unsuccessful drug development trajectories.7 To our knowledge, our study is the first measure of the volume of information lost in unsuccessful drug development and provides an evidentiary rationale for current rulemaking. We further suggest two policy changes. First, in keeping with human protection policies that require trial publication,12 institutions hosting studies should condition trial approval on an affirmative statement that sponsors will publish all results in full within two years of the last datum collection of the primary endpoint. Second, public funding agencies such as the National Institutes of Health or Canadian Institutes of Health Research should buttress such policies by conditioning transfer of funds to institutions on adherence to obligatory data deposition policies.
Publication need not involve the expenses of preparing manuscripts and disclosure of commercially sensitive information. The latter may be diminished by policies establishing a level playing field for disclosure. In the absence of economic analyses measuring social costs against social benefits, there is little evidence to support the ongoing practice of withholding evidence from stalled drug trajectories. Research ethics suggest the burden of proof should be on defenders of the status quo.
What is already known on this topic
Many completed clinical trials are never published
Such non-publication distorts the evidence base for developing new drugs and delivering care, and violates many research ethics policies
About 80–90% of drugs put into trials are never licensed
What this study adds
This study of registered trials finds that, more often than not, the results of trials for unsuccessful drugs are never published
This is ethically troubling, given that such trials expose patients to unsafe or ineffective drugs but are not redeemed by any gain in scientific knowledge
Notes
Cite this as: BMJ 2014;350:h1116
Footnotes
Contributors: JK conceived the idea for the article and performed a preliminary study of trial availability for licensed drugs. DF helped JK with study design and provided statistical planning and analysis. BC and GF in consultation with JK elaborated a search strategy and built the cohort of stalled drugs. GF and AH constructed the trial cohorts. AH performed the literature searches. GF double coded the literature search for 10% of the trials in each cohort. BC constructed a database to handle email queries. AH queried trial contacts. JK and DF designed and AH and DF performed the analysis of the data. AH and JK wrote the article. All authors edited, reviewed, and approved the final version. JK is the guarantor of the study.
Funding: This work was funded by the Canadian Institutes of Health Research (EOG111391). The study sponsor played no role in study design; collection, analysis, or interpretation of data; writing of the article; or the decision to submit the manuscript for publication. All researchers are independent from the funder.
Competing interests: All authors have completed the ICMJE uniform disclosure form at http://www.icmje.org/coi_disclosure.pdf. DF and JK declare grants from the Canadian Institutes of Health Research for the submitted work. JK also declares grants from Genome Canada. All authors declare no financial relationships with any organisations that might have an interest in the submitted work in the previous three years and no relationships or activities that could appear to have influenced the submitted work.
Ethical approval: Our study received ethical approval from the McGill University Institutional Review Board. Contacts queried about trial publication status were informed that response would be interpreted as consent to participate in the study.
Transparency: The guarantor affirms that this manuscript is an honest, accurate, and transparent account of the study being reported; that no important aspects of the study have been omitted; and that any discrepancies from the study as planned have been explained.
Data access: All authors had full access to the data and can be held responsible for its accuracy. The corresponding author agrees to share all data files upon request.
This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.