- Andrew P Prayle, NIHR doctoral research fellow,
- Matthew N Hurley, Wellcome Trust paediatric clinical research fellow,
- Alan R Smyth, professor of child health
- 1University of Nottingham, Division of Child Health, School of Clinical Sciences, Queens Medical Centre, Nottingham NG7 2UH, UK
- Correspondence to: A P Prayle
- Accepted 4 November 2011
Objective To examine compliance with mandatory reporting of summary clinical trial results (within one year of completion of trial) on ClinicalTrials.gov for studies that fall under the recent Food and Drug Administration Amendments Act (FDAAA) legislation.
Design Registry based study of clinical trial summaries.
Data sources ClinicalTrials.gov, searched on 19 January 2011, with cross referencing with Drugs@FDA to determine for which trials mandatory reporting was required within one year.
Selection criteria Studies registered on ClinicalTrials.gov with US sites which completed between 1 January and 31 December 2009.
Main outcome measure Proportion of trials for which results had been reported.
Results The ClinicalTrials.gov registry contained 83 579 entries for interventional trials, of which 5642 were completed within the timescale of interest. We identified trials as falling within the mandatory reporting rules if they were covered by the FDAAA (trials of a drug, device, or biological agent, which have at least one US site, and are of phase II or later) and if they investigated a drug that already had approval from the Food and Drug Administration. Of these, 163/738 (22%) had reported results within one year of completion of the trial compared with 76/727 (10%) trials that were not subject to mandatory reporting (95% confidence interval for the difference in proportions 7.8% to 15.5%; χ2 test, P=2.6×10−9). Later phase trials were more likely to report results (P=4.4×10−11), as were industry funded trials (P=2.2×10−16).
Conclusion Most trials subject to mandatory reporting did not report results within a year of completion.
A key principle in the good conduct of clinical trials is that a summary of the trial protocol should be freely available while the study is ongoing and that, on completion of the study, the results should be readily accessible to all in a timely fashion. In February 2000 the Food and Drug Administration (FDA) Modernization Act (1997) prompted the creation of a national clinical trials registry (ClinicalTrials.gov).1 2 Similar databases (such as the ISRCTN) have been established elsewhere. From 2005 the International Committee of Medical Journal Editors (ICMJE) required that clinical trials should be indexed in a clinical trial registry to qualify for publication in a journal following the uniform requirements for manuscripts.3 Subsequently, the FDA Amendments Act (FDAAA) of 2007 required registration of summaries of trial protocols for “applicable clinical trials” (trials that are covered by the FDAAA).4 These are trials that have at least one site in the United States; are of a drug, device, or biological agent; and are “initiated or ongoing as of September 2007,” excluding phase I studies and early feasibility trials of devices.5 Clinical trials are registered with ClinicalTrials.gov (clinicaltrials.gov) by “responsible parties” and uploaded to the website by using the Protocol Registration System (http://prsinfo.clinicaltrials.gov). The uploading of trial results is achieved in a similar fashion and reviewed by a Protocol Registration System administrator before publication on ClinicalTrials.gov. At present, clinical trials of drugs that already have FDA approval are required to report results within one year of completion of the trial (with some provisions for delayed reporting), although in the future applicable clinical trials of unapproved drugs or biological agents that are regulated by the FDA may also be required to report results.6 These results are posted in the form of a table of values for each of the pre-specified primary and secondary outcome measures for each arm of the clinical trial, with associated statistical tests.
This new legislation should help to overcome the problem of trials that are done but not reported.7 Zarin et al have recently described the current activity of ClinicalTrials.gov,8 and Wood has given a thoughtful discussion of the current legislation,6 but no systematic evaluation of compliance with the FDAAA has been published (to the best of our knowledge). Zarin et al recently commented that the “usefulness [of ClinicalTrials.gov] depends upon the research community to submit accurate, informative data.” While searching the ClinicalTrials.gov database, we noted that studies under the jurisdiction of the FDAAA had not yet reported basic results. We decided to look into this further.
Studies of a drug, device, or biological agent with a site in the United States that completed in 2009 are subject to the FDAAA regulations and are required to submit a protocol summary to ClinicalTrials.gov. Of these, most trials of drugs approved by the FDA are subject to mandatory reporting within a year. Our aim was to systematically assess the compliance with mandatory reporting on ClinicalTrials.gov of interventional clinical trials falling under the FDAAA and to look for evidence of reporting bias by using publicly available datasets.
We did a database search of trials registered on ClinicalTrials.gov which completed in 2009 and are covered by the FDAAA. To identify the subgroup of these studies for which mandatory reporting is required within one year of study completion, we cross referenced our search results with Drugs@FDA (the US database of FDA approved drugs).
Not all trials registered with ClinicalTrials.gov are covered by the FDAAA, and only a proportion of those covered by the FDAAA are required to report results within one year (“trials subject to mandatory reporting”). We identified this group of trials as follows. We searched ClinicalTrials.gov (search date 19 January 2011) for all intervention studies. We identified completed studies by using the “primary completion date” field (or, where this was not recorded, the “completion date”). The primary completion date, as defined by ClinicalTrials.gov, is the date of collection of primary outcome data on the last patient to be enrolled. The completion date is the date of completion of the study, as defined by the individual trial’s investigators. We selected trials subject to mandatory reporting (within one year) which were completed between 1 January and 31 December 2009. We chose these dates because all the studies completing within this timeframe would have had at least one year in which to report results.
We excluded phase 0 and I trials, as well as trials that did not report a phase. We used a script to programmatically obtain the study site(s) for each trial. We excluded studies that did not have a site in the United States. Additionally, we excluded studies that were not clinical trials of a drug, biological therapy, or device. The remaining studies therefore consisted of trials that are covered by the FDAAA.
Trials of drugs that are not approved by the FDA at the time of the 12 month deadline are not yet required to be reported. We cross referenced the investigational drug for each of the applicable clinical trials with the FDA database of approved drugs, Drugs@FDA (www.accessdata.fda.gov/scripts/cder/drugsatfda/index.cfm), to identify trials of approved drugs (hereafter termed “trials subject to mandatory reporting”—see data supplement (http://dx.doi.org/10.5061/dryad.j512f21p) for further description and examples of this classification). Owing to the nature of the records available on Drugs@FDA, this removed devices and some biological agents from the mandatory reporting dataset. For the primary analysis, we categorised a drug as being FDA approved if the active compound listed on ClinicalTrials.gov was listed as an approved drug on Drugs@FDA. For example, if tobramycin was administered by injection in the trial, we classified this as an approved drug, as tobramycin products administered by injection are approved on Drugs@FDA. However, for a trial in which tobramycin was administered as a dry powder for inhalation, we did not classify this as an approved drug, as this formulation was not approved on Drugs@FDA at the time of our search. One investigator (APP) classified all drugs. Another investigator (MNH) classified a 10% sample of these to measure inter-rater agreement.
We occasionally had difficulty in deciding whether a generic drug listed in a ClinicalTrials.gov record was in fact a formulation approved by the FDA. For this reason, we did a subgroup analysis in which only drugs approved by the FDA and identified by brand name on ClinicalTrials.gov were considered approved drugs.
We compared the reporting rate of trials subject to mandatory reporting with trials that fall under the FDAAA but are not subject to mandatory reporting. We examined the relation of phase of study and funder of the trial with reporting of results for trials subject to mandatory reporting.
We imported the dataset from ClinicalTrials.gov into Microsoft Excel. We used R (R-Foundation for Statistical Computing, Vienna, Austria; version 2.12.1) to “webscrape” additional information from the ClinicalTrials.gov website.9 We analysed the final reported dataset by using R. Further details of the categorisation of trials, the complete dataset, and R scripts to produce the results that we report here are available as data supplements (http://dx.doi.org/10.5061/dryad.j512f21p).
At the time of our search, the ClinicalTrials.gov registry included 83 579 entries for intervention studies (figure⇓). Of these, we identified 31 556 as having “completed,” and 5642 had completion dates between 1 January and 31 December 2009. We excluded phase 0 and I studies (n=1098), as well as trials that did not indicate a phase of clinical study (1170), those that did not have a US site (1752), and trials that were not of a device, drug, or biological agent (157). This established a group of 1465 trials covered by the FDAAA, for which more than a year had passed since completion in which to report results.
Reporting for applicable trials is (at present) mandatory only for drugs, biological agents, or devices approved by the FDA. By cross referencing with Drugs@FDA (www.accessdata.fda.gov/scripts/cder/drugsatfda/index.cfm), we found that 738 trials covered by the FDAAA were subject to mandatory reporting. APP categorised drugs from trials as previously approved by the FDA. MNH repeated this classification for a sample of 150 (10%) of the records. Disagreement resolved by discussion led to a change in classification in 3/150 (2%) of records.
Of the 738 trials that were classified as subject to mandatory reporting, 163 (22%) had reported results. In comparison, 76/727 (10%) trials covered by the FDAAA but not subject to mandatory reporting had reported results (95% confidence interval for the difference in proportions 7.8% to 15.5%; χ2 test, P=2.6×10−9).
The proportion of trials subject to mandatory reporting that had reported results was influenced by the phase of the study (table 1⇓). Fewer phase II trials had reported results compared with phase III and IV trials (χ2 test, P=4.4×10−11). We categorised the funder of the trial as industry, mixed, National Institutes of Health/government, or other and found that the funder of the trial influenced the proportion that had reported results (Fisher’s exact test, P=2.2×10−16) (table 2⇓). Industry funded trials were more likely to report results. We therefore grouped trials subject to mandatory reporting into “solely industry funded” and “not solely industry funded.” More solely industry funded studies (126/317; 40%) had reported results than had those not solely industry funded (37/421 (9%); 95% confidence interval for the difference 24.7% to 37.3%; χ2 test, P=2.2×10−16).
We did a subgroup analysis to determine the proportion of trials that gave the brand name of the investigated drug on the ClinicalTrials.gov record (for which we had more confidence in our categorisation of the drug as being approved by the FDA) which reported results. Of the 738 drug trials that we had classified as being subject to mandatory reporting, 347 trials gave the brand name for each and every drug in the trial. Of these 347 trials, 96 (28%) had reported results.
Reporting of summary results on ClinicalTrials.gov is an important step forward in reducing bias in the literature. We have cross referenced the ClinicalTrials.gov and Drugs@FDA databases to produce a dataset of trials that should have reported results at the time of our search, and we found that only 22% of trials had done so. If the reporting rate does not increase, the laudable FDAAA legislation will not achieve its goal of improving the accessibility of trial results.
The influence of funding body and sponsor seems to be considerable. Industry funded trials subject to mandatory reporting were more likely to report results compared with other funders. Phase III and IV studies seem more likely to be reported than phase II studies.
Comparison with literature
Many potential sources of bias may skew the literature.10 11 Apart from citation bias, these biases can be ascribed to the investigator, sponsor, and publisher. Although studies with positive results are published more quickly,12 13 14 no significant relation seems to exist between trial results and time between submission of a manuscript to a journal and publication.15 ClinicalTrials.gov allows dissemination of summary results independent of a publisher. Our study supports the suggestion that study investigators and sponsors act as the principal sources of reporting bias; reporting of results to ClinicalTrials.gov is independent of peer review, manuscript preparation, and editorial priorities.
Strengths and limitations of study
The FDAAA requirements for mandatory reporting came into force during 2008. By selecting studies that completed during 2009, and doing our search in early 2011, we have ensured that the trials completed during the period when the FDAAA was applicable and that at least one year had elapsed for data to be submitted to ClinicalTrials.gov. We are reliant on responsible parties from each study uploading accurate data to ClinicalTrials.gov. We believe that our search in January 2011 represents the earliest reasonable time to do our study. We anticipate (and hope) that as more investigators become familiar with the legislation, reporting rates will increase. As we present a cross sectional study from a single year, we cannot comment on trends in reporting of results.
An important limitation of the study is that we have identified trials subject to mandatory reporting through publicly available summary data on the trials. We categorised a trial as covered by the FDAAA if it had at least one site in the United States, was Phase II or later, and investigated a device, drug, or biological agent. The possibility remains of misclassification of trials as either FDAAA applicable or FDAAA non-applicable, owing to the nature of the information available to us on ClinicalTrials.gov.
We identified a group of trials subject to mandatory reporting by cross referencing the ClinicalTrials.gov record with Drugs@FDA (the online database of FDA approved drugs). This was straightforward when the brand name of the drug was listed in the ClinicalTrials.gov record and that product had FDA approval. Similarly, categorising drugs as non-FDA approved when the brand name drug was not listed on Drugs@FDA or when a generic form with a different formulation or route of administration to the FDA approved product was used in the trial was a simple process. We had more difficulty in categorising a drug administered in a trial if the generic name given in the ClinicalTrials.gov record was approved by the FDA, as the approved drug may not necessarily have been the formulation administered in the trial. In these cases, we took an inclusive approach and included these trials in our mandatory reporting required group. We did a subgroup analysis using only trials of drugs for which the brand name of the drug was given in the trial (thus eliminating uncertainty as to whether the formulation in the trial was or was not FDA approved). In this analysis, 28% of trials had reported results, broadly similar to the 22% that we found with the more inclusive approach.
We have no data on the number of studies that applied for exemptions from the requirement for reporting. Industry funded studies are perhaps most likely to form part of an application for licensing/marketing, and therefore most likely to apply for an exemption, but we have found that industry funded studies are more likely to report results than are studies funded by other means. We also note that trials investigating new indications for drugs previously approved by the FDA are not required to upload results until two years after completion, but our methods did not allow us to identify these trials.
Although these various effects could increase our denominator and falsely decrease the overall reporting effect, we believe that this is unlikely to account for approximately 78% of the trials not reporting. Phase IV trials are (by definition) trials of drugs that have marketing approval, and so are of an FDA approved drug, and the indication studied in the trial should be the same as the approved indication. All of these studies should have reported results within one year, and yet only 31.3% had done so (table 1⇑). Although our denominator is necessarily somewhat uncertain, owing to the nature of the data that are publicly available, we believe that many clinical trials that should have reported basic results had not.
Conclusions and policy implications
Through studying the ClinicalTrials.gov database, which mandates the reporting of outcomes of completed but unpublished studies, we believe that an understanding of the extent of bias in the literature due to unpublished studies can be gained. This will be of great benefit to the clinicians who prescribe new treatments, the patients who take them, healthcare funders, and especially researchers doing systematic reviews of treatment interventions which inform all of the above. The availability of both the protocol and results in outline form helps to overcome both outcome reporting bias and publication bias.
We report a systematic approach to examining compliance with mandatory reporting. In a study such as ours, which uses only publicly available data and cross references one dataset to another, some debate will inevitably take place about whether we have accurately identified trials that fall under the FDAAA requirement for mandatory reporting. Within the limitations of our study design, we have identified a group of trials of which the vast majority should have reported summary results within a year and had not done so.
We believe that the FDAAA is an important milestone on the path to a future in which all the available evidence can be used to make decisions about treatment, and we applaud the farsighted nature of the legislation. We have found that trials subject to mandatory reporting are more likely to have reported results compared with other trials, illustrating the positive effect of the legislation and efforts of ClinicalTrials.gov. Nevertheless, overall reporting is poor.
What is already known on this topic
Reporting bias prevents the dissemination of results of clinical trials
Where trial data are not accessible to practitioners, clinical decisions cannot be based on the best evidence and may be flawed
Recent US legislation mandates publication of a results summary on ClincalTrials.gov for recent trials of drugs approved by the Food and Drug Administration and should improve access to trial results
What this study adds
At the time of the search, many trials that should have published data on ClincalTrials.gov had not done so
Cite this as: BMJ 2012;344:d7373
Contributors: APP, MNH, and ARS designed the study. APP wrote the script to webscrape additional data from ClinicalTrials.gov. APP categorised the trials, MNH cross checked a subset, and ARS arbitrated. All authors had full access to the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis. All authors wrote the manuscript. APP and MNH contributed equally to this work. ARS is the guarantor.
Funding: The study was not externally funded.
Competing interests: All authors have completed the ICMJE uniform disclosure form at www.icmje.org/coi_disclosure.pdf (available on request from the corresponding author) and declare that APP is supported by a National Institute for Health Research fellowship (DRF-2009-02-112) and MNH is supported by a Wellcome Trust fellowship (WT092295AIA). ARS declares relevant activities outside the submitted work of membership of a REMPEX steering committee, consultancies for Novartis and Biocontrol, and a lecture paid for by Chiesi Pharma. ARS has registered trials on ClinicalTrials.gov and other registries.
Ethical approval: Not needed.
Data sharing: Technical appendix, statistical code, and dataset available from the corresponding author (http://dx.doi.org/10.5061/dryad.j512f21p. Participants’ consent for data sharing not required.) or at
This is an open-access article distributed under the terms of the Creative Commons Attribution Non-commercial License, which permits use, distribution, and reproduction in any medium, provided the original work is properly cited, the use is non commercial and is otherwise in compliance with the license. See: http://creativecommons.org/licenses/by-nc/2.0/ and http://creativecommons.org/licenses/by-nc/2.0/legalcode.