Clinical trial design and dissemination: comprehensive analysis of clinicaltrials.gov and PubMed data since 2005BMJ 2018; 361 doi: https://doi.org/10.1136/bmj.k2130 (Published 06 June 2018) Cite this as: BMJ 2018;361:k2130
All rapid responses
Re: Clinical trial design and dissemination: comprehensive analysis of clinicaltrials.gov and PubMed data since 2005
Dr. Crawley's excellent critique and questions about the weaknesses in the Zwierzyna, Davies, Hingorani, Hunter clinical trials.gov data comprehensive analysis touch on the critical matters of verifiability, replicability, and potential reporting bias.
We are constrained to parsing the descriptive characteristics of research, the research sponsors, and publication outcomes until medical journal editors take action to require that researchers and their sponsors provide access to anonymized data and code books to independent researchers so that they can address essential questions relating to verifiability, replicability, and potential reporting bias. Until this happens registered studies will remain out of reach and unexamined (1).
Clearly, mandated registration of studies, while important, is not enough, and the current registration rate of 50% in clinicaltrials.gov bespeaks the impotency of US law and government regulators.
Is there anything more that can be done to convince medical journal editors that they are uniquely positioned to overcome government inertia and to take control of the future progress of biomedical research and evidence based medicine?
1. Noble, JH. Transparency as a condition for JRSM publication? JRSM 2014; 107(7): 258. Doi: 10.1177/0141076814541772.
Competing interests: No competing interests
Questions raised by the Zwierzyna, Davies, Hingorani, Hunter clinicaltrials.gov data comprehensive analysis
Aggregated data from clinical trials registries has the potential to provide insight into clinical trial designs, results, and trends according to trial characteristics and trial locations. The comprehensive analysis performed by Zwierzyna, Davies, Hingorani, and Hunter shed light on characteristics of the clinical trials registered with clinicaltrials.gov. The methodology employed by these authors as well as their call for further action by the International Committee of Medical Journal Editors (ICMJE) also gives rise to questions; questions regarding just who is investing in, and leading in, clinical trials; questions regarding clinical trials transparency; and questions regarding the scientific return on the investment in clinical trials and the public health value of clinical trials generally.
Firstly, the authors limited their analysis to clinicaltrials.gov and did not include the primary registries of the World Health Organization's International Clinical Trials Registry Platform (ICTRP), including the European Union Clinical Trial Registry (EudraCT). The inclusion of these primary registries would have provided a far more comprehensive 'comprehensive analysis', and likely also have shed some light on global trends and gaps in the registration of clinical trials.
Secondly, the methodology for assigning clinical trial categories appears skewed. As the authors state: 'if NIH was listed either as the lead sponsor of a trial or a collaborator, we classified the study as NIH funded. If NIH was not involved, but either the lead sponsor or a collaborator was from industry, we classified the study as funded by either big pharma if it included any big pharma organisation or otherwise as small pharma. Remaining studies were assigned “other” as the funding source.' Thus, apparently whether or not NIH was the major funder of a clinical trial, so long as NIH was in one way or another mentioned as either 'lead sponsor' (which can indicate a number of possible roles, not necessarily including funding) or as 'collaborator', the clinical trial was put under the category 'NIH'. With the clinical trials that remained, the same method was used to assign clinical trials to 'big pharma' and then to 'small pharma'. However, this method of categorising studies provides no reliable information on just who is taking the leading role in the clinical trials and who ultimately is responsible. (It does make NIH look good.)
One asks further why NIH studies alone were so categorised. Why were not European Commission funded clinical trials or the European Union & Developing Countries Clinical Trials Partnership (EDCTP) funded clinical trials not so categorised, or perhaps grouped with NIH and other public funding organisations? The categorising results in a further problem of accuracy: the distribution percentages these authors arrive at are far different than the monthly reporting from EudraCT where consistently since 2004 79% of clinical trials are reported as 'commercial' and 21% as 'non-commercial' (https://eudract.ema.europa.eu/statistics.html).
The reader is provided no clear insight as to why the number of clinical trials in the clinicaltrials.gov database is increasing and, more importantly, to what extent clinicaltrials.gov captures all clinical trials taking place in the USA and globally.
The authors do shed light on the large number of clinical trials in clinicaltrials.gov database for which no results are reported. Their analysis suggests that just under 50% of the clinical trials registered in clinicaltrials.gov were of no value, made no contribution to science and ultimately no contribution to public health. To the extent that this is the case, these trials with unreported results are unethical.
The authors' suggestion (following Anderson and colleagues) for improving the reporting of clinical trial results is not well considered: 'journal editors from ICMJE could call for submission of results to clinicaltrials.gov as a requirement for article publication'. Firstly, it would be wrong for the ICMJE (or any organisation) to mandate that the entire world reported results to clinicaltrials.gov. Reporting to any primary registry of the WHO's ICTRP would be more appropriate. And the authors should have considered calling for clinicaltrials.gov to share with the global community and participate fully in WHO's ICTRP.
Perhaps even more importantly, the for-profit publication industry is not the place to solve the question of results reporting and clinical trial transparency. This industry is entrenched in commercial interests. It is far more effective to have results reporting as a legal requirement. In areas where this is not legally required, the funders of clinical trials should require results reporting and ethics committees (IECs/IRBs) should ensure compliance.
The authors fail to consider the precise language of the 2013 Declaration of Helsinki: 'Every research study involving human subjects must be registered in a publicly accessible database . . ..' and 'Researchers have a duty to make publicly available the results of their research . . .. Apparently, just under half of the clinical trials registered on clinicaltrials.gov do not comply with the Declaration of Helsinki. And beyond this, an even larger number of 'research studies involving human subjects' take no heed of Helsinki’s requirements regarding registration and publishing.
The authors also fail to understand who is responsible for registering and publishing the results of clinical trials: 'it can be argued that compliance is part of investigators’ responsibility to ensure that their research is transparent and discoverable'. According to US and European law, the ‘responsible party’ is clearly the sponsor, not the investigator. And where compliance is at stake, this distinction is tremendously important.
Moreover, ‘discoverable’ is not the norm for results reporting. ‘Publicly available’ is the norm. And while the ‘impact factors’ of journals may have a value in science, what matters for public health, individual patient health, and treating physicians and prescribers is ‘access’: unfettered access to the results of research. ‘Annual subscriptions’ and ‘pay-per-view’ places health science outside the reach of most healthcare workers and patients. Society can no longer accept that the results of clinical trials performed on voluntary human subjects are simply 'discoverable'. They must be 'publicly available'.
Transparency in clinical trials is about trust. But ultimately transparency is about ensuring that doctors and patients have direct access to reliable information about medical interventions as well as the interventions themselves.
Competing interests: Francis P. Crawley has advised the World Health Organization on a variety of ethical and medical access topics, including for the development of the International Clinical Trials Registry Platform (ICTRP). He also participated in discussions with the European Commission and the European Parliament leading to the creation and updating of EudraCT. He is also a past member of the Steering Group of the Chinese Ethics Committee of Registering Clinical Trials (ChiECRCT).