Liberating the data from clinical trials

Despite the importance of reproducibility in research, clinical trials are rarely subject to independent reanalysis. In a linked paper, Le Noury and colleagues (doi:10.1136/bmj.h4320) have restored and reanalysed the controversial “study 329,” which incorrectly portrayed paroxetine as an effective and acceptably safe treatment for children and adolescents with major depression.1 2 The accompanying article by Doshi (doi:10.1136/bmj.h4629) details the mis-steps of the investigators, staff from the sponsoring drug company, the lead author’s home academic institution, and the publication journal.3 Study 329 is a model example for the movement to restore invisible and abandoned trials (RIAT), which calls on investigators to publish unreported trials and republish and correct misleading reports.4

In a recent review, Ebrahim and colleagues identified just 37 published reanalyses of clinical trials.5 Only five were conducted by investigators not associated with the original report. A third of the reanalyses led to interpretations that were different from those of the original articles. In a recent blog, Ben Goldacre, co-founder of the +AllTrials initiative, which calls for all trials to be registered and published,6 7 8 highlighted the example of an influential trial of intestinal “deworming” treatment. Reanalysis uncovered important errors and changed some central conclusions of the original report.9 10 Goldacre applauds the original authors for having the courage to share their data, despite the potential for errors to be identified and having to go through the discomfort of their mistakes being made public.

While rare among clinical trialists, the idea of sharing scientific data is not new and is common practice within some disciplines, such as genomics, astronomy, and particle physics.11 In a bold move by the standards of the time, researchers from the landmark Diabetes Control and Complications Trial (DCCT) made their data available to other investigators after they published the results of the original trial. To date, there have been over 220 ancillary studies using DCCT data, several of which have had an impact on the clinical management of diabetes.12 13 These highlight the substantial added value that can be derived from sharing of trial data.

The move to access original trial data is part of the broadening open data movement in health, which has received support from major research funding agencies in the United States, Canada, the United Kingdom, Australia, and Europe.11 14 Notably, the National Institutes of Health (NIH), which strongly encourage NIH funded investigators to share their data, provide secure data repositories for both clinical data and biological samples.15 Recent reports from the Institute of Medicine (US), the Wellcome Trust (UK), and the Council of Canadian Academies argue for, and recommend, best practices to ensure safe sharing of clinical data.16 17 18 And of course many journals, such as The BMJ, now encourage authors to make datasets available on request.19

Data sharing, however, is not without its risks.18 As Ebrahim and colleagues point out, threats to patient confidentiality, data dredging with a risk of chance findings, and “rogue reanalyses” by investigators with their own agenda must be considered.5 Data sharing also increases the responsibilities and burdens placed on investigators and institutions, for whom trials can become consuming, long term commitments. As illustrated by Le Noury and colleagues,1 trial restoration can be a major undertaking for investigators carrying out the reanalysis, requiring substantial human and analytical resources.

Should restoration end with reanalysis, or should we do more? Data storage in repositories will enable independent researchers to repurpose trial data for new research questions—as shown by the successes of the DCCT.13 If participants’ data are stored with the identifying information needed to link to data stored in administrative claims or electronic medical record (EMR) databases, this will allow independent researchers to reactivate some “dormant” trials. Adding extra years of follow-up, via the linked databases, will allow the study of long term outcomes, including those not part of the original protocol.

The use of administrative and EMR data to capture clinical trial outcomes is becoming commonplace. Sometimes this has been part of the original trial protocol.20 21 22 Less commonly, data linkage is used subsequently to capture additional years of follow-up. An important example of the latter is the 20 year report of the West of Scotland Coronary Prevention Study (WOSCOPS).23 24

Reactivation of dormant trials will not be without barriers. In the case of older trials, data might have been destroyed, misplaced, exist in paper form only, or lack the variables necessary for linkage. The original trial consent forms might not have included permission to link the data, which will require research ethics boards to consider approval of “post hoc” linkage. In the case of the Canadian National Breast Cancer Screening Study, the research ethics board judged that the original approval to retrieve death certificates could be extended to electronic linkage to different outcomes by staff at Statistics Canada.25

The first step to reactivating dormant trials will be to identify which trials have been conducted and where the data are held. This can be done by searching trial registries and approaching research ethics boards, funding bodies, and investigators. Such an exercise has commenced in Ontario as a “meta-data mapping project,” under the Strategy for Patient-Oriented Research (SPOR) initiative of the Canadian Institutes of Health Research.26

To enable reactivation of important clinical trials we will need to review some policies and procedures. Trial consent processes should routinely request permission to link the data to study long term outcomes. Stored data from participants should always include linkable fields, particularly health insurance numbers. Data management and retention policies should be reviewed to enable preservation of the data needed to enable long term follow-up of important clinical outcomes.

Most clinical trials are extremely expensive, and we believe that the pay-off from a systematic effort to reactivate selected clinical trials will be high and will further justify the original huge investments of time and money.