Intended for healthcare professionals

CCBYNC Open access
Analysis

Restoring invisible and abandoned trials: a call for people to publish the findings

BMJ 2013; 346 doi: https://doi.org/10.1136/bmj.f2865 (Published 13 June 2013) Cite this as: BMJ 2013;346:f2865

This article has a correction. Please see:

  1. Peter Doshi, postdoctoral fellow1,
  2. Kay Dickersin, professor, director234,
  3. David Healy, professor of psychiatry5,
  4. S Swaroop Vedula, postdoctoral fellow6,
  5. Tom Jefferson, researcher7
  1. 1Johns Hopkins University School of Medicine, Baltimore, Maryland, USA
  2. 2Epidemiology, Johns Hopkins Bloomberg School of Public Health, Baltimore
  3. 3Center for Clinical Trials, Johns Hopkins Bloomberg School of Public Health
  4. 4US Cochrane Center, Baltimore
  5. 5Bangor University, Bangor, UK
  6. 6Johns Hopkins University, Baltimore
  7. 7Cochrane Collaboration, Rome, Italy
  1. Correspondence to: P Doshi, Reed Hall West, Room 201-C, 1620 McElderry Street, Baltimore, MD 21205, USA pdoshi{at}post.harvard.edu
  • Accepted 5 April 2013

Unpublished and misreported studies make it difficult to determine the true value of a treatment. Peter Doshi and colleagues call for sponsors and investigators of abandoned studies to publish (or republish) and propose a system for independent publishing if sponsors fail to respond

Well designed and well performed randomised controlled trials are considered to provide the most reliable evidence on the effects of health related interventions. However, the credibility of findings from individual trials and from summaries of trials examining a similar research question (that is, systematic reviews and meta-analyses) has been undermined by numerous reporting biases in the published medical literature.1 2 3 4 5 6 7 8 9 10 11 12 13 14 Reporting biases are often difficult to detect, but have the potential to discredit earnest efforts towards evidence based decision making.

Two basic problems of representation are driving growing concerns about relying on published research to reflect the truth.10 15 The first is no representation (invisibility), which occurs when a trial remains unpublished years after completion. The second is distorted representation (distortion), which occurs when publications in medical journals present a biased or misleading description of the design, conduct, or results of a trial.1 6 10 14 Both go against the fundamental scientific and ethical responsibility that all research on humans be used to advance knowledge and are symptomatic of a general culture of data secrecy. The end result is that the healthcare, biomedical research, and policy communities may, despite best intentions and best practices, end up drawing scientifically invalid conclusions based on only those parts of the evidence base they can see.

A call to publish—or be published

Despite near universal agreement that reporting biases are harmful, efforts to correct the problem have largely focused on forward looking initiatives. Prospective registration of trials has made major strides in ensuring that the biomedical community is aware of trials at their inception, but at best only around half of registered trials on ClinicalTrials.gov were registered before they began enrolling patients.16 Recent studies have also shown that even when disclosure of study findings is mandated by law, results often remain invisible.17 18 19 In addition, trial registration does not address the problems of invisibility and distortion for trials that took place before registers were widely used. Most importantly, those demanding correcting action lacked the data required to actually correct the scientific record. However, with increasing amounts of data entering the public domain, it is now becoming possible to move from words to action and publish (or republish) abandoned trials.

We have access to around 178 000 pages of previously confidential company research documents (table 1, box). For drugs such as paroxetine, quetiapine, and gabapentin, litigation over illegal off-label marketing put thousands of pages of trial reports in the public domain. Other trial reports, such as for oseltamivir and clopidogrel, were obtained through new freedom of information policies at the European Medicines Agency (EMA) that have revolutionised the public’s ability to access trial data.20 21 22 23 The documents are a substantial resource of information about trials. We expect that other independent groups will also have access to many additional trial reports.

Table 1

 Clinical study reports in our possession

View this table:

The documents we have obtained include trial reports for studies that remain unpublished years after completion (such as Roche’s study M76001, the largest treatment trial of oseltamivir, and Pfizer’s study A945-1008, the largest trial of gabapentin for painful diabetic neuropathy). We also have thousands of pages of clinical study reports associated with trials that have been published in scientific journals but shown to contain inaccuracies, such as Roche’s oseltamivir study WV15671, GlaxoSmithKline’s paroxetine study 329, and Pfizer’s gabapentin study 945-291.3 12 24 25 We consider these to be examples of abandoned trials: either unpublished trials for which sponsors are no longer actively working to publish or published trials that are documented as misreported but for which authors do not correct the record using established means such as a correction or retraction (which is an abandonment of responsibility) (box 1).25 Because abandonment can lead to false conclusions about effectiveness and safety, we believe that it should be tackled through independent publication and republication of trials.

Box 1: Understanding the evidence iceberg

What’s above the waterline
  • Published trial—A trial for which there is a permanent public report that has the individual trial as the central focus and at a minimum presents a summary of the trial protocol and study results. Such reports usually appear in scientific journals. Reports of trials presented with other trials (such as in pooled analyses), do not by themselves confer “published” status

  • Misreported trial—A trial that the biomedical literature documents has been erroneously reported or reported in a biased manner, such as with outcome reporting biases. Journal editors will have to decide whether misreporting is serious enough to warrant republication

What’s partly above and partly below the waterline
  • Abandoned trial—A trial can become abandoned in two ways. Firstly, a published trial can become abandoned when it is documented to be misreported and its authors do not correct their trial publication. Secondly, an unpublished trial can become abandoned when its sponsors are not working to ensure the trial is published. Detecting this form of abandonment is difficult. Comparing the amount of time that has elapsed since completion of follow-up to other published trials of the same drug and sponsor may help, but confirmation that an unpublished trial is abandoned may require contacting trial sponsors or clinical investigators

What’s (typically) below the waterline
  • Clinical study report—A report using a highly structured format for integrated and complete reporting of the planning, execution, results, and analysis of a clinical trial (box 2)

  • Individual participant data—Data for each participant in a trial. This contrasts with aggregate or summary data, which is produced by combining data from multiple participants. Individual participant data allows for the replication of all analyses in study reports and exploration of further analyses

  • Unpublished data—Data of any type (measurements, analyses, narratives, or judgments) from a trial that have not been published, irrespective of whether the trial is published. Since trial reports in peer reviewed scientific journals typically provide only highly compressed summaries of trial data,26 27 large amounts of unpublished data will remain for these trials

  • Case report form—The original paper or electronic forms on which individual participants’ data (demographic, efficacy, safety, etc) are recorded during the clinical trial. The forms are typically the most “raw” form of detailed data available for understanding what happened in a clinical trial, and the data they contain are statistically analysed only after they have been entered into an electronic database of individual patient data. Forms can vary in length, from a few pages to hundreds of pages, and each trial can have multiple forms—for example, for different visits or for the different tests or procedures the participant undergoes.

  • Trial protocol—A document, written prospectively before recruiting participants into a trial, which records the general rules and intended methods of conducting, analysing, and reporting the trial. Many also include a statistical analysis plan. Trial protocols are tens to hundreds of pages in length. A protocol may be required by the research ethics board, a data and safety monitoring board, or a funding body. Any planned or actual changes from the original written protocol in the conduct should be documented with formal protocol amendments.28 29

  • Investigators’ brochure—A document written by a sponsor and intended for clinical investigators interested in becoming involved in a study. It summarises the current body of evidence about an intervention under investigation, typically based on preclinical and human studies. The document is periodically updated in light of new information.

A call to action

We call on institutions that funded and investigators who conducted abandoned trials to publish (in the case of unpublished trials) or formally correct or republish (in the case of misreported trials) their studies within the next year. This should allow sufficient time for manuscript preparation, peer review, and publication. We will email a copy of this article to manufacturers of trials listed in table 1, asking them to signal their intent to publish by sending an electronic response to the article within 30 days. We propose that if anyone who declares an intention to publish or correct does not do so within one year, all available data for such trials should be considered “public access data” that others are allowed to publish.

We are committed to seeing the findings from abandoned trials published and describe here a minimum set of criteria for responsible publication and republication of abandoned studies (box 2). We call this concept restoring invisible and abandoned trials (RIAT). As the concept develops, interested individuals and organisations will ideally work together to develop detailed policies aimed at improving trial publication practices. We see RIAT as a collaborative, global effort, and over the next year we hope to discuss and debate our proposal at appropriate venues.

Box 2: Proposal for restoring invisible and abandoned trials (RIAT)

1. Obtain clinical study reports and any other study data
2. Collect documentation of trial abandonment
  • For unpublished trials—No primary publication detected by systematic search of the literature and/or confirmation from original trial sponsor or current responsible party that no publication exists

  • For misreported trials—Evidence of misreporting (ideally, published letters or other articles in the scientific literature or documentation of communication with the original trial publication author(s) detailing the misreporting) and a failure to correct the scientific record.

3. Issue a “call to action” by publicly registering your possession of data sufficient for publication

At least initially, this should be by an electronic response to this article and should include, as a minimum, trial identifiers, number of participants, date completed, publication status, pages in your holding, and level of access to trial data. This declaration offers original sponsors and trialists an opportunity to publish or formally correct their studies within the next 365 days. Send a copy of the rapid response by email to trial sponsors (and authors, for published trials), requesting confirmation of receipt.

4. Collect documentation of the need for restoration

Save time stamped copies of all rapid responses to this article (or other relevant websites) to document the time elapsed and consequent need for third party restoration.

5. Presubmission inquiry to RIAT friendly journal

Present editors with documentation from steps 1-4 and seek confirmation of editors’ interest.

6. Prepare and submit manuscript according to RIAT procedures
  • Include explanation (with references) in the Introduction of why this trial is being restored

  • Provide auditable record of decisions (use RIATAR template), documenting which parts of the clinical study report (page number and paragraph) were used

  • Report analyses specified in protocol

  • Denote any analyses that were not prespecified

  • Make all underlying data available electronically

The concept of publishing trials that we neither participated in, nor paid for, is an extension of what, in certain cases, we currently have in place: public use of epidemiological and clinical trial datasets from government sources30 31 and public access to summary trial results on ClinicalTrials.gov.32 Thus the scientific community has already accepted that investigators not associated with the original trial will produce and publish additional or confirmatory analyses. Furthermore, there are precedents for both publishing unpublished studies and republishing distorted studies. Examples include the description of design and findings from unpublished studies on the effects of nicotine on hypothalamic functions by using previously confidential (but now publicly available) company documents,33 34 and reports of case studies derived from the clinical observations of neurosurgeon Harvey Cushing.35 More recently, an investigator unconnected to Amgen’s epoetin alfa study 930107 republished this trial, documenting serious distortions in the original publication.36 37

Publishing trials, credibly

The major factor that makes publication of invisible and abandoned trials possible is the existence of clinical study reports (CSRs), documents produced by the pharmaceutical industry that include an unabridged and detailed summary of the planning, conduct, and results of a clinical trial.38 The reports are rigidly structured according to guidelines that industry and regulators agreed to in 1995 (box 3) and are almost always hundreds, if not thousands, of pages long. Manufacturers submit clinical study reports to the US Food and Drug Administration as part of applications for new drugs. In addition, the FDA typically also requires submission of the protocol and individual participant data. The European Medicines Agency does not routinely request individual participant data or clinical study reports.39 Although clinical study reports may be unfamiliar to the academic world, and in our experience are typically not produced for trials sponsored by non-commercial funders, when those in industry or the FDA want to know what occurred in an industry sponsored trial, they may refer to a clinical study report. When industry statisticians wish to carry out further analyses of the data, they can turn to their database of individual participant data. The rest of us, however—doctors, medical and public health researchers, patients, and non-regulatory government agencies including many health technology assessment groups—are left with only what is in the public domain (usually at best, synopses of the trials in the form of journal articles) (figure)..

Box 3: Clinical trials and clinical study reports

Writing clinical study reports is now industry standard practice, but it was not always so. In the middle of the 20th century, the early years of randomised controlled trials, studies were small—initiated, carried out, and reported by a handful of investigators, often in an academic medical centre. Reporting was likewise more concise and simple—something that could be explained in a journal length article, with more detailed datasets available on request.

The 1980s, however, saw a dramatic rise in the complexity of clinical trials. Multisite, international trials funded by industry and use of contract research organisations, became common.40 41 Whereas formerly those reporting trials had direct involvement in the conduct of the trial, the involvement of individual investigators in multicentre trials is often limited to a particular centre and a single “principal investigator” may see only the data they collected, not the complete dataset across all trial centres.42 43 44 45

In 1995, US, European, and Japanese regulators along with the respective industry trade organisations established a standardised way to organise a CSR to help simplify the new drug application process across global regulatory regimes.46 The CSR’s length and complexity helped fuel the growing industry of paid medical writers, with people who had no involvement in the clinical trial authoring clinical study reports.47

The guideline defines a CSR as “an ‘integrated’ full report of an individual study of any therapeutic, prophylactic, or diagnostic agent . . . conducted in patients. The clinical and statistical description, presentations, and analyses are integrated into a single report, incorporating tables and figures into the main text of the report or at the end of the text, with appendices containing such information as the protocol, sample case report forms, investigator-related information, information related to the test drugs/investigational products including active control/comparators, technical statistical documentation, related publications, patient data listings, and technical statistical details such as derivations, computations, analyses, and computer output. The integrated full report of a study should not be derived by simply joining a separate clinical and statistical report.”46

Although by definition no journal publication exists for “unpublished trials,” clinical study reports for industry funded trials often do exist for these unpublished trials, but they have been traditionally treated as secret.48 49 However, litigation and new freedom of information rules in Europe have helped many clinical study reports to emerge in the public domain, thereby making the restorative authorship possible. In addition, some drug companies have recently pledged to release their reports.50 51 Not all of the clinical study reports and other materials we have obtained are complete. However, many contain sufficient detail to form a comprehensive understanding of the trials and would enable someone to produce a journal length manuscript for publication.

We believe it is important to publish unpublished and other abandoned studies, even though they will at best represent a brief synopsis of all the publicly available data. This is because we live in a research and practice environment based on publications, and unpublished trials remain largely invisible. There is still no PubMed-like indexing system for unpublished clinical study reports. Moreover, most researchers will not have the time to sift through hundreds or thousands of pages to understand what occurred in a single clinical trial. We therefore need a shorthand representation, and the best we know of is journal publication.

To avoid a continuation of journal papers with selective reporting, we propose that trial publications adhere to reporting standards that ensure accountability. With a compression factor in some cases well above 1000:1 (table 2), summarising a clinical study report into a journal length manuscript inevitably requires value judgments about which information to include. These decisions should be transparent so that any bias can be identified and discussed. Responsible restorative authorship requires those publishing articles to also make the underlying trial data available simultaneously as an electronic appendix. In addition, there should be public access to an auditable record that documents which parts of the clinical study report (page numbers and paragraphs) were incorporated into the new publication, to help make restorative authors’ value judgments about what to include in the summary explicit and transparent. We have designed the RIAT audit record (RIATAR), a tool to ensure this is done systematically, based on the CONSORT checklist for reporting randomised trials (see web appendix).52

Table 2

 Compression factor ratio of pages in clinical study reports (CSRs) to published study

View this table:

Providing public access to both source documents and an audit record gives readers a quick way to find (and cross check) the relevant and more detailed information within the original clinical study report. We think that it should apply to all trials published, irrespective of authorship, and follows on from previous calls by journal editors for improved research reporting standards.53 54 55 56 57 It would enable independent verification of the accuracy of journal publications and permit better evidence synthesis and other forms of research.49 58

RIAT reports should also provide the context for the study to help the readers understand why the trial is being restored. This means including references to any previous publications of the trial and to details and evidence of the trial’s abandonment. RIAT analyses should follow the analyses specified in the protocol (including any specified in amendments). Any other analyses are discouraged, but if done must be clearly noted as exploratory and not prespecified. At the same time, RIAT authors may wish to critically appraise the trials they report. This can be useful, but the critique should be clearly identifiable and placed in the discussion section.

Important details are still to be worked out (box 4 lists some of them), and we welcome discussion on how to get it right.

Box 4: Issues for further discussion and development

  • Should the original trial investigators have the right to join a group of RIAT authors?

  • Should RIAT authors contact original investigators for help in interpreting trial documents?

  • Should old trials that are going to be subject to RIAT publication be registered in trial registries such as ClinicalTrials.gov (assuming this becomes possible)? Should registration preclude other restorative author teams from using the same data?

  • Should RIAT publication related clinical study reports and audit records be stored on the publishing journal’s server or is a publicly available database required?

  • Where should the declaration of intent to publish a RIAT trial be published?

  • How should RIAT publications be cited? Should restoration be clearly identified in the publication title or the authorship byline? If yes, how?

  • What (if any) additional rules or safeguards should RIAT authors apply to ensure patient confidentiality of individual patient data and clinical report forms that are already in the public domain?

  • For misreported articles, what level of narrative detail regarding the misreporting should the RIAT publication contain?

  • Should RIAT authors seek approval from an appropriate research ethics committee before publishing a trial with unclear prior ethical basis?

Potentially controversial aspects of our proposal

The idea of restorative writing may be seen as taking on responsibility and credit for other people’s actions, regardless of the trial’s sponsor, but it takes on a slightly different cast when trials are funded by commercial sponsors rather than public money. Some people may think that publications based on clinical study reports with which the authors have no connection is equivalent to intellectual property theft, but you cannot steal what is already in the public domain (and only in the public domain because a drug regulator or judge had the documents unconditionally released or the sponsor waived their confidentiality claims over the documents). The considerable discussion about the need for public access to trial data and data ownership has not yet resolved how to handle the thorny but important question of proper scientific credit.59 60 61 RIAT authorship will not usurp proper credit for a trial. Rather, it will show how problematic the concepts of authorship and results reporting are in the modern clinical trial. RIAT authors would be able to claim credit for bringing to light what was previously invisible or distorted but not for carrying out the trial.

In the case of the Roche sponsored oseltamivir trials, we have so far identified eight different layers of responsibility, perhaps partially overlapping: those who designed the trial, those who sponsored it, those who conducted it, those who analysed the data, those who wrote or assembled the clinical study report, those who decided the publication policy, those who decided which parts to publish (and in some cases not to publish), those who presented results at meetings or conferences, and, lastly, those who put their names to the published manuscript. None of these roles has a clear thread of accountability and authors of the published trials have confirmed that they did not have access to the underlying study data.45 62 In sum, in the context of the modern clinical trial research enterprise, the traditional journal article publication model obscures responsibility more than it illuminates.

Is restorative writing fundamentally different from professional medical writing and “ghost writing”? One important difference is that hired medical writers are paid for their services by those who stand to gain from the publication and restorative authors are not. Restorative authors are also likely to have access to more detailed trial records than medical writers. Another difference is that medical writers are often instructed to insert “key messages” in publication ready manuscripts.11 63 Finally, medical writers are often unacknowledged in the publication and so are not accorded any responsibility for the work they produce. By contrast, RIAT authors will take full responsibility for publishing abandoned studies, although we will refer to RIAT to make it clear that the article is a work of restoration, not primary authorship. We are also contemplating how best to document RIAT authorship in our CVs: at a minimum, such publications need to be listed under a separate heading, identifying them as such.

Recently, a group of drug manufacturers and medical journal editors published “ten recommendations for closing the credibility gap in reporting industry-sponsored clinical research,” aimed at eliminating reporting biases.64 Their recommendations do not go far enough to address the problems. They do not mention publishing abandoned trials and ignore responsibility for correcting reporting biases persisting in existing trial publications. Furrthermore, their recommendation to “make public all data” refers to publication of journal article length manuscripts rather than the full clinical study reports, individual participant data, investigators’ brochures, case report forms, and many other of the semi-secret documents that would help people to understand a trial and its place in the research or regulatory approval programme—meaning the published results would have to be taken on trust without the possibility of verification.

Will the publication of detailed clinical study reports enable subsequent ill intentioned or otherwise misleading analyses by others (such as spurious findings from data dredging)? We challenge readers to provide an example of open clinical trial data sharing that has led to major public health harm. If RIAT evokes the spectre of data mining, it is important to remember that we currently have no way to judge the fidelity of the process of synthesising thousands of pages of a clinical study report into a journal publication. RIAT publication is important even for poorly conducted or unethical studies that many editors may not feel merit publication. Without public documentation that a trial was poorly done, researchers will be left guessing about the value of the study. A very brief trial report (without results if they would be misleading) may suffice.

Finally, some people may argue that RIAT republication of a misreported study is muddying the published record for dubious gain, especially with older trials. We believe that correcting the scientific record is preferable to ignoring inaccuracies. If the accompanying data support what is reported in the RIAT republication, doubts about which publication is correct should not be a problem.

Call for restorative authors and participating journals

The data we have obtained (table 1) relate to only a small fraction of the masses of abandoned clinical trials. We call on others to join us, to contribute trial documents they have obtained from public sources that need publishing or republishing, and to help us with the writing. We need volunteers to act in place of those who should have but did not make trial reports visible and accessible.

Litigation and freedom of information promise to usher increasing amounts of clinical trial documents into the public domain. This reality necessitates an urgent discussion about what constitutes this new public commons and how it should function. Should there be a central repository for once secret trial documents and, if so, who should or can responsibly house, index, and maintain a public database of documents that span regulatory and legal boundaries? The tens of millions of pages of internal tobacco industry documents released in 1998 illustrate the enormity and importance of rising to the challenge.65

Endorsement of the concept of restorative authorship by medical journal editors will help the effort to complete and correct the scientific record. Journals can signal their willingness to accept RIAT publications by including details in their “instructions for authors.” We suggest that journals ask restorative authors to provide documentation of a trial’s status as abandoned, the provenance of data on which the RIAT publication is written (to ensure it is in the public domain), and to agree to submit the clinical study report and all other data used to write the manuscript as well as an audit record documenting what data were used. We suggest that to reduce wasted time on behalf of both authors and editors, authors submit a presubmission inquiry to discuss their case.

Our declaration to publish will be the first step towards public and open debate on an issue that affects everyone and has for too long been the preserve of people acting behind closed doors.

Notes

Cite this as: BMJ 2013;346:f2865

Footnotes

  • We thank Yuko Hara, Iain Chalmers, and Betsy Humphries for many helpful conversations and comments on the idea and draft manuscript. We also wish to thank the many others who made important contributions during the peer review process: external peer reviewers James Miller, Douglas McCrory, Margaret Winker, Emma Veitch, Andrew Vickers, Deborah Zarin, and Tony Tse and the BMJ editors, particularly Elizabeth Loder.

  • Contributors and sources: PD conceived of RIAT and wrote the first draft of the manuscript with TJ. Numerous oral and written discussions among all authors shaped the RIAT concept. All authors contributed to the many rounds of editing. PD is guarantor.

  • Competing interests: We have read and understood the BMJ Group policy on declaration of interests and declare the following: PD and TJ have published on influenza antivirals and received a UK National Institute of Health research grant for the update and amalgamation of two Cochrane reviews: neuraminidase inhibitors for preventing and treating influenza in healthy adults and children. PD received support from the European Respiratory Society for travel to the society’s September 2012 annual congress where he gave an invited talk. He is funded by an institutional training grant from the US Agency for Healthcare Research and Quality. AHRQ had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. KD served as an expert witness in 2008 litigation against Pfizer relating to gabapentin. Funds from this effort were donated by the plaintiff’s lawyers to Johns Hopkins to support academic scholarship in the area of reporting biases. DH has been an expert witness in legal cases that have brought to light issues surrounding non-publication of trials and ghostwriting and an action by New York State that led some of the documents this article references to come to light. SSV was paid by the plaintiffs’ lawyers for assisting KD with preparing her expert witness report in 2008 litigation against Pfizer. SSV’s research related to reporting of trials in off-label uses of gabapentin was supported by an academic scholarship established at Johns Hopkins using funds from KD’s effort as an expert witness in the 2008 litigation against Pfizer. TJ receives royalties from his books published by Blackwell and Il Pensiero Scientifico Editore. He is occasionally interviewed by market research companies for anonymous interviews about Phase 1 or 2 products. He was a consultant in a legal case regarding oseltamivir.

  • Provenance and peer review: Not commissioned; externally peer reviewed.

This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 3.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/3.0/.

References

View Abstract