Are clinical trial data shared sufficiently today? NoBMJ 2013; 347 doi: https://doi.org/10.1136/bmj.f1880 (Published 09 July 2013) Cite this as: BMJ 2013;347:f1880
- Ben Goldacre, Wellcome research fellow in epidemiology
When discussing transparency it is important to be clear on what is being requested, as obfuscation is sometimes used to avoid discussing simple fixes. At stake are four levels of information about trials: (1) knowledge that a trial has been conducted, from a clinical trials register; (2) a brief summary of a trial’s results, in an academic journal article or regulatory summary; (3) longer details about the trial’s methods and results, from a clinical study report where available; (4) individual patient data. The AllTrials campaign calls only for the first three to be published.
The status quo is plainly unsatisfactory. The most current review—with no cherry picking permitted—estimates that around half of all trials for the treatments being used today have gone unpublished; and that trials with positive results are twice as likely to be disseminated.1 This is a problem for both industry and academic trials.
Although some in industry claim that these problems are in the past, in reality all supposed fixes have failed. In 2005, journal editors passed regulations stating that they would publish only registered trials: the evidence now shows these regulations have been widely ignored.2 In 2007, US legislation was passed requiring all trials since 2008 to post results on clinicaltrials.gov within a year of completion: the best published evidence shows this law has been ignored by 60-90% of trials.3 If industry representatives believe these problems have been fixed, they should present published evidence to support their case, with methods and results that are available for public scrutiny.
Even if the latest rules on transparency were to be implemented perfectly—starting from now—they would still do nothing to improve the evidence base for the treatments we use today, because they all cover only trials from the past few years. More than 80% of the medicines prescribed this year were generic, and came on the market more than a decade ago. We need the results of trials on these treatments, which are still available, albeit on paper. It is both practical and reasonable to request that these documents should be simply scanned, and shared.
The arguments against this level of transparency are conflicted and misguided. John Castellani, of the Pharmaceutical Research and Manufacturers of America (PhRMA), has claimed previously that it’s enough for regulators alone to see all the information on trials, and to see it behind closed doors. But this goes against the fundamental principles of science: we rely on transparency about methods and results, so that every experiment can be double checked and critically appraised. Although he might not realise it, Castellani’s position also exposes patients to real and unnecessary risks. Many of the most notable recent problems with medicines—problems with rofecoxib (Vioxx) and rosiglitazone (Avandia), for example, and problems with the evidence base for oseltamivir (Tamiflu)—were spotted by independent academics and doctors, and not by regulators. This isn’t because regulators are incompetent; on the contrary, they are highly trained, intelligent, and well motivated. But risks and benefits can be difficult to detect, and like everything in science, these problems benefit from many eyes.
For similar reasons, it is peculiar to see industry argue that information should not be shared simply because there might be disputes about interpretation: disputed interpretations are widespread throughout science and medicine, they are normal, and this open debate is how we get closer to the truth. And likewise, we do not silence medical scaremongers in the media by hiding information about trials; if anything, routinely withholding trial results is more likely to undermine public trust.
Overall, the lack of progress on transparency has been startling. Some worry that these problems should not be discussed in public, while we fix them quietly behind closed doors. But the problem of withheld trial results has been documented since at least 1986,4 and industry has successfully delayed remedial efforts for three decades. The latest strategy has been to raise the spectre of patient privacy.
In February, for example, PhRMA released a colourful statement that misleadingly suggested that I and the BMJ have somehow called for the reckless public release of full individual patient data sets. They made this claim, despite the head of press relations at PhRMA already knowing that neither I nor the AllTrials campaign call for individual patient data to be published.5 6
The BMJ has recently called for individual patient data to be made more widely available, in an editorial.7
Was this reckless and unreasonable? I don’t believe so. Where industry has shared data with researchers, it has been only piecemeal, and after enormous battles. But in many fields, there is already a long history of sensible and cautious sharing of detailed datasets—for example, to conduct individual patient data meta-analyses. These produce better estimates of treatment benefits, and improve care for patients, with appropriate concern for confidentiality. The Early Breast Cancer Trialists Collaborative Group’s meta-analyses, already published, represent just one notable example.8 The YODA project at Yale is looking at best practice for data sharing, as are many other groups.9 What’s more, the European Medicines Agency (EMA) has fully committed to sharing individual patient data after 2014, and are consulting only on the best mechanism to do so.10 These are reasonable and responsible things to discuss, as evidence based medicine moves forwards and becomes more effective.
Is patient confidentiality also an issue when clinical study reports are shared, as AllTrials and I have suggested they should be? Clinical study reports are long documents—often thousands of pages—but they are important, because analyses have shown that the information published in academic journal reports on clinical trials can be misleading or inaccurate, when compared with these longer, definitive sources of information.11 12 These reports certainly do contain some information about individuals—for example, in narrative descriptions of adverse events—but such information can easily be removed, or shared only with named researchers, if this is deemed necessary. Some industry figures have claimed that removing this material is either impossible or prohibitively expensive. But in 2010 the European ombudsman made a ruling of maladministration against the EMA, for claiming exactly that. The ombudsman examined the clinical study reports requested from the agency in detail, and concluded that the administrative burden of removing patient information, where necessary, was small. The European ombudsman has also stated clearly that there is no important commercially confidential information in these reports—the fact that a drug is not as good as claimed is not, in itself, something any company can hope to ethically withhold from doctors and patients.13 Since then, the EMA has released 1.6 million pages of clinical study reports14 under its new policy.13 Because these documents are so informative—and because the EMA holds only a small proportion of all the clinical study reports in existence—alltrials.net is asking for all existing clinical study reports to be made available, on all medicines currently in use.
This campaign has rapidly snowballed to become the mainstream position in the United Kingdom. AllTrials is now supported by more than 50 000 individuals, and 250 organizations, including more than 100 patient groups, the National Institute for Health and Care Excellence, academic funders such as the Medical Research Council and the Wellcome Trust, royal colleges, the Royal Pharmaceutical Society, the British Pharmacological Society, and the Faculty of Pharmaceutical Medicine, to name but a few. Ironically, within 24 hours of PhRMA denouncing our calls for greater transparency, GlaxoSmithKline—the world’s fourth largest drug company—signed up as supporters of alltrials.net. They have committed to do the very thing that PhRMA says is impossible, and share all clinical study reports going back to the foundation of the company.
If the transparency we ask for is practical, and reasonable, then what lies behind the colourful denunciations of PhRMA? Speaking to policy staff in some signatory organizations, one worrying theme recurs. We knew that withholding trial data was common, people have said, and we knew that it harms patients, but we felt embarrassed to talk about it, because even raising the issue seemed somehow subversive. This is a worrying state of affairs, and a testament to the power of aggressive lobbying by industry. But it is also perhaps a testament to the capture of key opinion leaders, and the dangers of longstanding inaction at senior levels in the medical establishment. In the UK, we have seen the same phenomena during prominent inquiries into failing hospitals: many senior staff, in numerous organizations, all saw a problem, but most were too busy—or too anxious about workplace conflict—to put patients first.
The problem of missing trials is one of the greatest ethical and practical problems facing medicine today. It also represents a bizarre paradox: we can spend millions of dollars on a trial, hoping it is free from bias, trying to detect a modest difference between two treatment groups; and then at the final moment we let all those biases and errors back in, by permitting half the results to disappear. Future generations may well look back at our tolerating this in amazement, in the same way that we look back on mediaeval bloodletting. The AllTrials movement is driving the solution forwards: patients need industry to engage constructively with this widespread consensus, on the practical details—urgently—so that we can all move on.
Cite this as: BMJ 2013;347:f1880
Competing interests: I have read and understood the BMJ Group policy on declaration of interests and declare the following interests: I am a doctor, academic, and writer. I make income from talking and writing about problems in science, including publication bias. My research fellowship in epidemiology is funded by the Wellcome Trust, who are signatories to AllTrials, and support other aspects of open science.
Provenance and peer review: commissioned; not peer reviewed.