The Tamiflu trials

This week sees the publication of the updated Cochrane systematic review on the neuraminidase inhibitors oseltamivir and zanamivir.1 The review, which is also reported in two papers published in The BMJ,2 3 provides the most complete analysis so far of what is known from randomised trials about the effectiveness and safety of these antiviral drugs. It is also the culmination of a four and a half year battle for access to the raw data from industry funded trials of oseltamivir, a drug on which the world has spent billions of dollars.4

Through their exhaustive scrutiny of the data contained in clinical study reports (the lengthy documents held by the drug’s manufacturer Roche and previously seen only in part by drug regulators), the Cochrane authors have set exacting new standards for systematic reviewers and decision makers. Their fight for the data has also shown us, in more detail than ever, that the entire ecosystem of drug evaluation and regulation is deeply flawed.

The Cochrane review and BMJ papers represent a huge amount of work. The authors—Tom Jefferson, Carl Heneghan, and colleagues—are among just a handful of systematic reviewers who have made use of clinical study reports (CSRs) to reach their conclusions. CSRs are intended to provide regulatory authorities with a structured detailed report of a clinical trial.5 In contrast to the abbreviated information about a trial contained in a journal article, one CSR can extend to hundreds of pages. The “compression factor”—the length (in pages) of the CSR of a trial compared with the length of the corresponding journal article—has been found to range from one to an astonishing 8805.6 7

It is a formidable task to sift through this amount of information, which is perhaps why it is so rarely done. In the process of the oseltamivir review, and during the long battle for access to the raw data, the Cochrane reviewers have been forced to become pioneers, adapting systematic review methodology, developing new alliances, and navigating uncharted territory.

Early on they enlisted the help of the media, including The BMJ, to push for access to the trial data. A joint investigation by The BMJ and Channel 4 News in 20098 9 elicited a promise from Roche to make the data available.10 A subsequent series of open letters in The BMJ added to the pressure on Roche and the regulators,11 12 as did the authors’ decision to conduct all communication by email and to post the full correspondence on bmj.com (www.bmj.com/tamiflu).

From the start the authors refused to sign confidentiality agreements because they wanted others to be able to scrutinise their analysis and conclusions. In the end, although redacted to remove patient and investigator identifiers, the CSRs were delivered to them with no conditions or restrictions. These CSRs are being made available in full on the Dryad data repository (http://datadryad.org/), as is the entire combined peer review history from the Cochrane Library and The BMJ, making this one of, if not the most, transparent systematic reviews ever done.

Methodologically too the reviewers had to improvise. The Cochrane risk of bias tool was developed to judge the quality of randomised trials as reported in medical journals, and not to measure the quality of CSRs. The authors had to adapt the tool, as they explain. They plan to give further details in a separate paper.

As summarised in the accompanying editorial by Harlan Krumholz,13 the complete evidence from the CSRs paints a much less positive picture of oseltamivir than was presented to regulators, policy makers, clinicians, and the public. Important benefits have been overestimated and harms under-reported. In particular, the review found no compelling evidence to support claims that oseltamivir reduces the risk of complications of influenza, such as pneumonia and hospital admission, claims that were used to justify international stockpiling of the drug.

The review’s conclusion should lead to serious soul searching among policy makers. So too should the story behind the review, illustrating as it does the entrenched flaws in the current system.14 Why did no one else demand this level of scrutiny before spending such huge sums of money on one drug? And why do we have a system of drug evaluation and regulation that is incapable of providing patients, clinicians, and policy makers with timely, reliable, and independent information. Indeed, the current system seems to be designed with the opposite end in mind.

The Cochrane authors uncovered what they characterise as “multisystem failure.”14 Reporting problems caused the reviewers to believe that most of the trials of oseltamivir were at high risk of bias. Important endpoints, such as pneumonia, were poorly defined. None of the trials was independent of the drug’s manufacturer. All were against placebo rather than against standard drugs for relieving symptoms, such as paracetamol. No trial was undertaken during the pandemic—a squandered opportunity that may come back to haunt us.

The published studies were in some cases ghost written,8 and even from the CSRs it is sometimes impossible to work out who carried out the research, raising serious questions about academic accountability and independence. The published studies also represent only a highly selective slice of the complete clinical trial data, leading the Cochrane reviewers to discard the published literature as a source for their review and to rely entirely on the CSRs, irrespective of the trials’ publication status.

Licensing and reimbursement decisions were shown to have been widely inconsistent,15 reflecting differences in the rigour applied by various agencies and in the evidence requested by and provided to them by Roche. The regulators were reported to have been under political pressure to provide a pharmaceutical solution to the threat of a pandemic.8 The World Health Organization was reported to have been influenced by industry’s paid opinion leaders.16 Yet again this prompts questions as to whether the current regulatory system is fit for purpose, over stretched as it is and insufficiently independent of industry and government.

The importance of oseltamivir as a test case for data transparency was most strikingly signalled when the UK’s Public Accounts Committee concluded that “the case for stockpiling antiviral medicines at the current levels is based on judgement rather than evidence of their effectiveness during an influenza pandemic.” The committee called on the UK government to ensure that all clinical trial data for all drugs in current use be made available for independent scrutiny.17

Other developments have worked in tandem with the Tamiflu saga to build momentum for data sharing and transparency. The European Medicines Agency announced that it would make all the information it had available as from 2014.18 GlaxoSmithKline, Roche, and a handful of other major drug companies announced new policies on access to their trial data.19 The BMJ made willingness to share data a precondition of peer review for clinical trials of drugs and devices.20 Ben Goldacre published Bad Pharma, and the Alltrials campaign (www.alltrials.org) was launched, calling for all trials to be registered and all results reported. The UK Health Research Agency made trial registration a condition of ethics approval in the United Kingdom.21 The Institute of Medicine set about developing “strategies for responsible sharing of clinical trial data.”22 And last week the European Union passed a new regulation that will require all clinical trials to be registered, all results reported, and all available CSRs to be made public.23

This undoubted progress is however limited to data relating to future drugs. Access to data relating to drugs in current use will still be largely at the discretion of the drug manufacturer or through individual requests to regulators. The Cochrane reviewers’ exceptional efforts have achieved what should have been a matter of routine—the independent scrutiny of deindentified clinical trial data. They have shown with greater clarity than ever that the current system is broken. There are substantial battles still to fight before we have a system of drug evaluation and regulation that truly serves patients and the public interest.