What did we learn from Tamiflu?

Hidden data

In 2009, when the World Health Organization declared the novel type A H1N1 “swine” flu to be a pandemic, and global spending on stockpiling oseltamivir reached $6.9bn, the NHS commissioned a systematic review of the drug from Cochrane. The Cochrane reviewers had already concluded in 2006 that oseltamivir reduced complications such as pneumonia and shortened symptoms in seasonal flu.2

They anticipated a simple update, until a Japanese paediatrician, Keiji Hayashi, challenged them to dig deeper. He pointed out that the paper that had driven their 2006 review, a 2003 pooled analysis by Laurent Kaiser and colleagues,3 was based on 10 randomised controlled trials of which only two had been published. Most of the data supporting oseltamivir’s claim to reduce lower respiratory tract complications had never seen the light of day.

The reviewers contacted the authors of the 2003 paper but were told they did not have the data on the missing eight studies. So the Cochrane team went to the source, the manufacturer, Roche. The company refused to release the data unless the reviewers signed a confidentiality agreement with a secrecy clause. This they weren’t prepared to do, as it could stop them reporting their findings.

So began a campaign of public pressure that lasted four years, much of it playing out in the pages of The BMJ (bmj.com/Tamiflu). It would ultimately transform the research landscape, dragging into the daylight the critical details of methods and results in drug trials that the industry had previously jealously guarded as commercial secrets.

A freedom of information request shook loose 20 000 pages of incomplete oseltamivir data from the European Medicines Agency in 2011. In 2013 the drug giant GlaxoSmithKline, the maker of the other stockpiled antiviral, zanamivir, released its data on 30 trials. Later that year Roche finally relented and released 77 full clinical study reports of oseltamivir trials.

A Cochrane review in 2014 that used the newly released data found insufficient evidence to support claims that oseltamivir reduced lower respiratory tract complications or impeded viral transmission.4 The reviewers also raised new questions about the drug’s harm profile.

Yet these findings did not affect in any noticeable way governments’ reliance on oseltamivir. The finding did largely correspond to the assessment of the US Food and Drug Administration, whose oseltamivir label explicitly says that the drug has not been shown to prevent serious bacterial infections or complications of cardiac and respiratory diseases. But the US Centers for Disease Control and Prevention (CDC) took a far more expansive view of the drug’s qualities, and the European Centre for Disease Control and Prevention (ECDC) agreed. Today pandemic stockpiles are still being scrupulously topped up, albeit at lower cost since generic oseltamivir appeared in 2016. The response plans of the UK and US have barely changed in a decade.

Meanwhile, one of the Cochrane team members, Thomas Jefferson, has pursued Roche to the present day. Papers unsealed in a US federal court in January showed that he is suing Roche as a whistleblower under the US False Claims Act to recover $1.5bn he says the company wrongfully billed for, by selling oseltamivir to US public health authorities as a pandemic response drug.5

It is perhaps worth noting that the shelf life of stockpiled oseltamivir is generally set at about seven years. Hence by delaying disclosure of its trial data from 2009 to 2013 Roche would allow many governments to replenish their first big stockpile, dating from 2004-5, without being troubled by new data. By 2014 global spending on stockpiling oseltamivir had reached $9bn, a trend that has continued untroubled by the conclusions of the Cochrane reviewers that year that the manufacturer’s claims for the drug were not based on evidence. Speaking to The BMJ, Jefferson claimed that oseltamivir could have actually hastened the spread of the 2009 pandemic. This comes down to a disagreement between Cochrane reviewers and Roche as to how the drug works.

“It’s not specific to the flu virus at all,” says Jefferson. “It has a central action, lowering temperature, and thus making the patient feel better.”

If prophylaxis with oseltamivir only suppresses symptoms, “then infected people could be going to work and school feeling fine, while passing on the flu virus,” he says.

A missed opportunity to reduce stockpiles?

In 2017 WHO shifted its position on oseltamivir, moving the drug from the core to the complementary list of essential medicines. WHO recommended restricting its use to severe illness in critically ill hospital patients with confirmed or suspected influenza virus infection.6

If this recommendation were followed, says Cochrane reviewer and BMJ associate editor Peter Doshi, stockpiles could be reduced a hundredfold or more. But an editorial in The BMJ welcoming WHO’s move brought a sharp response from a group of physicians working for public health authorities, including Public Health England, the ECDC, the CDC, and even WHO itself.67

They pointed to more recent observational studies that seemed to show reduced complications and admissions to hospital among patients treated with oseltamivir.

Doshi is dismissive. “The original rationale for pandemic use was that randomised controlled trials showed oseltamivir reduced complications. When we showed that wasn’t true, they turned to observational studies that found reduced complications, but they don’t mention other observational studies showing the opposite, and they don’t mention that the observational studies they rely on were funded by Roche.”

WHO’s downgrading of oseltamivir did not move any of the stockpiling agencies to reconsider. The next month the ECDC published an expert opinion supporting existing recommendations in European countries and endorsing oseltamivir’s role in flu prophylaxis.8

Oseltamivir for pandemics is a unique drug: while other drugs were approved and then used, oseltamivir was approved and then stockpiled. Other drugs’ harms and benefits are likely to become apparent through clinical use, but oseltamivir’s efficacy in a pandemic remains untested—even though there was a declared pandemic in 2009 and oseltamivir was used. Few randomised trials were conducted, and some of their results have gone missing.9 This is despite repeated calls to conduct studies during pandemics, dating as far back as a JAMA editorial in 1919, after the outbreak of the 1918 flu pandemic.

Not only is the drug a relatively unknown quantity, but so is the disease. The 2009 pandemic may well have been overblown,10 but the potential threat from virulent flu is daunting. Add to this the costs sunk in oseltamivir stockpiles, and it’s easy to see why governments defend them.

“You could argue that it may sometimes be legitimate to give false reassurance to the public that you’re doing something useful, to reduce public panic,” says Ben Goldacre, the Nuffield Trust epidemiologist who cofounded All Trials, which campaigns for all trials to be registered at their outset so that inconvenient results cannot easily disappear. “A large procurement choice for a nation state is a complex business. I can’t say for sure whether the government should or should not have paid all the money it did for Tamiflu. What I can say is that neither the government nor the public were able to make an informed decision when they didn’t have access to all the methods and results from the trials that were conducted.”

Doshi says the investment in stockpiling shows the skewed priorities of public health agencies that show no interest in new trials of oseltamivir to determine whether the underlying evidence base is solid. The amount of funding required for a strong trial, he said, “seems pretty insignificant compared with public health decisions that cost billions of dollars, like stockpiling Tamiflu.”

Half the battle

Access to such data is the real legacy of the battle over oseltamivir. Like Roche, the rest of the drug industry has abandoned the claim that the detailed clinical study reports (CSRs) it provides to regulators should be off limits to researchers and the public.

“Tamiflu quite arbitrarily became the poster child for transparency,” says Goldacre. “Because it so happened that the reviewers involved, when they hit this wall which many systematic reviewers hit, instead stood up and said, ‘This is not OK.’”

CSRs have become the holy grail of researchers who want to look behind published data. Often running to thousands of pages, they can show vital flaws in trial methods, such as trial endpoints changing in mid-course.11 Research also indicates that CSRs often show more adverse events than published versions of the same studies.121314

CSRs “tend to be more complete,” says Goldacre. “You’d expect that, because when you’re obliged to put a structured report of your results on a clinical trials register, there’s a series of boxes you have to fill in. When you’re reporting your trial in an academic journal, you’re sort of writing an essay, modelled on a 19th century template for how a scientific experiment should be reported.”

Several portals have opened since 2014 to provide access to clinical study reports. An FDA pilot programme that asked drug makers to voluntarily submit CSRs “may fairly be qualified as a failure,” says Doshi. Only one maker, Janssen, submitted a single CSR.15

One portal run by the industry offers access to far more data, but on request only.16 It holds 3123 studies, for which 564 requests have been made and 276 granted. But researchers thinking of using CSRs for the first time are put off by the bureaucratic hurdles, says Doshi. “In general it takes six, nine months. They need your protocol, reviewed by your institutional board, so there are huge layers of bureaucracy that intervene between your interest in the data and actually getting access.”

Finally, there are the portals run by Health Canada and by the European Medicines Agency.1718 These are easy to access, with CSRs available for download within minutes of logging on. Both aim to add the CSRs of new drugs as they are approved. Older CSRs can still only be obtained on request.

The transition to transparency for Health Canada was, however, not entirely straightforward. In 2016 Doshi requested CSRs from Health Canada on oseltamivir, zanamivir, Gardasil, Gardasil 9, and Cervarix. But Doshi had to sue and win a judicial review against Health Canada to stop them imposing a confidentiality agreement when he sought data on oseltamivir, zanamivir, Gardasil, and Cervarix. The judge left no doubt that the presumption of confidentiality of drug companies’ data was a thing of the past, and, says Doshi, the system now works smoothly.19

The European Medicines Agency, conversely, faced legal challenges from the industry when it sought to publish CSRs. But there too the European Court of Justice last month came down unambiguously on the side of transparency.20 The creation of anonymised CSRs and the processing of requests has been halted since 2018, however, by Brexit and the agency’s forced relocation to Amsterdam. It may resume this summer, but in the long term UK researchers will have to find European sponsors, as the agency no longer accepts requests from outside the European Union.

One group that has not embraced CSRs, surprisingly, is Cochrane reviewers. A 2018 survey of 160 of them found that just 13% had used CSRs in a review, while 83% had never considered the possibility.21

A European Medicines Agency survey of users of its service in 2017 found that 62% were affiliated to the drug industry. Academic researchers accounted for 14%, while patients and medical professionals made up 8% each.22 But researchers probably account for close to half of the total pages requested.23

Some researchers, like the Cochrane oseltamivir group, want to test drug makers’ claims against their original data. But, says Doshi, “more people are using the data for some novel analyses than to redo the maker’s original analysis—looking for biomarkers using a greater number of studies, for example.”

The New England Journal of Medicine attracted no fewer than 143 teams when it offered a prize for original uses of clinical trial data.24

Nevertheless, debunking drug makers’ claims remains a popular goal. Oseltamivir is not the only drug to face a reckoning with its own past through CSR analysis. Researchers using CSRs found publication bias overstating benefits or undercounting harms in the cases of off-label uses of gabapentin and reboxetine.2526 But Pfizer had already pleaded guilty to criminal charges in the off-label marketing of gabapentin. As for reboxetine, its sales continued to grow despite a barrage of research questioning its safety and efficacy.

We have yet to see a drug actually “taken down” by researchers delving into unpublished data, with the possible exception of rosiglitazone (Avandia). Finding a problem is one thing, getting regulators to listen is another. The researchers are, after all, looking at the same documents the regulator saw when approving the drug. They cannot see the unpublished data until the drug is already on the market. And, as the case of oseltamivir shows, when it comes to drugs and governments, first impressions count.