Search for evidence goes onBMJ 2012; 344 doi: http://dx.doi.org/10.1136/bmj.e458 (Published 17 January 2012) Cite this as: BMJ 2012;344:e458
Researchers working on the latest Cochrane Collaboration meta-analysis of the evidence on oseltamivir (Tamiflu) have claimed that the drug’s manufacturer is still denying them access to full trial data. This is despite Roche, the Swiss company behind oseltamivir, pledging in the BMJ two years ago that it would make “full study reports” available.1 As a consequence, confusion still surrounds the evidence on oseltamivir and guidance on how doctors should prescribe it.
The latest Cochrane review on oseltamivir, a drug on which governments around the world have spent billions of pounds, is published today (18 January). But the Cochrane reviewers have received only part of the clinical study reports—the summary of the study methods and the results. The company says this is enough for the Cochrane group to conduct their review, but Cochrane denies this.
Meanwhile, GlaxoSmithKline—the makers of the less popular antiviral drug zanamivir (Relenza)—have offered individual patient data. When the BMJ asked Roche why it was refusing to make its data available, despite GSK’s promises, it said it refused to answer until it had had the chance to see the full Cochrane review.
Clinicians can be forgiven for being confused about what the evidence on oseltamivir says. Last September, the UK Department of Health announced that in the event of a flu outbreak general practitioners would be able to prescribe the drug to all patients, not just those who were at risk of complications of influenza.2
Meanwhile, the European Centre for Disease Prevention and Control, the US Centers for Disease Control and Prevention, and the World Health Organization differ in their conclusions about what the drug does (box 1). And in some cases these contradict the product claims on drug labels allowed by the US and EU regulators. Confusion also surrounds whether the trials conducted on seasonal influenza as part of the drug’s approval process apply to influenza pandemics.
Policy makers have come to contradictory conclusions at various timesAs Birmingham public health consultants Jacky Chambers and Andrew Rouse have recently asked, how could a drug that the National Institute for Health and Clinical Excellence did not deem to be cost effective in people who were not at risk of complications now be readily available to everyone?3
Hospital clinicians have told the BMJ that up until three years ago oseltamivir was on the market for treatment and prevention of influenza but was rarely used in hospital. The evidence at that time did not support the use of oseltamivir for reducing the duration of symptoms or preventing transmission. What’s more, in a hospital setting it was unlikely that clinicians would know how many days a patient had had symptoms before being admitted. The trial evidence suggested that it was effective only if given within 48 hours of the onset of symptoms.
Nevertheless, clinicians said they were under pressure, particularly from parents, to prescribe oseltamivir, despite their concerns about the evidence. The then chief medical officer for England, Liam Donaldson, and the Health Protection Agency requested that all flu deaths were investigated. Part of the investigation was to establish whether patients had been given oseltamivir. Now oseltamivir has become a mainstay of flu treatment in critical care.4
Concerns about that evidence
The latest Department of Health advice comes two years after a joint investigation by the BMJ and the Cochrane Collaboration first raised concerns over the evidence base for oseltamivir.4 5 6 7 One of the main reasons for stockpiling oseltamivir in case of a pandemic was the company’s claim that the drug reduced complications and hospital admissions from influenza when given to healthy adults—claims based on a Roche funded non-systematic review of treatment trials in healthy adults published in the Archives of Internal Medicine in 2003.8
Since failing to verify these claims from the published literature, the Cochrane Collaboration and the BMJ have been pushing Roche to provide the full trial data.
But the company has said in an email to the Cochrane group, “Concerning the request for further detail, we have reviewed the proposed protocol and believe all the trial data you require for your meta-analysis are more than adequately covered in the access you already have to the study reports of the 10 Kaiser paper clinical trials. These comprise 3200 pages of information and are much greater than would typically be available from literature publications.”
Do data matter?
Controversy and questions about the evidence base seem not to have affected Roche’s blockbuster influenza drug—which William Burns, Roche’s head of pharmaceuticals, said in a presentation in September 2009 was a “hidden surprise.”9
It has earned the company vast revenues. In 2009 alone, oseltamivir sales grew sharply to SFr3.2bn (£2.2bn; €2.7bn; $3.4bn) after substantially increased demand during the pandemic.10
In December 2009, after the release of the Cochrane findings, Laurent Kaiser—lead author of the Roche funded review—told Swiss newspaper Neue Zürcher Zeitung, “Many researchers, including me, kept on saying that there are not enough data that clearly prove a reduction of severe complications.”
A point with which Pekka Kurki, rapporteur for the European Medicines Agency (EMA), concurred. Interviewed for a previous BMJ investigation, he said: “What was unclear and is still unclear is what is the impact of oseltamivir on serious complications. Circulating influenza was very mild when oseltamivir was developed and therefore it is very difficult to say anything about serious complications. The data did not clearly show an effect on serious complications—it was not demonstrated by the RCTs [randomised controlled trials].”11
Roche, however, managed to shrug off these observations. In 2002, the EMA approved the drug for the prevention of secondary complications in healthy adults, drawing the same conclusions as the Roche funded review.
Documents obtained under the Freedom of Information Act from the agency suggest that influenza experts Anneke Linde and Rene Snacken (who had also appeared in oseltamivir marketing material) discussed the benefits of oseltamivir during the drug approval process, citing trials already published in medical journals. Some of these papers had used ghostwriters, as Roche has previously admitted.
Anneke Linde, a Swedish state epidemiologist, said in her presentation that vaccine failures had been identified—this was therefore an indication for the use of oseltamivir in prophylaxis and treatment. Belgian public health expert, Rene Snacken also presented to the EMA during the drug approval process in 18 February 2002 and discussed the need for chemoprophylaxis and called for the use of oseltamivir during a pandemic11—even though the trial data was conducted on a mild virus. Although the BMJ had asked for conflict of interest statements for these presentations, the EMA was never able to produce any.
The US Food and Drug Administration, on the other hand, warned that the drug had not been shown to reduce complications, even sending Roche a letter in 2000 warning it not to make this claim.
“You have claimed reductions in severity and incidence of secondary infections with Tamiflu that are misleading because they are not supported by substantial evidence,” the FDA said.
Although not specified in the product summary initially, the FDA required addition of a further note: “Serious bacterial infections may begin with influenza like symptoms or coexist with or occur as complications during the course of influenza. Tamiflu has not been shown to prevent such complications.”
No standard definitions of complications in either paediatric, elderly, or adult trials were ever prepared and incorporated in the trials.
However, the Centers for Disease Control and Prevention—which comes under the same governmental umbrella as the FDA—paid no heed to the FDA conclusions when issuing its advice on pandemic influenza (box 1). And the US Health and Human Service plan advised that oseltamivir would reduce complications, hospital admissions, and mortality.12
Box 1: Centers for Disease Control and Prevention
The most recent CDC review, Antiviral Agents for the Treatment and Chemoprophylaxis of Influenza in January 2011, continues to cite the Roche funded review. The CDC says that the risk of pneumonia was 50% less than among those receiving placebo and 34% lower for those at risk of complications—statistics that the FDA does not accept based on the same trial evidence. The report also suggested that the use of oseltamivir would lead to a 50% reduction in hospital admissions, although that figure wasn’t statistically significant. In effect, the CDC has chosen Roche’s interpretation of a dataset in the published literature rather than the assessment of the US drug regulator. This is the same dataset that Roche said they had submitted to the regulators.
When asked about the evidence it cited, the CDC side stepped, saying: “CDC does not comment on FDA actions.”
It also continues to cite key trials of oseltamivir published in the Lancet and JAMA that have been ghostwritten—as Roche has admitted.5 The BMJ asked the CDC if ghostwriting matters.
It replied: “CDC relies upon the ACIP [Advisory Committee on Immunization Practices] to make recommendations. The ACIP bases its recommendations upon the available published data.”
Its interpretation of the last Cochrane review is also flawed. Rather than saying that the Cochrane group was unable to verify the claims made that oseltamivir reduces complications and hospital admissions, the CDC says that the review concludes that: “Adverse events associated with antiviral medications are generally mild and self-limited (see Adverse Events) but might result in morbidity resulting from medication side effects that outweigh the potential benefit of antiviral chemoprophylaxis.”25
The Cochrane review published today might go some way to explaining the difference in opinion among the regulators. It also poses new questions about the drug’s proposed mode of action, conduct of trialists, and how the medical community approaches scientific evidence.
It also highlights the need for collaborative working between the US and European regulators—who are often under resourced—and the importance of making a complete set of trial data available for independent scrutiny by academics in a timely manner.
Other sources of trial data
After getting no joy from Roche, the Cochrane group turned to other sources in order to complete their data set. They asked the FDA and the European and Japanese regulators for the data they held on oseltamivir. As well as published comments in the public domain, they wanted the clinical study reports—detailed, unpublished clinical trial data containing in-depth descriptions of protocol rationale, methods analysis plans, trial results, and organisational documents (such as contracts).
Under current EU legislation, these are available under the Freedom of Information Act. But there is hope that the EMA will publish these in the future. In June 2011, Andreas Pott, now deputy executive director of the EMA, wrote to the BMJ to say that the agency planned to start publishing clinical trial reports in the next few years.13 Reports would be published for all drugs submitted for approval as soon as the EMA had made a decision on the drug, he said.
The EMA has just confirmed to the BMJ that a decision on this is still in discussion, but it expects that it will publish all the annexes that come with drug trial submissions—a helpful development to all those who depend on the EMA to provide details on trials on drugs, such as NICE.
In this, for once, the EMA is ahead of the FDA. While the EMA released more than 20 000 pages of regulatory documents in May 2011 to the Cochrane reviewers, the FDA has been far less efficient in handling a similar request.
According to one of the Cochrane reviewers, Tom Jefferson, the Cochrane group sent the FDA a freedom of information request for clinical study reports in January 2011 and case report forms for serious adverse events in October 2010. They have received neither.
The FDA has thousands of pages of documents about oseltamivir that are publicly available, but these give an incomplete picture of how the FDA handled questions over oseltamivir trials and, in some cases, pose more questions than they answer.
Different regulators, different approaches
What is glaringly apparent from comparing the regulatory documents is that the FDA and EMA took different approaches to the data submitted to them.
Firstly, they had different views on what constituted a full data set. In their detailed scrutiny of the documents obtained, the Cochrane reviewers discovered that there were five modules of Roche data relating to the oseltamivir trials. When the EMA released the clinical study reports to the Cochrane group, they sent only modules 1 and 2 for all but one cardiotoxicity trial. A letter from Xavier Luria, head of safety and efficacy at the EMA, dated 24 May 2011 stated correctly that this was all that they had on file. However, a spokesperson for the EMA admitted that the other three modules had not been received. This means that the agency has not seen the individual patient data—and whatever else was contained in the missing modules—despite having had the authority to ask for it.
This didn’t stop Daniel Brasseur, chair of the EMA’s paediatric committee, writing to the Lancet in 2006 to criticise an earlier Cochrane review that had concluded that oseltamivir did reduce complications and hospital admissions for underestimating the benefits of Roche’s pills.14
“In my opinion, the article gives an overly negative view on the use of antiviral medicinal products for prophylaxis of influenza, both seasonal and pandemic,” he said, adding: “The risk of facing an influenza pandemic without adequate stocks of antivirals is far greater than the risk of failing to accurately predict their usefulness.”
If staff at the EMA had closely inspected the submitted documents it would have spotted that the effect on complications was based on trial outcomes which were left to the discretion of the local clinician without microbiological proof —a point picked up by the FDA. The FDA also noted that there were inconsistencies with how participants were followed up and had their symptoms documented.
Even though the FDA seems to have done a much more thorough analysis of the data, it chose not to review the largest treatment trial—one that was conducted almost exclusively in North America. This trial remains unpublished, its only public airing having been at a conference presentation by US academic, John Treanor. Dr Treanor later told the BMJ his recollection of the trial was “dim.”5
The FDA said that its standard requires at least two adequate and well controlled trials to show efficacy. To date—and through piecemeal searching—the Cochrane group has identified 127 trials. Nowhere is the full trial programme publicly available.
Rather than make the data public when questions over the effectiveness of their drug arose, Roche turned to two epidemiologists at Harvard, Professors Marc Lipsitch and Miguel Hernan. They had access to individual patient data from trials in the 2003 Roche funded review plus one extra trial—more than Cochrane and the EMA had access to. They presented the results of their meta-analysis at a conference in Washington, DC, in December 2010. They suggested that the unpublished trials were no more favourable to oseltamivir than the published ones 15—although the Cochrane group dispute some of their findings and suggest that there are more trials than are included in their review. In an email to the BMJ, Professor Hernan said that they fully “support adherence to the principles of reproducible research for all studies.”
But there was another issue concerning the trial design that—based on the documentation available to the Cochrane group—seems to have escaped the regulators. This has led to questions over the mode of action of the drug.
In the treatment trials available to the Cochrane group, patients were randomly allocated to receive oseltamivir or a placebo, and at this stage both groups were a similar size. Participants were then tested for influenza and antibody response. The analysis of the trial results was based on only those patients testing positive and those with a fourfold rise in antibody response. The Cochrane group noticed that the number in the oseltamivir arm was significantly smaller than in the placebo arm in several of the most cited studies, which later appeared in JAMA and the Lancet.16 17
The group found that those receiving oseltamivir were much less likely to have a fourfold rise in antibodies (odds ratio 0.79; 95% confidence interval 0.70 to 0.90, P<0.001). Why might this be? The Cochrane group says that oseltamivir seems to affect antibody production—something also seen in the post-exposure prophylaxis studies. They hypothesise that oseltamivir has a non-specific antipyretic rather than antiviral mode of action given its effect on symptoms.
In a document to investigators, Roche acknowledged the difference between the placebo and the oseltamivir arms in antibody production, saying that it “provides corroborative evidence of the antiviral effect of oseltamivir.” The company also noted that the decreased magnitude of antibody response did not mean that oseltamivir recipients would be at greater susceptibility of infection in future seasons.
Publicly, however, Roche, maintains that oseltamivir does not prevent infection nor affect antibody production.18 When asked by the BMJ, the company refused to explain how the drug worked.
“How is it possible that oseltamivir prevents cases of influenza when part of the definition of prevented cases in oseltamivir trials was based on absence of antibody response?” the Cochrane group asks.
Effects on transmission
As well as treatment, one major reason to stockpile oseltamivir before vaccines were ready for use was to stop the virus from spreading. WHO’s 2007 guidance assumed that the drug would contain the spread of inﬂuenza, either buying time for an organised response with longer term interventions such as vaccines, which take time to produce, or by completely stopping an emergent pandemic.19
“The basic containment strategy uses a geographically based approach in which antiviral medications and non-pharmaceutical measures are used in a defined area surrounding the initial cases (i.e. Containment Zone) to restrict the virus from spreading beyond the Containment Zone,” the guidance said.
NICE, too, also suggested in 2000 that oseltamivir “successfully interrupts the transmission of influenza within households,” saying that the drug would “control the spread of influenza in other closed communities associated with high risk of transmission.”20
It was a position that Gilead, the original makers of the drug, were understandably keen to promote. A November 1999 press release said: “Post-exposure prophylaxis, 75 mg once daily for seven days, protected close contacts of influenza-infected patients against influenza by 92%, and interrupted influenza transmission within households by 89%.”21
But if there had been a careful evaluation of the trial data, WHO and other agencies might have been less hasty to advise the use of the drug for stopping spread.
Roche conducted a trial that was subsequently published in JAMA in 2001.22 Far from providing evidence of stopping transmission of the virus, the design of the trial did not allow for comparison of the effects of treating index cases with oseltamivir on spread versus placebo. Nor, according to FDA documents, was there consistent measurement of viral shedding—although the published paper suggested that “viral shedding was inhibited in contacts taking oseltamivir.”22 Arnold Monto (an influential WHO adviser) was included as an author. Stopping viral shedding was cited by WHO and others as one of the benefits of the drug.
In a later trial conducted by Professor Fred Hayden,23—a Department of Health adviser, a key adviser to WHO, and the coordinator of the Wellcome Trust’s influenza centre—all index cases were given oseltamivir. The interruption of transmission had not been shown in trials.
Professor Hayden’s pivotal role in oseltamivir trials would come up again later. Unlike the EMA, the FDA carried out site inspections. A letter it sent in 2000 raised concerns over the consent form in his prophylaxis trial that had been published the previous year in the New England Journal of Medicine.
The FDA noted that using the phrase “‘will receive $300.00 for participating in and completing the study. No payment will be made to you if you withdraw from the study for personal reasons …’ to be an improper procedure. When subjects are to be paid for participating in a study, the payment should be prorated for the subject’s actual participation in the study in order to avoid the possibility of coercion.”
Given the documented side effects of oseltamivir, participants taking the drug might have dropped out—leading to what is known as attrition bias. If compliance with the drug was a problem, it would undermine its use as the drug of choice for prophylaxis.
In the published paper, the authors wrote that the most frequent reason for withdrawal was adverse events or intercurrent illness and these occurred in similar proportions. “According to the number of returned capsules, the degree of compliance with the study-drug regimen was high and was similar among the three groups,” it added.
What is unclear is if this is a standard patient consent form—Roche declined to comment, and the Cochrane group are awaiting further Freedom of Information requests from the FDA. The FDA, however, allowed this trial to stand as evidence of effectiveness for prophylaxis.
Professor Hayden, however, told the BMJ: “The patient consent form used in this study was drafted by my staff, edited by me, and reviewed and approved by both the sponsor (Roche) and our local IRB [institutional review board] prior to its use. Please note that our standard consistently has been to fully compensate participants who withdraw or are withdrawn for medical reasons; this information was included in the consent document used for the prophylaxis study.”
The Cochrane group now plans to analyse adverse reactions to oseltamivir and withdrawals from trials in more detail. But without a well supported independent assessment of all the data, confusing and contradictory messages reign.
Box 2: WHO and access to data
Since the early days of pandemic preparedness, WHO has become more conservative in its recommendations of oseltamivir. However, shortly after the 2009 Cochrane review was published, the drug came up for inclusion on WHO’s list of essential medicines— medicines that it considers essential for maintaining health.
WHO’s reviewers had raised concerns about the addition of the drug.
“Paucity of good data has undermined previous findings for oseltamivir’s prevention of complications from influenza. I am aware of the fact that the professionals cannot be left without any potential tool that could help them in the fight against pandemic influenza A/H1N1, but it seems difficult to me to recommend oseltamivir and zanamivir at the basis of lack of independently funded trials, with insufficient evidence of preventing complications,” one reviewer said.
Another suggested that given the lack of data “including oseltamivir in the EM[essential medicines] list would be the end of the essential medicines concept by WHO.”
But before it committed cash strapped health systems to buying up the drug—with some developing countries even turning to the World Bank for a loan to pay for it24—did WHO try to validate the unpublished data? Or did they rely on what was in the public domain and on the assessments of the regulators?
Publication of all trial data and clinical study reports by the regulators would have been useful for WHO too. Lisa Bero, chair of the group, confirmed that WHO does ask for as much trial data as possible before making these decisions. A source told the BMJ that Roche had refused to comply with WHO’s request for more data. Roche declined to confirm if they had turned WHO’s request down. Nevertheless, the drug made it on to the list and there was no new reassessment of additional trial data on the accompanying report—instead observational data formed the backbone of the decision.
Cite this as: BMJ 2012;344:e458
Competing interests: The author has completed the ICJME unified disclosure form at www.icmje.org/coi_disclosure.pdf (available on request from the corresponding author) and declares no support from any organisation for the submitted work; no financial relationships with any organisation that might have an interest in the submitted work in the previous three years; and no other relationships or activities that could appear to have influenced the submitted work.
Provenance and peer review: Commissioned; not externally peer reviewed.