Neuraminidase inhibitors—the story behind the Cochrane reviewBMJ 2009; 339 doi: https://doi.org/10.1136/bmj.b5164 (Published 08 December 2009) Cite this as: BMJ 2009;339:b5164
- Peter Doshi, doctoral student
- 1Program in History, Anthropology, Science, Technology and Society, Massachusetts Institute of Technology, Cambridge, MA 02139 USA
Since August 2009, our Cochrane review team has tried to obtain the data needed to verify claims that oseltamivir (Tamiflu) lowers serious complications of influenza such as pneumonia. We failed, but in failing discovered that the public evidence base for this global public health drug is fragmented, inconsistent, and contradictory. We are no longer sure that oseltamivir offers a therapeutic and public health policy advantage over cheap, over the counter drugs such as aspirin. If the public is to trust in public health policies, the scientific basis informing knowledge of the harms and effects of those interventions must be public and open to independent analysis.
How a Cochrane review update turned controversial
Systematic reviews are designed to synthesise the most reliable evidence on the effects of interventions. Following the outbreak of influenza A/H1N1 in April 2009, the UK NHS National Institute of Health Research commissioned an update of the Cochrane systematic review of neuraminidase inhibitors in healthy adults. In retrospect, our review began on a naive note. Although the review had last been updated in 2008, our new task was to include a safety assessment component. Tom Jefferson, who led the review, wrote to the group then just being formed, “Dear Friends…although it is always dangerous to pre-judge the issue, I expect no new effectiveness data but a lot of pharmacovigilance data.” Two days later, a paediatrician from Japan, Keiji Hayashi, submitted a comment to the Cochrane Collaboration that would ultimately leave us doubtful about the ability of systematic reviews to deal with the challenges of contemporary pharmaceutical evaluation1 (see Web Extra of cited paper: Hayashi’s criticism on previous review).
Hayashi pointed out that although Jefferson et al’s previous review2 found oseltamivir effective in reducing important complications of influenza such as pneumonia, that conclusion was drawn from a single peer-reviewed study by Kaiser et al.3 The Kaiser study itself had meta-analysed 10 manufacturer funded trials from the late 1990s, of which only two were published in peer reviewed journals.4 5 The remaining eight were apparently either unpublished or published only in abstract form. Hayashi suggested that the unpublished trials were central to demonstrating oseltamivir’s ability to reduce lower respiratory tract complications of influenza, and challenged us to “appraise the 8 trials rigidly.” Our team subsequently attempted to verify the data for ourselves, but in doing so found a series of inconsistencies in the evidence for oseltamivir’s effectiveness and safety.
A maze of inconsistencies
Despite funding the Kaiser meta-analysis, which concluded that oseltamivir reduces complications, oseltamivir’s manufacturer, Roche, apparently did not itself make any such claims about complications. A Tamiflu.com webpage reads, “Treatment with TAMIFLU has not been proven to have a positive impact on these outcomes,” referring to pneumonia, other respiratory diseases, and influenza related death.6
The previous Cochrane review had found oseltamivir effective in reducing the duration of symptoms in influenza-like illness. But here, again, Roche’s position countered Cochrane’s; Roche stated that oseltamivir was ineffective against influenza-like illnesses not caused by influenza.7 Drug product labelling in the United States, European Union, and Japan also states that oseltamivir only works for true influenza virus infections (box 1).
Box 1—Contradictory statements made about the potential benefits of oseltamivir
Complications of influenza
Roche (roche.com) (2005): “Tamiflu delivers ... [a] 67 percent reduction in secondary complications such as bronchitis, pneumonia and sinusitis in otherwise healthy individuals”8
Kaiser et al (2003): “Our analysis found that early treatment of influenza illness with the neuraminidase inhibitor oseltamivir significantly reduced influenza-related LRTCs, associated antibiotic use, and the risk of hospitalization. This effect was observed in both at-risk subjects and otherwise healthy individuals.”3
EU EMEA (2009): “The proportion of subjects who developed specified lower respiratory tract complications (mainly bronchitis) treated with antibiotics was reduced from 12.7 % (135/1063) in the placebo group to 8.6 % (116/1350) in the oseltamivir treated population (p = 0.0012).”22
US CDC (2008): “In a study that combined data from 10 clinical trials, the risk for pneumonia among those participants with laboratory-confirmed influenza receiving oseltamivir was approximately 50% lower than among those persons receiving a placebo and 34% lower among patients at risk for complications (p<0.05 for both comparisons) [Kaiser, 2003].”18
US HHS (2005): “Treatment with a neuraminidase inhibitor (oseltamivir [Tamiflu] or zanamivir [Relenza]) will be effective in decreasing risk of pneumonia, will decrease hospitalization by about half (as shown for interpandemic influenza), and will also decrease mortality.”19
Australia TGA (2009): “The overall incidence of secondary illnesses (such as bronchitis, otitis media, sinusitis and pneumonia) requiring antibiotic medication was reduced by 50% in TAMIFLU treated subjects when compared with placebo.”26
Previous Cochrane review (2008): “Oseltamivir 150 mg daily is effective in preventing lower respiratory tract complications in influenza cases (OR 0.32, 95% CI 0.18 to 0.57)” 2
Roche (tamiflu.com) (2009): “Treatment with TAMIFLU has not been proven to have a positive impact on [asthma, emphysema, other chronic lower respiratory diseases, pneumonia, other respiratory diseases, pneumonitis, and influenza-related death]”6
US FDA (2008): “Serious bacterial infections may begin with influenza-like symptoms or may coexist with or occur as complications during the course of influenza. TAMIFLU has not been shown to prevent such complications.”23
Japan PMDA (2009): no mention of complications on drug product information sheet24
Burch et al (2009): “Overall, little information was available on the effects of either drug on the incidence of complications, and there were very few events, in both the healthy adult and at-risk populations. Furthermore, weaknesses in the available evidence limit the reliability and the ability to generalise any results relating to the eff ect of these drugs on the rates of complications.”12
Canadian Coordinating Office for Health Technology Assessment (2002): “There is insufficient evidence to show that oseltamivir reduces complications, hospitalizations or death when used to treat: normally healthy people suspected of having influenza, or; those who are at risk for developing complications” 27
Duration of symptoms of influenza-like illness (ILI)
Nicholson et al (2000): “The duration of illness was significantly lower in the intention-to-treat [ILI] population than in the other subgroups because of the high proportion of influenza-infected patients in this population.”5
Treanor et al (2000): “As expected, the greatest benefit of therapy was seen in individuals with evidence of influenza virus infection. However, analysis of the entire population also demonstrated a significant benefit of treatment.”4
Previous Cochrane review (2008): “Time to alleviation of symptoms [for ILI were] … in favour of the [neuraminidase inhibitor] treated group … (hazard ratio 1.20, 95% CI 1.06 to 1.35).”2
Roche (2006): “We acknowledge that oseltamivir is ineffective against influenza-like illness caused by viruses other than influenza.”7
EU EMEA (2009): “Oseltamivir is effective only against illness caused by influenza viruses. There is no evidence for efficacy of oseltamivir in any illness caused by agents other than influenza viruses.”22
US FDA (2008): “There is no evidence for efficacy of TAMIFLU in any illness caused by agents other than influenza viruses Types A and B.”23
Japan PMDA (2009): “Tamiflu has no effect against infections except those caused by influenza viruses type A and type B.”24
CI=confidence interval; Australia TGA=Australia Therapeutic Goods Administration; EU EMEA=European Medicines Agency; ILI=influenza-like illness; LRTC=lower respiratory tract complications; Japan PMDA: Pharmaceuticals and Medical Devices Agency, Japan; US FDA=US Food and Drug Administration; US HHS =US Department of Health and Human Services.
These inconsistencies concerning the ability of oseltamivir to work against all influenza-like illness and reduce the risk of complications pointed to the uncomfortable conclusion that the Cochrane Collaboration had promoted—by trusting the validity of other work in the scientific literature—efficacy claims more optimistic than even the drug manufacturer’s.
Reality, however, proved more complex. The Tamiflu.com website where Roche declares that oseltamivir is not proven to reduce complications contains a footnote: “THIS [WEB]SITE IS INTENDED FOR U.S. AUDIENCES ONLY.” On Roche.com, the global website, the manufacturer asserts that “Tamiflu delivers ... [a] 67 percent reduction in secondary complications such as bronchitis, pneumonia and sinusitis in otherwise healthy individuals”.8 Furthermore, among international products labels we reviewed, only the European Medicines Agency approved the statement that oseltamivir reduces the complications of influenza (box 1), causing us to wonder whether governments had similar access to trial data.
Data pertaining to oseltamivir’s safety were equally confusing. We discovered the US Food and Drug Administration (FDA) postmarketing Adverse Event Reporting System, which collects reports of adverse events worldwide relating to FDA approved drugs, had fewer entries in total than Roche’s own postmarketing database held for neuropsychiatric classified adverse events alone.1 Of 2466 such neuropsychiatric events in the Roche global safety database between 1999 and 15 September 2007, Roche researchers classed 562 as “serious”.9 Over this period, the FDA Adverse Event Reporting System database only holds 1805 adverse event reports of any kind.
In publications—or secrecy—we trust?
Analyses of and reliance on publications in the scientific literature are the key elements of practising evidence based medicine. Essential to this practice is a trust that trials are carried out properly and that published reports accurately reflect the original study protocol (including pre-specified primary outcome measures) and the study data. Hayashi’s comment questioning the wisdom of trusting unpublished, industry-sponsored trial data revealed the degree to which Cochrane reviews are fundamentally based on the premise that the published literature about a drug’s efficacy and safety is backed by hard, verifiable data.
Obtaining raw data from properly carried out trials on complications is the only way to resolve the inconsistencies surrounding oseltamivir’s effect on reducing complications. On behalf of the review team, Jefferson wrote in August to the authors of the Kaiser paper, but was told that they no longer had the files and to contact Roche. Jefferson also wrote to authors of the two peer reviewed published trials used in Kaiser’s meta-analysis. One responded, but once again Jefferson was directed to the manufacturer.
Jefferson first requested data from Roche in early September. On 2 October, Roche indicated a willingness to share data, but not openly. It furnished Jefferson with a “confidentiality agreement,” containing a clause saying that the signee (Jefferson) agrees “not to disclose ... the existence and terms of this Agreement” (see Web Extra: Roche confidentiality agreement). Roche apparently intended not only to keep its data concealed, but also to conceal the fact that it was quieting people through a secrecy clause.
Jefferson did not sign the confidentiality agreement, but wrote the next day asking for clarification, which he never received. On 7 October the company asked Jefferson to restate which data he was seeking. After Jefferson’s answer, Roche said it was unable to provide data because it had already provided it for a similar meta-analysis being started by an independent expert influenza group. The Cochrane request, Roche said, might conflict with that review. In return, Jefferson challenged Roche to outline its concerns and explain why sending data to multiple groups of independent researchers should pose a problem. Roche did not answer these questions, but eight days later (21 October), it unexpectedly emailed Jefferson seven 10-17 page excerpts of company reports from all clinical trials used in the Kaiser meta-analysis.
Our team analysed the data, and Jefferson wrote to Roche explaining that the files were insufficient to verify the effects on complications claims in Kaiser and the methods used in the trials1 (see Web Extra of cited paper: Comments on Kaiser et al’s paper). Roche responded on 28 October, saying it would send more information the following week. Jefferson informed them that our deadline was now past, but that we would accept any additional information for future updates. As of 1 December we have heard nothing. [Since this article was finalised, Roche has made the data from company reports on the Kaiser trials available on its website and has committed to making the full study reports available on a password protected site shortly.]
The previous Cochrane review placed its trust in publications and included Kaiser’s unpublished data, but to do so once again, despite our inability to obtain data sufficient to perform an independent analysis, would have shifted our position from that of trust in publication to that of trust in secrecy. We dropped Kaiser’s paper from our analysis.
After four months of seeking the data used to support the findings of Kaiser and colleagues, we have come up empty-handed. This raises the troubling question of whether Cochrane reviewers should have ever included the study in their review in the first place. The previous reviewers endorsed the conclusion that oseltamivir reduces complications such as pneumonia and bronchitis by implicitly trusting that the unpublished data were verifiable. This trust now seems naive. The fact that a trust in unpublished data extends to many systematic reviews of neuraminidase inhibitors by other researchers10 11 12 and was not questioned until the Hayashi comment is even more troubling, raising questions about the ability of high quality reviews to be performed by volunteer organisations.
Although the excerpted reports Roche provided us were insufficient to verify claims about the complications of influenza, they did clarify an outstanding anomaly of the published trial reports. According to the published studies, patients randomised into trials had febrile influenza-like illness with at least one respiratory symptom (such as cough or sore throat) and at least one constitutional symptom (such as fatigue)—in other words, the clinical syndrome usually called “the flu” that presents in routine clinical care. Without laboratory testing, one cannot know whether influenza virus or some other agent is causing these patients’ discomfort.13 In past influenza seasons, US virological surveillance data indicate that at peak “flu season” the proportion of respiratory specimens testing positive for influenza typically reached between 25-35%, but over the entire season, influenza viruses were found in only a minority (14%) of tested patients. By contrast, in the ten Roche trials analysed by Kaiser, an average 68% of randomised patients tested positive for influenza (figure⇓). The discrepancy seems the likely outcome of a special patient inclusion methodology mentioned in company reports but absent in the corresponding published papers.4 5 Company reports state: “Centers were activated to recruit subjects during an influenza outbreak in the locality, detected using standardized surveillance techniques.” Thus, the trial population seems likely to have been unrepresentative of the general population of people with influenza-like illness, the majority of whom do not have influenza (seemingly even during the current pandemic14) and will not benefit from neuraminidase inhibitors.
If oseltamivir is no better than placebo in its ability to reduce the complications of influenza, and if it is also ineffective against influenza-like illness not caused by influenza, then the drug’s ability to treat the symptoms of influenza may be similar to that of an NSAID such as aspirin. Although aspirin is clearly not indicated for children because of its association with Reye’s syndrome, head to head trials of oseltamivir versus an NSAID or paracetamol (for children) may be the only way to establish the relative benefits of these drugs.
With respect to safety concerns, FDA reporting rules turn out to have important limitations. Although manufacturers are under mandatory reporting requirements, adverse events occurring outside the United States judged to not meet the “both serious and unexpected” criteria are under no requirement to be reported. Thus the public Adverse Event Reporting System database relies on manufacturers to honestly and accurately judge whether adverse events reported in conjunction with their products are “serious” and therefore must be reported—or not. In the case of oseltamivir, considering that 75% of global consumption has occurred in Japan, this has important implications for our knowledge of its safety.
Public health drugs
Since oseltamivir’s approval in 1999, neither American nor Japanese regulators have ever approved statements that the drug lowers rates of influenza related complications. The FDA reportedly even required Roche to declare: “Tamiflu has not been proven to have a positive impact on the potential consequences (such as hospitalizations, mortality, or economic impact) of seasonal, avian, or pandemic influenza.”15 Despite the work of these regulators, public health officials trusted the conclusions of the published literature at face value. Citing the Kaiser paper, several recommendations from the Centers for Disease Control and Prevention stated that oseltamivir reduces the risk of hospitalisation and pneumonia.16 17 18 The US even partly based its national pandemic preparedness strategy on similar assumptions (table).19 Billions of dollars were spent building drug stockpiles, and oseltamivir was elevated to the status of a public health drug.
Like vaccines, public health drugs get deployed on a population basis, directed by national or international level policy decisions. As witnessed in the UK, when the government declared that oseltamivir may be used to treat all symptomatic cases even without consultation with a physician or laboratory diagnosis, hundreds of thousands of courses of the drug were used in a fortnight.20 Mass prescription carries serious responsibilities. While the evidence base for all approved drugs should be sound, the evidence base for public health drugs must be of the highest quality, publicly available, and open to independent scrutiny.
Evidence based medicine should not hinge on a singular trust in any one institution, particularly in for-profit companies whose primary responsibility is to shareholders and investors rather than the public’s health. As John Abraham observed, there seems a tragic irony in the situation: when pharmaceutical companies do not trust each other, why should the public or government be asked to trust them?21 If governments have the authority to purchase and govern the use of multi-billion dollar drug stockpiles, they should require access to primary research data and commit the resources to independently evaluate the efficacy and safety of that drug. Box 2 contains ideas on where to start.
Box 2—A short (and incomplete) list of higher standards for evidence based public health decision making
Clarify expectations and provide evidence
Public health policies aiming to implement mass interventions should clearly state and identify (before approving the policy) the expected harms and benefits of that intervention. Clarity about the expectations of a drug can help reviewers assess whether a drug meets predefined performance targets and reveal important inconsistencies or shortcomings, flagging them as areas of uncertainty for which better evidence is needed.
Strengthen trial registration processes
All trials should be centrally registered (perhaps with the government in initiatives similar to ClinicalTrials.gov). A field for recoding the citation to any publications resulting from a given trial, and a field to explain why a study has not been published within a year of completion, would help third party investigators match clinical trial to publication, and bring more awareness of the importance of publishing “negative” results.
Make patient level data available
Individual patient data are often the only way to resolve questions about the effects of a drug. Publicly available anonymised patient level datasets on regulator websites would increase transparency and enable independent re-analyses of trial results.
Reduce the reliance on trust
Methods of data collection (such as adverse events reporting systems) that rely on companies to self-evaluate potential harms may lead to bias. Where mandatory reporting requirements already exist (for example, in the US FDA Adverse Events Reporting System25) reduce potential bias by making them apply to all known adverse events, and make these data publicly accessible, enabling independent researchers to investigate the possible significance of reports. For manufacturers, internet-only based reporting of adverse events would lessen the workload of regulators and facilitate entry of all known adverse events into public databases.
April 2009—CDC reports two cases of novel swine-origin A/H1N1 influenza
June—WHO declares A/H1N1 influenza a pandemic
July—UK NHS National Institute of Health Research commissions update of Cochrane review of neuraminidase inhibitors in healthy adults; lead researcher Tom Jefferson forms review team
14 July—Keiji Hayashi submits comment to Cochrane Collaboration stating that unpublished, manufacturer funded trial data are central to the claim that oseltamivir reduces complications
August—Jefferson attempts to obtain data necessary from authors of meta-analysis3 that used the unpublished data; he is directed to speak with the manufacturer (Roche)
September—Jefferson requests data directly from Roche
October—Roche sends Jefferson confidentiality contract. Contract is not signed, but Roche later sends Jefferson excerpts of trial reports, which are insufficient to verify the claims questioned by Hayashi
December—Cochrane review update goes to press unable to verify claims that oseltamivir reduces complications of influenza
Cite this as: BMJ 2009;339:b5164
I would like to acknowledge the support of my Cochrane co-authors, and in particular thank Yuko Hara, Tom Jefferson, two external referees, and the BMJ editors for their helpful comments on the draft manuscript as well as Ted Postol for his advice.
Contributors and sources: PD has studied and published on public health policy responses to epidemic disease and has a strong research interest in the intersection of science and society, particularly under situations of high uncertainty. This article arose out of a sense that the story behind the review needed to be told. PD is sole author and guarantor.
PD has completed the Unified Competing Interest form at www.icmje.org/coi_disclosure.pdf (available on request from author) and declares that (1) PD has no financial support from Roche for the submitted work; (2) PD has no relationship with commercial entities that might have an interest in the submitted work in the previous three years; (3) their spouses, partners, or children have no financial relationships that may be relevant to the submitted work; and (4) PD is a co-author of the Cochrane review about which this article is written, which may be a relevant competing interest to the submitted work.
Provenance and peer review: Commissioned, externally peer reviewed.