Why don’t we have all the evidence on oseltamivir?

This week the BMJ publishes an updated Cochrane review on neuraminidase inhibitors in adults with influenza.1 The review and a linked investigation undertaken jointly by the BMJ and Channel 4 News2 cast doubt not only on the effectiveness and safety of oseltamivir (Tamiflu) but on the system by which drugs are evaluated, regulated, and promoted.

In the process of updating their review, Jefferson et al found several important inconsistencies. Prompted by a reader of their previous update,3 they attempted to reconstruct the evidence from a much cited analysis on which they had based their previous conclusions. The analysis, by Kaiser et al,4 looked specifically at the effects of oseltamivir on the risk of hospital admission and complications (pneumonia and other lower respiratory tract infections) in people with influenza. Jefferson et al noted that the Kaiser analysis was funded by the drug’s manufacturer Roche and was based entirely on 10 trials funded by Roche, only two of which had been published as articles in peer reviewed journals. All 10 included trials were authored by Roche employees and paid academic consultants. The Cochrane reviewers could find no independently funded trials of oseltamivir in healthy adults.

Other questions also arose, as outlined by one of the Cochrane reviewers Peter Doshi5 and by Deborah Cohen in our feature.2 Which authors of the trials had seen the raw data? Why were key authors of the published papers and abstracts not named in company documents submitted for drug approval, while people named in these documents were not included in the published papers? Were ghost writers involved in some of the manuscripts, as alleged by former employees of the medical communication company hired by Roche? Why were the rates of influenza infection in the trials so high? And why were serious adverse events under-reported?

Jefferson et al asked for clarification from the authors, who directed them to Roche. After initially declining to provide the necessary data, staff at Roche sent the Cochrane reviewers a selection of files, which answered some questions but still left the reviewers unable to reconstruct the Kaiser dataset. Of particular concern, the eight unpublished trials, involving 2691 patients, included one of the biggest trials of oseltamivir. More worrying still, the academic author named on the trial’s abstract in the Kaiser analysis and in company documents has told the BMJ he was not involved in the trial.

Noting that the number of complications in the trial populations was small, Roche also provided some observational studies of oseltamivir as evidence of its “real world” effects on hospital admissions. The BMJ has since commissioned Nick Freemantle and Mel Calvert to review these and additional observational studies. They concluded that the studies provided only weak evidence and that any absolute benefit was small.6

As documented in the accompanying feature,2 what should have been a straightforward exercise to confirm the evidence base for current policy and practice became instead a complex investigation. In the absence of access to the necessary data, and in contradiction to their previous update,3 Jefferson et al concluded that they have no confidence in claims that oseltamivir reduces the risk of complications and hospital admission in people with influenza.1 In doing so they have reached a similar conclusion to the Food and Drug Administration in the United States and the recent health technology assessment performed for the UK’s National Institute for Health and Clinical Excellence (NICE), which both conclude that there is insufficient evidence on complications, as Peter Doshi explains.5 Yet claims that oseltamivir reduces complications have been a key justification for promoting the drug’s widespread use. Governments around the world have spent billions of pounds on a drug7 that the scientific community has found itself unable to judge.

After being approached by the BMJ and Channel Four News, Roche committed to making the “study summaries (including key data)” available on their website. The same files as were sent to Jefferson have now been posted on the site (see roche.com). Roche has also committed to making the “full study reports” available on a password protected site soon.8 9

This case exposes how much of the evidence on drug safety and effectiveness is taken on trust. Governments and international bodies have relied heavily on the Kaiser et al analysis and on observational studies to justify the stockpiling and widespread use of oseltamivir.2 7 10 No doubt they assumed that others had looked critically and comprehensively at the complete dataset. Those others might include drug regulators and health technology assessors such as NICE, as well as journal editors and Cochrane reviewers. But that isn’t how the system works.

Drug regulators ask for certain drug company data before licensing a drug and may do their own analyses, but many data go unreviewed.11 12 NICE assessments are based on selected information from the drug companies as well as systematic reviews.13 Cochrane systematic reviews involve extensive searches for published and unpublished research reports but only rarely go back to the raw data. Systematic reviews based on individual patient data are the gold standard,14 but they are time consuming and expensive. As for journals, they may ask to see the full dataset if they suspect foul play, but this is rare. They routinely review only what is presented in the manuscript.

In being less then forthcoming with the raw data, Roche has done nothing wrong by current standards and even less by standards of 10 years ago. It has done exactly what the current system allows.

Several recent policies have been introduced to improve research reporting,15 16 17 including the posting of summary trial data by many drug companies on their websites. But even the latest policies implemented in full would not have helped in this case. An exception might be JAMA’s 2005 policy that drug company trials must include an independent statistical evaluation.18

The current system isn’t working. Worse than that, it gives a false sense of security. The system’s failures have left a legacy of drug evaluations for which, in the absence of better information, we must assume the same levels of confusion and uncertainty as for oseltamivir. The drug industry directly or indirectly undertakes the majority of all drug evaluations, so most of the evidence used to support drug policy and treatment remains shrouded in secrecy. In only a minority of cases will the data have been subject to full independent analysis and interpretation. In many if not most cases, the only people who have seen the entire dataset are company employees.

What needs to be done? Firstly, we need many more publicly funded trials and should explore ways in which the drug industry could be made to fund independent trials for licensing their drugs.19 20 Secondly, governments should enact laws that go beyond what limited legislation there is, such as that in the US. As well as timely release of summary data once a trial is completed, as required by the FDA Amendments Act, governments should mandate ready access to the raw data behind any analyses used to license and market a drug. They also need to establish mechanisms for dealing with the legacy of unreleased data on drugs already on the market. Thirdly, researchers who said in their protocols that they would do certain analyses should be required to do them or explain why they have not done so, and to make the results available. Fourthly, although the ICMJE authorship criteria (www.icmje.org/ethical_1author.html) and the Good Publication Practice guidelines17 are useful, they currently focus too much on the manuscript rather than the underlying data. Authors should be asked to state who saw the full primary dataset and who saw only summary data, who decided what analyses should be done and reported, and who is the “custodian” for the data. The custodian should have access to the full primary dataset and be able and willing to show this to systematic reviewers and other researchers. Specifically for oseltamivir, Roche should clarify without delay exactly who did what in relation to its trials. Finally, journals should consider adopting JAMA’s policy on independent statistical review and should push for independent individual patient data meta-analyses wherever possible, especially in relation to drugs to be used in common conditions and purchased in large quantities to protect the public’s health.

Researchers might not want to share their data before they themselves have been able to analyse them. They might have responsibilities to the people who took part in their trials to maintain confidentiality, or they might be cautious about possible misinterpretation. However, these concerns should not prevent access if the findings have already become part of the public record through the researchers’ own publications. Equally misplaced are industry’s concerns about threats to their intellectual property.21 This is not about releasing information on the molecular structure of the drug. Where is the legitimate commercial sensitivity around the data for analyses on which drug licensing and marketing rely? Why should the public have to rely on detective work by academics and journalists to patch together the evidence on such a potentially important drug? When vast quantities of public money, and large amounts of public trust, are placed in drugs, the full data must be accessible for scrutiny by the scientific community. Pending full disclosure and independent review of the raw data from Roche, the risks and benefits of oseltamivir remain uncertain.