Intended for healthcare professionals


Ensuring safe and effective drugs: who can do what it takes?

BMJ 2011; 342 doi: (Published 11 January 2011) Cite this as: BMJ 2011;342:c7258
  1. Tom Jefferson, researcher1,
  2. Peter Doshi, doctoral student2,
  3. Matthew Thompson, senior clinical scientist34,
  4. Carl Heneghan, clinical reader and director CEBM3,
  5. Cochrane Acute Respiratory Infections Group
  1. 1Acute Respiratory Infections Group, Cochrane Collaboration, Rome, Italy
  2. 2Program in History, Anthropology, Science, Technology and Society, Massachusetts Institute of Technology, Cambridge, MA, USA
  3. 3Department of Primary Health Care, University of Oxford, Oxford, UK
  4. 4Department of Family Medicine, Oregon Health and Sciences University, Portland, Oregon, USA
  1. Correspondence to: P Doshi pnd{at}
  • Accepted 7 November 2010

Drawing on their experience in producing a Cochrane review of neuraminidase inhibitors for influenza, Tom Jefferson and colleagues discuss how to improve the reliability of systematic reviews

In the midst of the H1N1 flu “pandemic,” the Australian and UK governments commissioned an update of our longstanding Cochrane review on neuraminidase inhibitors for influenza in (otherwise) healthy adults. The review had first been published in 1999 with updates in 2006 and 2008. While preparing the 2009 update, we received a comment from a Japanese paediatrician. He questioned our conclusion that oseltamivir (Tamiflu) reduces the risk of complications (such as pneumonia) and pointed out that the evidence underlying this conclusion in our 2006 review was based on a single paper—a manufacturer funded meta-analysis1 of 10 manufacturer trials, of which only two had been published in the peer reviewed literature.2 3 To verify the quality and reliability of our previous conclusions, we wrote to the lead author of the meta-analysis, Laurent Kaiser, to obtain original data. We also wrote to Nicholson and Treanor, authors of the two published trials in the Kaiser meta-analysis. Months later, and despite additional correspondence with Roche, oseltamivir’s manufacturer, we remained unable to access any data. In our updated 2009 review we did not include the unpublished trial data from the Kaiser paper, and concluded that the ability of oseltamivir to reduce complications was unknown.

Although our review was praised by some for highlighting important questions about the evidence base of a global public health drug,4 we were left feeling that conclusions drawn from only a proportion of all existing trials (that is, just the published ones) are wholly inadequate. The extent to which unpublished data are included in evidence syntheses is low; a recent survey found that less than 10% of Cochrane reviews did.5

Since our review, and perhaps in response to the enormous publicity generated by the joint BMJ-Channel 4 News investigation of oseltamivir,6 Roche publicly pledged to make its unpublished full clinical study reports available (box).7 We expected that these reports would provide sufficient detail to verify the findings of the Kaiser meta-analysis. However, what Roche provided was not the full study reports of the 10 trials but module 1 of seven trial reports. The tables of contents showed that the full reports probably comprise four or five modules. Unfortunately, module 1 does not include the analysis plan, randomisation schedule details, the study protocol with a list of deviations, or detailed case histories for patients experiencing adverse events.

The scope of clinical trial data

  • Raw data might be considered the most broad, detailed, and comprehensive type of data, comprising any records created in preparing for and carrying out clinical trials—trial methods with protocol, investigator notes, individual patient data, ethics committee reports, clinical case notes, management committee’s minutes, transcripts/videos of meetings, contracts, book keeping, and so on. Access to raw data “enables data checking, thorough exploration, and re-analysis of the data in a consistent way”8

  • Clinical study reports are a distillation and summary of the raw data from a given individual trial, but, importantly, are unabridged reports that (depending on study size) can be thousands of pages in length. They should report a trial’s background and rationale, methods, results, and discussion and also include important study documents such as the analysis plan, randomisation schedule, study protocol (with a list of deviations and amendment history), detailed case histories for patients who have adverse events, example case report forms, and list of ethics committees who approved the research

  • Published trials are those trials for which a primary publication appears in the scientific press, typically written in the structure of Introduction, Methods, Results, and Discussion (IMRAD). Published trials usually fit on a few sheets of paper and as such represent a summary of the much larger clinical study report

  • Unpublished trials are those trials for which no primary publication appears in the scientific press. Information and details about unpublished trials may however appear as secondary publications in the so called grey literature, scientific congresses, conference posters, or other abstracted form

  • Abstracts are IMRAD structured extreme syntheses of a trial or a group of trials, usually not longer than 200 words

  • Regulatory data contextualise other types of data about a trial. When the correspondence between a regulator and drug manufacturer is accessible to third parties, independent researchers gain access to an additional level of data about a trial (or trials) relevant to systematic reviews but not directly produced from a trial. This and other regulatory data such as regulators’ medical and statistical reviews can be essential to understanding individual trials within the context of a trial programme

This additional material, although incomplete, raises an additional reason to doubt the integrity of the published evidence. For example, the first of the two published studies in the Kaiser meta-analysis does not mention serious adverse events,9 and the second states that “there were no drug-related serious adverse events.”10 However, the partial study reports that Roche made available to us list 10 serious adverse events (in nine subjects) in the two trials, three of which were classified as possibly related to oseltamivir.

If published studies are incomplete and do not report important outcomes, the current process for conducting systematic reviews is not sufficiently rigorous, and in some cases it risks turning into unsolicited authoritative advertising for the drug industry. In other words, we need access to all unpublished data, even of trials published in the peer reviewed literature.

Change of methods

To make sense of the evidence we must look not at single trials individually but at the whole trial programme, a point recently made forcefully by Ioannidis and Karassa.11 It is naive to think that single trials and narrow questions relevant to, say, 10 trials out of a total of 28 in the dataset would provide a manageably sized and reliable systematic review. A broad study question such as ours (“What are the effects of neuraminidase inhibitors?”) requires evidence from the whole trial programme to answer it, and we have published a formal protocol detailing our new methodology.12

The first steps are fairly clear, if somewhat laborious: compiling a full list of trials carried out by the drug industry and non-industry funders. Independent investigators must do this themselves before contacting drug companies to verify the completeness and accuracy of lists. For example, we identified a large oseltamivir trial by Roche Shanghai13 (ML16369) that Roche Basel did not appear to know about—it was omitted from Roche’s list of 101 sponsored and supported trials—despite the existence of an English language study report dated July 2001 (Mengzhao Wang, personal correspondence, 2 Oct 2009).

Next we must request full clinical study reports for each trial. However, even if the manufacturers agree, there is no guarantee that the reports are reliable. And what should we do when incomplete reports are provided, as we found with Roche? Regulatory material can help us answer the questions. The US Food and Drug Administration and Japan’s regulatory body make available medical officers’ reports written as part of drug evaluation. Although regulatory documents do not provide primary trial data, they can offer invaluable insights into trials. Regulatory materials, including correspondence and scientific reports, can potentially answer why, for example, US regulators today still require that oseltamivir’s label clearly states that it has not been shown to reduce complications whereas the opposite is stated on European and Australian labels. If, as it appears, the trials did not use standardised definitions of secondary complications (such as pneumonia), this may preclude meta-analysis.1

We then believe reviewers should construct a table of all the evidence to organise efforts and clarify what kind and amount of information exists for each of the trials included in a review. By doing this we found that the largest phase III treatment trial of oseltamivir (M76001) was not only never published but is little mentioned in regulatory documents. Why so little discussion would have occurred about such an important trial remains unclear.

To assess the integrity of a full trial programme (as opposed to that of individual trials, for which Cochrane methods probably suffice) we need new tools. Using insights from the growing literature on reporting bias,14 and guided by our experience with neuraminidase inhibitors, we constructed a table of null hypotheses to test for the presence of biases potentially affecting trial programmes and their conclusions (table). The series of questions based on documented cases allow a critical overview of the trial programme.

Reporting bias testing framework for comparing evidence from multiple regulators, manufacturer clinical study reports, trial registries, and published trials

View this table:

Finally, a decision must be reached about whether a traditional review (including quantitative meta-analysis) can be done or whether the full trial programme and its dataset is of insufficient integrity to allow quantitative synthesis. We believe that you can pool data only if the evidence base is reasonably complete and sound. If a quantitative analysis is not possible, independent investigators must report the reasons why they are unable to satisfactorily assess the effects of a drug. Saying why you cannot do what you set out to do may prove as valuable as providing numerical results.

Publication and role of medical journals

By looking at some of the earlier versions of the industry sponsored reviews of neuraminidase inhibitors we discovered that the report by Kaiser et al was not the first meta-analysis mixing published and unpublished material; GlaxoSmithKline had done the same for zanamivir in a 1999 review in which only four of the seven trials included were published.15 Clearly the inclusion criteria for our 2006 Cochrane review 16 were inconsistent as we included the Kaiser1 meta-analysis but not the zanamivir meta-analysis. In retrospect, Kaiser’s paper should never have been included.

Both our review and the Kaiser meta-analysis were published in two of the world’s most prestigious peer reviewed medical journals.1 17 The fact that they included data from unpublished randomised trials shows the extent to which trust underlies current practice, but is this any longer acceptable? We cannot expect busy doctors to be aware that trials and meta-analyses of drugs in respected publications are heavily influenced by drug companies’ marketing decisions on what is and isn’t published. We believe we need to change the way information is identified, appraised, and synthesised and regard any industry sponsored trial published in journals as marketing, unless proved otherwise.

Who can ensure safe and effective drugs?

It is important to not lose sight of the goal: providing doctors, patients, and policy makers with unbiased systematic assessments of drugs. Governments currently have this responsibility, but regulators are under-resourced18 and powerful disincentives for rigorous review exist because candid analyses may undermine current policy. In the case of oseltamivir, regulators also disagree about whether it reduces complications, and agencies or departments within a single country may make inconsistent claims.3

The case of the diabetes drug rosiglitazone also shows that regulators are fallible. It took a legal settlement in 2004 requiring GlaxoSmithKline to post results of all of its clinical trials18 before Nissen et al could do a meta-analysis of all 42 clinical trials of rosiglitazone, 35 of which were unpublished.19 Their analyses showed a substantial increase in risk of myocardial infarction and cardiovascular death, results that were later replicated by the FDA using patient level data.19 20 Thus independent systematic analyses remain vital.

How can robust independent assessment of drugs be carried out in a world where data are privately owned? The answer is to make the data freely available: we should accept nothing less than a full dataset. Before licensing a drug—and certainly before large purchase decisions are made—our governments and policy makers should ensure that all researchers can access data in sufficient detail to allow for the independent exploration and re-analysis of trials.

Researchers, the public, and lay and scientific media will need to work together to put pressure on industry to embrace the ethical responsibility to release data in the public interest. Currently legal action is often required to achieve this (as happened with rosiglitazone). Ideally, regulators would make full clinical study reports publicly available once a regulatory decision is reached. This is a daunting task considering that the FDA submission for oseltamivir was at least 363 volumes (although we do not know the number of pages).

For the present, we call on journals to require, as a condition for consideration of publication of a randomised trial, submission of the most detailed report available (anonymised to protect patient privacy), in addition to the summary manuscript for publication. This concept is not dissimilar to the format of a short print version with full text online that many journals already use. Access to both parts would allow peer reviewers to carry out random checks and to compare detailed reports with submitted manuscripts. If patient privacy can be assured, posting the detailed report as an online supplement will also improve post-publication review. Had this happened 10 years ago, the omission of serious adverse events possibly related to oseltamivir may have been noticed and corrected before publication. An overview of the trial programme should also be submitted explaining the rationale and findings of each trial. We need to understand how and why a particular trial was designed and conducted—that is, how every trial can potentially advance our knowledge.

Anyone conducting experiments on humans has obligations transcending patents and commercial confidentiality. Physician involvement in any type of reporting bias may be unethical, which carries implications for professional misconduct. We must remember that trial participants are performing a service to humanity, entering a potentially risky situation for the sake of determining the toxicity and effectiveness of a new drug. Withholding the results of such trials, and in some cases archiving them in such a way that they come to light only after prolonged and detailed investigation, seems ethically dubious. Ethical committees should be among the most vocal calling for the freeing of data.

We do not yet know whether sound scrutiny is feasible with a journal’s resources, but we intend to use the proposed review of neuraminidase inhibitors to attempt to quantify these tasks. We will be keeping a journal of our review with a resource tally, and proposing methods for in-depth scrutiny of a whole trial programme of a new drug. It is time the media, the Cochrane Collaboration, and any reader interested in knowing what they are prescribing or are being prescribed increase the pressure on policy makers. If you swallow a medication, you need to know how it works—for real.


Cite this as: BMJ 2011;342:c7258


  • Research, doi:10.1136/bmj.c7153
  • We thank Chris Del Mar, Mark Jones, and Rokuro Hama. Thanks also to the Australian NHMRC and the UK NHS Research and Development fund for grants to enable the 2009 healthy adults review update, and to the UK NIHR for granting funding for our current review update. Also thanks to the Hugh Hampton Young fellowship committee at MIT for its generosity. The NIHR school of Primary Care Research provides financial support for Dr Carl Heneghan and funding for an investigators’ meeting in Oxford (UK).

  • Contributors and sources: All authors were authors of the separate relevant Cochrane reviews. All authors contributed to the writing of this essay.

  • Competing interests: All authors have completed the unified competing interest form at (available on request from the corresponding author) and declare no support from any organisation for the submitted work. In 1998-9, TJ was an ad hoc consultant for Hoffman-La Roche; all authors are co-applicants for a NIHR grant to carry out the Cochrane review.


View Abstract