Intended for healthcare professionals

Education And Debate

Synthesising licensing data to assess drug safety

BMJ 2004; 328 doi: (Published 26 February 2004) Cite this as: BMJ 2004;328:518
  1. Klim McPherson, visiting professor in public health epidemiology (klim.mcpherson{at},
  2. Elina Hemminki, research professor2
  1. 1Nuffield Department of Obstetrics and Gynaecology, Research Institute, Churchill Hospital, Oxford OX3 7LJ
  2. 2National Research and Development Centre for Welfare and Health, Health and Social Services, PO Box 220, 00531, Helsinki, Finland
  1. Correspondence to: K McPherson

    Small randomised trials conducted for licensing purposes should record data on adverse results and be made public

    The safety of drugs is important. For full information we need to assess not only the immediate effects but also unexpected longer term effects on serious disease like coronary heart disease or cancer, especially for drugs that will be widely used. Reliably assessing the safety of drugs, however, is fraught with problems such as rare events, long follow up, strong vested interests, and biased reporting. The example of hormone replacement therapy and risk of cardiovascular disease shows some of the problems and presents useful lessons.

    Lessons from hormone replacement therapy

    Observational studies and trials on intermediate cardiovascular variables indicated that oestrogen and progesterone supplements might protect menopausal women from cardiovascular disease as well as menopausal symptoms. The evidence was convincing. For example, the nurses' study of 120 000 women followed for 30 years estimated the adjusted relative risk for coronary heart disease at 0.47 (95% confidence interval 0.32 0.69) for women currently taking hormone replacement therapy compared with never users.1 This was a common finding in observational studies and understandably led to the strong belief that hormone replacement therapy would be protective. Since the drugs also benefited lipid profiles,2 the argument seemed invincible.

    If the results were correct, the risk:benefit ratio of hormone replacement therapy was unambiguously positive.3 Hormone replacement therapy would be beneficial even for asymptomatic women, notwithstanding possible detrimental effects such as an increased risk of breast cancer. These data affected marketing and prescribing; prevention of coronary heart disease became an added indication among symptomatic women, although the licensed indications were for menopausal symptoms and the prevention of osteoporosis.

    Embedded Image

    Hormone replacement patch: better use should have been made of existing evidence on the risks of HRT

    Credit: AJ PHOTO/SPL

    The enthusiasm, however, was based on a leap of faith—an assumption that the results would be the same in an unbiased comparison of use of hormone replacement therapy between otherwise similar women. Observational comparisons create many opportunities for bias, and randomised trials were thus clearly required. The large US Women's Health Initiative randomised study was ultimately stopped prematurely in 2002 because the data monitoring process indicated that after five years the perceived balance of benefits did not compensate for the observed increased risks of coronary heart disease, breast cancer, and stroke.4 The UK study was stopped prematurely because the funders considered recruitment slow and considered it unlikely that the trial would provide substantial evidence to influence clinical practice in the next 10 years.

    Scraping the randomised barrel

    These trials might not have been so necessary if better use had been made of existing evidence. Analyses suggested selection bias in observational studies of use of hormone replacement therapy,5 and this was supported by many similar analyses. Also, in 1980 the Coronary Drug Project, a double blind secondary coronary prevention trial among men, had shown that compliers to placebo had a highly significant reduced relative risk of death from coronary heart disease over five year (0.64) compared with men who did not comply.6 This protection remained even after 40 baseline variables had been adjusted for, and remains difficult to explain. Those who take drugs are compliers, and thus compliance bias could be important. None the less, enthusiasts for hormone replacement therapy thought these doubts had little relevance because most cohort studies suggested such large benefits.

    We remained concerned about the validity of the results. Therefore, well before the publication of the Women's Health Initiative trial results, we retrieved and analysed the available randomised studies of the short term efficacy of various hormone replacement therapies.7 Many of these studies were used to provide evidence for licensing. We chose randomised studies with a non-hormonal control, three or more months of treatment, and mention of adverse events, including cardiovascular episodes by allocation.

    Twenty three trials met the criteria and included a total of around 2000 women allocated to hormone replacement therapy and 1300 to control treatment. A higher proportion of the women taking hormone replacement therapy had cardiovascular events than women in the control groups. Crude estimates put the relative risk at around 1.39 for cardiovascular outcomes and 1.64 for outcomes including thromboembolic events, neither of which was significant. However, if the true relative risk was actually 0.5 for use of hormone replacement therapy, these estimates were both significantly different. This suggested that hormone replacement therapy was not as protective as the observational data had shown.

    When we published these findings in 1997, we were ridiculed.8 “For one, I shall continue to tell my patients that hormone replacement therapy is likely to help prevent coronary disease,” asserted one expert commentator.9 Critics claimed that the choice of trials was selective, the quality of trials was inadequate, and the follow up too short. Against all the observational evidence, these results just looked wrong.

    We sought to improve the methods by seeking unpublished randomised licensing data using the same criteria as for published data. We were able to obtain data in Finland, ultimately by resorting to the High Court, which rejected objections from the companies to the Ministry of Health.10 Apparently obtaining such data would not be possible in the United Kingdom (Michael Rawlins, personal communication 2003).11

    When the extra data from the six unpublished studies that met our criteria were added, the pooled relative risk for cardiovascular events increased to 1.78. We tested this against a protective relative risk value of 0.7 and 0.5, and it was significant in both cases. The evidence now hinted at publication bias; the relative risk in the unpublished trials was around 4.25 for cardiovascular events. Altogether 29 of the 200 existing trials (15%) provided useful information; 30% of the 200 trials had reasonable controls, but only 4% properly recorded cardiovascular events. We often had to retrieve this information from data sheets.

    Our 1997 results agreed well with the those of the Women's Health Initiative primary prevention trial, which reported in 2002 an overall relative risk for coronary heart disease with current use of hormone replacement therapy of 1.29 (95% confidence interval 1.07 to 1.85). Beral et al's overview of primary prevention trials and secondary prevention trials,12 which omitted the small trials we had used, estimated the effect of hormone replacement therapy to be 1.11 (0.96 to 1.30). Since the relative risk of coronary heart disease in the first year of the Women's Health Initiative study was also 1.78 (later revised to 1.8113), our results can no longer be accused of being systematically biased because of the low proportion of satisfactory trials we were able to include.

    Embedded Image

    Credit: TEK/SPL

    Summary points

    Efficacy of new drugs has to be proved in randomised trials

    Recording of rare adverse events is currently haphazard and unreliable in efficacy trials

    Many of these trials are not in the public research domain

    Systematic synthesis of trials with reliable recording of adverse events would enable earlier detection of unexpected effects

    Regulators should require drug manufacturers to record adverse effects and make the results public

    Why did observational and randomised results differ?

    The differences between the estimates from randomised trials and those from other studies must be a consequence of systematic biases. Selection bias is common but generally difficult to entirely accommodate. Compliance bias is poorly understood and must be caused by other subtle influences on risk related to compliance, since the most important evidence comes from findings in the placebo group. It seems from a systematic comparison of observational and randomised evidence on hormone replacement therapy that only coronary heart disease is subject to these biases importantly.14 The risk estimates for all the other common diseases associated with hormone replacement therapy were similar between observational and randomised data. Perhaps compliance bias affects coronary heart disease particularly.

    Making randomised licensing data more useful

    Systematic synthesis of all data from well conducted small clinical (efficacy) trials would have revealed the effect of hormone replacement therapy on cardiovascular risk much earlier, even than 1997. At least 200 trials had studied the impact of hormones on physiological phenomena, laboratory values, osteoporosis, symptoms, or various health problems but few fully reported adverse effects. Many of the studies were not publicly available. Small trials required for licensing would be more useful in studying unintended effects if they were more systematically reported and analysed. So how can we ensure similar effects are not unnecessarily missed in future?

    Regulators should require that all efficacy studies record all outcomes, whether or not they are thought contextually relevant. Such studies should be in registers of clinical trials, and when legitimate anxiety about the safety of the products is raised, the data from all such studies should be made available to independent scientists and regulators. This is not a great deal to ask, given the importance of the questions. Skegg and Doll first requested it a long time ago for precisely these reasons.15 How long will it take us to learn? How many women were needlessly exposed to an increased risk of cardiovascular disease?


    This paper was first presented at a meeting jointly organised by BMJ Knowledge and the Cochrane Collaboration on balancing benefits and harms in 2003.16


    • Contributors KMcP sits on the Committee on Safety of Medicines and on its hormone replacement therapy expert working party and has a protracted interest in long term drug safety. EH has an interest in health technology assessment and the methodology of clinical trials. She assessed all the trials included in these studies, extracted the data on events when not recorded in summary, and obtained permission from the Finnish High Court to access unpublished trials. Both authors have been involved in health and sociological research on hormone therapy.

    • Competing interests EH led a trial on postmenopausal hormone replacement therapy in Estonia in which drugs were donated by Wyeth.


    View Abstract

    Log in

    Log in through your institution


    * For online subscription