Problems with UK government's risk sharing scheme for assessing drugs for multiple sclerosis
BMJ 2003; 326 doi: https://doi.org/10.1136/bmj.326.7385.388 (Published 15 February 2003) Cite this as: BMJ 2003;326:388- Cathie L M Sudlow (csudlow{at}skull.dcn.ed.ac.uk), Wellcome clinician scientista,
- Carl E Counsell, senior lecturerb
- a Department of Clinical Neurosciences, University of Edinburgh, Western General Hospital, Edinburgh EH4 2XU,
- b Department of Neurology, Aberdeen Royal Infirmary, Aberdeen AB25 2ZN
- Correspondence to: C L M Sudlow
- Accepted 21 October 2002
The National Institute for Clinical Excellence (NICE) recently announced that interferon beta and glatiramer acetate were not cost effective treatments for multiple sclerosis and could not be recommended for NHS funding.1 As a result, the Department of Health and the manufacturers developed a “risk sharing scheme” aimed at providing these drugs more cost effectively. 2 3 Treatment will be provided to ambulating patients with two or more disabling relapses in the past two years (about 15% of all patients with multiple sclerosis)4 and their progress monitored over 10 years. However, the scheme has several scientific and practical problems that we believe limit its ability to improve the care of patients in the long term. In this paper, we review the quality of the evidence on which NICE and the Department of Health reached their decisions, consider some of the problems of the risk sharing scheme, and suggest an alternative approach.
Summary points
NICE has announced that neither interferon beta nor glatiramer can be recommended for multiple sclerosis in the NHS
The Department of Health plans to make these drugs available through a risk sharing scheme that is scientifically unsound and impractical
Randomised trials suggest that azathioprine (which is 20 times cheaper) may be just as effective
The long term effectiveness of these drugs is unknown
Government money would be better spent on a long term randomised trial comparing interferon beta or glatiramer with azathioprine and no treatment
Methods
We identified randomised trials of disease modifying drugs in patients with multiple sclerosis from systematic reviews, 5–7 the Cochrane controlled trials register, and the treatment guidelines of the Association of British Neurologists.4 We also got information from discussion with colleagues, including several international experts in multiple sclerosis. We used Cochrane RevMan software to produce summary relative risks.
What is the evidence that the drugs are effective?
NICE considered data from placebo controlled trials of interferon beta and glatiramer acetate but did not assess azathioprine, which has also been widely tested in multiple sclerosis.5–31 The three drugs produce a similar reduction (15-30%) in the relative risk of a relapse at two years (fig 1). Interferon beta and glatiramer may also reduce disability (fig 2), but appropriate data were not available for azathioprine. Although interferon beta, glatiramer, and azathioprine were all associated with more patient withdrawals than placebo, the side effects were generally mild. Azathioprine may be associated with a small increased risk of neoplasia after 10 years of treatment, 32 33 but there is not enough long term experience with either interferon beta or glatiramer to exclude an increased risk of cancer.
These results, although promising, are based on limited, short term data (a few hundred patients for each drug, usually followed up for no more than two years). We therefore do not know whether the effects are sustained over the long term. Several other previously noted methodological problems7 also limit the interpretation of the results and may have biased them in favour of active treatment. These include uncertainty about the adequacy of randomisation,34 which is not always clearly described; unavoidable patient unblinding35; difficulty interpreting the outcome of confirmed progression of disability, which was generally based on the widely criticised expanded disability status score 36 37; substantial losses to follow up in a few trials; publication bias (the largest randomised trial assessing interferon beta in secondary progressive multiple sclerosis showed no overall effect on progression of disability but remains unpublished17); and funding of the trials by pharmaceutical companies, which own the data and were involved in the trials' design, conduct, analysis, and reporting.9–24Although many trials had independent data monitoring committees, it is concerning that some data from some trials have not been placed in the public domain.
Limitations of NICE's assessment of cost effectiveness
NICE's conclusions on cost effectiveness were based mainly on an analysis commissioned from the Sheffield University School of Health and Related Research. 1 38 The researchers calculated the cost per quality adjusted life year (QALY) gained at 5, 10, and 20 years after starting treatment (table). The model suggests that the threshold of £36 000 per QALY (set by the Department of Health for the risk sharing scheme) is approached only after 20 years.
Although this economic model is probably the best available for multiple sclerosis, it has several unavoidable flaws. Firstly, it depends on the quality of the evidence for effectiveness of treatment, which, as highlighted above, has major deficiencies. Treatment effects were estimated mainly from published reports. Two companies provided some additional confidential data, one refused, and one withdrew its additional data after seeing its effects on the results. 1 38
Secondly, because of the lack of long term placebo controlled randomised trials, the model compares the effects of treatment with the experience of a cohort of 1000 Canadian patients with multiple sclerosis recruited in the 1970s and 1980s and followed for an average of 25 years. 39 40 It assumes that treatment remains effective for as long as the patient takes it and that the benefit accrued is maintained after treatment stops. NICE acknowledges that this extrapolation of treatment effects becomes increasingly unreliable as the time horizon is increased. 1 38
Thirdly, the model is heavily influenced by assumptions about future discounting of costs and benefits, the proportion of patients who stop treatment prematurely and what happens to them, and the way in which the costs of disability related to multiple sclerosis are estimated (table).38 Finally, it does not consider azathioprine, which has an annual treatment cost of only about £300 a patient.
Scientific flaws of risk sharing scheme
The risk sharing scheme plans to use the Sheffield model as a basis for assessing and adjusting the real life cost effectiveness of interferon beta and glatiramer. Azathioprine has been ignored. Patients meeting the Association of British Neurologists treatment criteria4 will be assessed annually for 10 years to determine the rate of progression from no disability (expanded disability status score <4), through mild (4-5.5) and moderate (6-6.5), to severe disability (≥7). The effects of each treatment will be determined every two years by comparison with the expected progression without treatment derived from the Canadian cohort. 3 39 40 Target treatment effects have been agreed with the drug companies, and if these are not achieved, the drug costs will be reduced to maintain cost effectiveness at a threshold of £36 000 per QALY over 20 years.3 Unfortunately, the scheme has several major problems.
Non-randomised comparisons
The Department of Health circular states that the scheme is not a further trial of clinical effectiveness but a study to establish long term cost effectiveness.3 However, a reliable estimate of long term cost effectiveness first requires a reliable estimate of long term clinical effectiveness. This will not be achieved by comparing a modern cohort of patients treated in the United Kingdom with a historical cohort of Canadian patients since non-randomised comparisons give unreliable, biased results. 41 42
Lack of power calculations
The scheme will include about 7000 patients in England and Wales (plus more patients from Scotland), but the circular gives no power calculations to justify this number.3 It recognises that the non-randomised comparison will be biased and that chance may lead to imprecise measures of treatment effect. However, rather than randomising large numbers of patients, the Department of Health proposes to incorporate a tolerance margin of 10-20% in the comparison between the treated and untreated cohorts.3 It does not explain how this margin was chosen; nor is it clear whether the margin represents a relative or absolute difference in outcome.
Other biases
The risk sharing scheme is subject to several other biases. Firstly, patients already receiving treatment at the start of the scheme will be included if they fulfilled the inclusion criteria at the start of treatment and their pre-treatment disability score and other prognostic data are available.3 This will bias the comparison in favour of treatment because patients who started and then stopped treatment before the scheme because of adverse effects or perceived lack of effectiveness will not be included.
Secondly, the circular states that patients who stop taking treatment during the scheme are “to be monitored as far as possible.”3 This is not good enough. It is essential to follow up such patients because we do not know how patients respond once they stop treatment. This information is critical to the cost effectiveness calculations.
Thirdly, the scheme does not intend to have blinded assessment of outcome. Unblinded assessment of outcome in multiple sclerosis trials can result in overestimates of the effect of treatment on progression of disease.35 Hence, an apparent treatment benefit may simply be due to the expectation bias of patients and their neurologists or specialist nurses. An additional competing interest bias may be introduced by unblinded assessment of outcome being done by specialist nurses whose salaries are paid for by pharmaceutical companies.
Calculation of cost effectiveness
The estimates of cost effectiveness depend critically on the various assumptions used in the modelling process, but the actual assumptions to be used are not mentioned in the circular. These will have to be made explicit and justified before the scheme starts.3
Other issues
The circular does not state what will happen if a treatment seems ineffective. Neither does it tell us which patients will be included in the analyses or whether these will be on a truly intention to treat basis. Ideally, a proper intention to treat analysis would be ensured by information on patients giving their consent being telephoned or faxed immediately to a central site. This would avoid the loss or non-registration of patients who do not do well on their chosen treatment. However, no details of this sort have been provided.
Practical problems with risk sharing scheme
The Department of Health proposed that patient recruitment would start on 6 May 2002. However, the national coordinating team was not appointed until July 2002, ethical approval has had to be sought, and many neurologists have yet to see a detailed protocol.
The cost of the drugs (more than £50m a year) for the scheme will have to be met from existing NHS budgets. In addition, collecting data is likely to put further strain on NHS resources. The circular states that “the scheme should as far as possible build on normal clinical practice without requiring elaborate additional infrastructure” and that “data entry should be as simple as possible and arise out of normal patient contacts.”3 However, neurological services are already extremely stretched (median outpatient waiting times are about 26 weeks), and many potentially eligible patients do not have regular contact with a neurologist.43 Many additional consultant neurology and specialist nurse sessions (with appropriate administrative support) will be needed to evaluate patients who may be eligible and to follow up those who join the scheme. Normal patient contacts do not include assessment of the expanded disability status score, and so appointments will have to be longer to allow for this. It is not clear how all these additional sessions can be provided without seriously compromising the existing service, although the pharmaceutical companies will fund some. Local staff (probably specialist nurses) will need to be trained to collect, store, and transfer the additional data.
Alternative proposal
We believe that the government could spend the extra resources for patients with multiple sclerosis more effectively. Firstly, it should commission an independent, individual patient data overview of all relevant published and unpublished randomised trials of disease modifying drugs for multiple sclerosis. The overview would address unanswered questions about the trials and may go some way towards resolving the uncertainties about the effects of interferon beta, glatiramer, and azathioprine.
Secondly, the risk sharing scheme should be modified to include a concurrent, randomised control group rather than a historical cohort. Given that the Department of Health is committed to providing resources for the assessment, long term follow up, and drug costs for several thousand patients with multiple sclerosis, a long term randomised trial, run independently of the pharmaceutical industry, would be a far more scientifically (and so ethically) justifiable use of this money. Patients could be randomised three ways (interferon beta or glatiramer versus azathioprine versus no treatment) and followed up in the same manner and with the same outcomes as the existing scheme. Additional resources would be required for blinded outcome assessment and perhaps inclusion of a quality of life outcome, but a trial would probably be less expensive than the present scheme because only one third of patients would be taking an expensive drug. Careful explanation would have to be given to patient groups about why a randomised trial is the best way forward as fewer patients would be receiving active treatment. However, it is the patients who have most to gain by reliably establishing the long term clinical effectiveness as well as cost effectiveness of these treatments.
Conclusions
Any additional resources for patients with multiple sclerosis are welcome. However, these should be used to provide services that we know benefit patients and to support further, properly designed research into interventions about which there is still uncertainty. Uncertainty remains about the effectiveness and cost effectiveness of interferon beta and glatiramer, and the risk sharing scheme will neither resolve these nor determine the possible role of promising but far less expensive drugs such as azathioprine. All patients with multiple sclerosis, whether eligible for treatment under the terms of the scheme or not, deserve much better than this. The government should consider a more appropriate use of this large amount of public money.
Acknowledgments
Contributors: CS identified relevant articles and analysed the data. Both authors drafted and modified the paper and both will act as guarantors.
Footnotes
-
Funding CS is supported by a Wellcome Trust clinician scientist award.
-
Competing interests None declared.