Observational studies of disease modifying therapies for multiple sclerosis: methodological challenges and opportunities
We read with great interest the review article titled “Disease modifying therapies for relapsing multiple sclerosis” by Wingerchuk and Weinshenker in the BMJ on 22 August 2016. (1) The authors review the current evidence about the efficacy of the approved disease modifying therapies (DMTs) for multiple sclerosis (MS) based on pivotal clinical trials, discuss various therapeutic strategies, and suggest relevant factors to consider in order to incorporate a personalized treatment approach in MS. The authors also highlight the efforts of others to examine the long-term effects of DMTs for MS including extension studies of clinical trials, and observational studies. Our comments are focused on the authors’ methodological appraisal and interpretations regarding some of the observational studies discussed in the article.
Wingerchuk and Weinshenker refer to our earlier study when stating “A large retrospective population based cohort study showed no benefit on progression of disability; it lacked initial randomization, however, and biases may have contributed to lack of evidence of benefit (for example, exclusion of patients with perceived “benign” disease)…”. (1) We investigated the association between interferon beta exposure and disability progression in patients with relapsing-remitting MS in the “real-world” setting by conducting a retrospective cohort study using “prospectively” collected data (1985-2008) from British Columbia, Canada. (2) We found no strong evidence of an overall association of interferon beta exposure with the hazard of reaching an Expanded Disability Status Scale (EDSS) score of 6, a clinically relevant and important disability milestone. (2) Similar findings were observed when using EDSS 4 as well as another important outcome in MS – onset of secondary progressive MS. (2,3) We also applied additional sophisticated analytical methods (e.g., marginal structural models) and reached the same conclusions. (4,5) However, it does remain possible that certain subgroups of patients may gain long-term benefit from interferon beta treatment. (6)
While we agree that a major threat to any observational study is bias from factors that cannot be observed, controlled, and/or inadequately incorporated into the analysis, (7) it is rather misleading to merely refer to lack of “initial randomization” (an inevitable limitation of all observational studies) without referring to the methods used to minimize confounding by indication including propensity score adjustment based on patient’s baseline characteristics, inclusion of both historical (pre-interferon beta era) and contemporary (post-interferon beta era) controls, and consideration of a broad range of potential confounders. We also addressed immortal time bias, (8) a common pitfall in observational studies, by treating exposure to interferon beta as a time-dependent variable. As discussed in our paper at length, residual confounding by indication, or any unmeasured confounding could still be present. (2)
We did not exclude “patients with perceived benign disease” as stated by Wingerchuk and Weinshenker. Patients with fewer than two prospective EDSS measurements from baseline to study end were excluded because our main outcome was based on time to reach a confirmed and sustained EDSS score of 6. (2) Our study included only patients attending British Columbia MS clinics (given the British Columbia government’s reimbursement scheme for DMTs). We therefore acknowledged the possibility that very mild or very severe disease might have prevented attendance at clinic. However, we believe the systematic occurrence of one of these scenarios is unlikely. (2)
Wingerchuk and Weinshenker add “… other studies that adjusted for risk of disability to achieve more uniform cohorts report benefit”, (1) referring to two observational studies from Italian centers which reported a positive effect of immunomodulatory therapies, using similar outcomes to us (EDSS disability milestones and onset of secondary progressive MS). (9,10) However, issues regarding immortal time bias, and the use of a control group including individuals too ill to start DMTs raised concerns about those findings. (11,12) A subsequent independent reanalysis (12) of one of the Italian studies (10) found no apparent beneficial association once the immortal time bias was considered. Wingerchuk and Weinshenker also cite a large well-designed observational study from the UK, where the 6 year analysis suggested that DMTs have a modest effect on slowing down disability progression (but do not completely halt the progression of disability) when compared to a natural history cohort, providing evidence of cost-effectiveness of the DMTs. (13) However, it is also worth noting that the magnitude of the effect was rather similar to that found in our British Columbia, Canada study; both were modest. (2,13) We reported a hazard ratio of 0.77 (95% CI 0.58–1.02) for confirmed EDSS 6 using the historical controls (pre-interferon beta era); (2) the UK study authors indicated that this was consistent with theirs, as “the EDSS progression ratio for most of (the) analyses (was) about 70%.” (13) Findings from the UK’s 10 year analyses were equally modest, leading the authors of the UK study authors to caution on the impact of these drugs. (14)
We agree with Wingerchuk and Weinshenker that observational studies of DMTs in MS highlight “the challenges inherent in establishing meaningful therapeutic benefits in relatively unpredictable, multiphasic, chronic diseases for which patients are free to start, switch, and stop treatments”. (1) We would also like to emphasize the value and potential of well-designed observational studies, and appropriate statistical methods in providing evidence of associations between treatment exposure and long-term disease outcomes in the “real-world” setting – outside the idealized and rather artificial environment of clinical trials with strict inclusion/exclusion criteria and rigid protocols. (15) Observational studies thus provide important complementary information to patients, health providers, payers, and policy makers.
1. Wingerchuk DM, Weinshenker BG. Disease modifying therapies for relapsing multiple sclerosis. BMJ 2016;354:i3518.
2. Shirani A, Zhao Y, Karim ME, et al. Association between use of interferon beta and progression of disability in patients with relapsing-remitting multiple sclerosis. JAMA 2012;308:247-56.
3. Zhang T, Shirani A, Zhao Y, et al. Beta-interferon exposure and onset of secondary progressive multiple sclerosis. Eur J Neurol 2015;22:990-1000.
4. Karim ME, Gustafson P, Petkau J, et al. Marginal structural Cox models for estimating the association between beta-interferon exposure and disease progression in a multiple sclerosis cohort. Am J Epidemiol 2014;180:160-71.
5. Karim ME, Petkau J, Gustafson P, Platt RW, Tremlett H, Be AMSSG. Comparison of statistical approaches dealing with time-dependent confounding in drug effectiveness studies. Stat Methods Med Res 2016 [Epub ahead of print].
6. Shirani A, Zhao Y, Karim ME, et al. Investigation of heterogeneity in the association between interferon beta and disability progression in multiple sclerosis: an observational study. Eur J Neurol 2014;21:835-44.
7. Jepsen P, Johnsen SP, Gillman MW, Sorensen HT. Interpretation of observational studies. Heart 2004;90:956-960.
8. Suissa S. Immortal time bias in observational studies of drug effects. Pharmacoepidemiol Drug Saf 2007;16:241-9.
9. Bergamaschi R, Quaglini S, Tavazzi E, et al. Immunomodulatory therapies delay disease progression in multiple sclerosis. Mult Scler 2012; 22:1732-40.
10. Trojano M, Pellegrini F, Fuiani A, et al. New natural history of interferon-beta-treated relapsing multiple sclerosis. Ann Neurol 2007;61:300-6.
11. Koch M, Mostert J, De Keyser J, Tremlett H, Filippini G. Interferon-beta treatment and the natural history of relapsing-remitting multiple sclerosis. Ann Neurol 2008;63:125-6; author reply 126-7.
12. Renoux C, Suissa S. Immortal time bias in the study of effectiveness of interferon-beta in multiple sclerosis. Ann Neurol 2008;64:109-10; author reply 110.
13. Palace J, Duddy M, Bregenzer T, et al. Effectiveness and cost-effectiveness of interferon beta and glatiramer acetate in the UK Multiple Sclerosis Risk Sharing Scheme at 6 years: a clinical cohort study with natural history comparator. Lancet Neurol 2015;14:497-505.
14. Duddy M, Palace J, Lilford R, et al. The United Kingdom multiple sclerosis risk-sharing scheme: final 10 year results. ECTRIMS Online Library 2016; 211. 32nd Congress of the European Committee for Treatment and Research in Multiple Sclerosis. London, UK. September 2016. Available at: http://onlinelibrary.ectrims-congress.eu/ectrims/2016/32nd/147053/martin....
15. Frakt AB. An observational study goes where randomized clinical trials have not. JAMA 2015;313:1091-2.
Competing interests: No competing interests