Re: Two Ebola treatments halve deaths in trial in DRC outbreak
To the Editor:
A recent news item in the BMJ [1] reported that a trial of the anti-viral vaccines mAB114 and REGN-EB3 were ended early because they met early stopping criteria for effectiveness. This is indeed encouraging for patients suffering from Ebola, given its high mortality rate, and it is very understandable that health care workers would want to get these drugs to patients as rapidly as possible. However, stopping clinical trials prematurely based on interim analyses is fraught with difficulties. As pointed out in a forthcoming article, [2] trials that have been ended early tend to overestimate the effects of the intervention, as Pocock and Hughes [3] warned 30 years ago. This was reinforced by the systematic review by Montori et al. [4] In their review of 143 studies, they concluded that “Trials stopped early for benefit, particularly those with few events, often report treatment effects that are larger than typical of interventions that have been definitively studied” (p. 2209). That team also showed that the smaller the number of events, the larger the overestimates, and that even when there were 500 events, the overestimates were small but important. [5]
Complicating the picture even further, Ebola is not a single disease, but consists of five viruses, four of which are harmful to humans. [6] Decreasing the sample size by ending the study early may preclude sub-group analyses, such as determining if these treatments are effective for some of the viruses but not others, or if they are more effective for one sex rather than another.
Thus, if the past is any guide, we may expect that the magnitude of the effect of these drugs will decrease as they are administered to more patients. Needless to say, we hope that this does not occur, but we should not be overly surprised if it does.
References
1. Dyer O. Two Ebola treatments halve deaths in trial in DRC outbreak. BMJ 2019;366:l5140 doi: 10.1136/bmj.l5140.
2. Streiner DL. Should clinical trials be terminated early? Clinical Therapeutics in press.
3. Pocock SJ, Hughes MD. Practical problems in interim analyses, with particular regard to estimation. Control Clin Trials. 1989;10(suppl 4):209S-221S. doi.org/10.1016/0197-2456(89)90059-7
4. Montori VM, Devereaux PJ, Adhikari NKJ et al. Randomized trials stopped early for benefit: A systematic review. JAMA. 2005;294:2203-2209. doi:10.1001/jama.294.17.2203
5. Bassler D, Briel M, Montori VM, et al. Stopping randomized trials early for benefit and estimation of treatment effects: Systematic review and meta-regression analysis. JAMA. 2010;303:1180-1187. doi: 10.1001/jama.2010.310
6. Kuhn JH, Becker S, Ebihara H, Geisbert TW, Johnson KM, Kawaoka Y, et al. (December 2010). "Proposal for a revised taxonomy of the family Filoviridae: Classification, names of taxa and viruses, and virus abbreviations". Archives of Virology. 155 (12): 2083–103. doi: 10.1007/s00705-010-0814-x
Competing interests:
No competing interests
26 August 2019
David L. Streiner
Emeritus Professor
McMaster University
St. Joseph's Healthcare, Mountain Campus, Hamilton, Ontario, Canada L8N 3K7
Rapid Response:
Re: Two Ebola treatments halve deaths in trial in DRC outbreak
To the Editor:
A recent news item in the BMJ [1] reported that a trial of the anti-viral vaccines mAB114 and REGN-EB3 were ended early because they met early stopping criteria for effectiveness. This is indeed encouraging for patients suffering from Ebola, given its high mortality rate, and it is very understandable that health care workers would want to get these drugs to patients as rapidly as possible. However, stopping clinical trials prematurely based on interim analyses is fraught with difficulties. As pointed out in a forthcoming article, [2] trials that have been ended early tend to overestimate the effects of the intervention, as Pocock and Hughes [3] warned 30 years ago. This was reinforced by the systematic review by Montori et al. [4] In their review of 143 studies, they concluded that “Trials stopped early for benefit, particularly those with few events, often report treatment effects that are larger than typical of interventions that have been definitively studied” (p. 2209). That team also showed that the smaller the number of events, the larger the overestimates, and that even when there were 500 events, the overestimates were small but important. [5]
Complicating the picture even further, Ebola is not a single disease, but consists of five viruses, four of which are harmful to humans. [6] Decreasing the sample size by ending the study early may preclude sub-group analyses, such as determining if these treatments are effective for some of the viruses but not others, or if they are more effective for one sex rather than another.
Thus, if the past is any guide, we may expect that the magnitude of the effect of these drugs will decrease as they are administered to more patients. Needless to say, we hope that this does not occur, but we should not be overly surprised if it does.
References
1. Dyer O. Two Ebola treatments halve deaths in trial in DRC outbreak. BMJ 2019;366:l5140 doi: 10.1136/bmj.l5140.
2. Streiner DL. Should clinical trials be terminated early? Clinical Therapeutics in press.
3. Pocock SJ, Hughes MD. Practical problems in interim analyses, with particular regard to estimation. Control Clin Trials. 1989;10(suppl 4):209S-221S. doi.org/10.1016/0197-2456(89)90059-7
4. Montori VM, Devereaux PJ, Adhikari NKJ et al. Randomized trials stopped early for benefit: A systematic review. JAMA. 2005;294:2203-2209. doi:10.1001/jama.294.17.2203
5. Bassler D, Briel M, Montori VM, et al. Stopping randomized trials early for benefit and estimation of treatment effects: Systematic review and meta-regression analysis. JAMA. 2010;303:1180-1187. doi: 10.1001/jama.2010.310
6. Kuhn JH, Becker S, Ebihara H, Geisbert TW, Johnson KM, Kawaoka Y, et al. (December 2010). "Proposal for a revised taxonomy of the family Filoviridae: Classification, names of taxa and viruses, and virus abbreviations". Archives of Virology. 155 (12): 2083–103. doi: 10.1007/s00705-010-0814-x
Competing interests: No competing interests