Intended for healthcare professionals

Rapid response to:

Research News

Steroid injections through the eardrum reduce dizziness in Meniere’s disease, study finds

BMJ 2016; 355 doi: https://doi.org/10.1136/bmj.i6185 (Published 18 November 2016) Cite this as: BMJ 2016;355:i6185

Rapid Response:

Lessons learned from a recent superiority trial on intratympanic injections in refractory unilateral Meniere’s disease? Commentary on Patel et al.

Patel and colleagues [1] report on a double-blind, two-armed, randomized controlled study in patients with unilateral refractory Meniere's disease (MD) comparing treatment effects of two different intratympanic injections: methylprednisolone (experimental) and gentamicin (comparator). The trial is based on evidence that intratympanic gentamicin seems effective compared to placebo at reducing vertigo attack rates in this target population [2,3].

The authors conclude that there is no evidence that the therapies differ with respect to a reduction of attack rates. The primary comparison was the mean difference (experimental minus comparator) in attack rates during the predefined 6-month assessment period (months 19 to 24) resulting in delta = –0.9 (95% CI –3.4 to 1.6), implying that the mean attack rate during the evaluation period is 0.9 (but not significantly) higher in the active control compared to the experimental group. However, no evidence of a difference is not evidence of no difference [4].

In terms of the primary efficacy analysis it would have been more appropriate to use a difference in absolute change approach ("difference in difference") comparing both groups with respect to the mean change in attack rates between baseline (6-month pre-enrollment phase) and end-of-trial (months 19 to 24). A change of 17.4 on active control and of 14.8 on experimental treatment implies a mean difference in absolute change (experimental minus comparator) of –2.6, i.e. the change on active control is 2.6 attacks larger than on the experimental drug. Unfortunately, Patel et al. did not report the standard error for the change from baseline per group, and a corresponding confidence interval cannot be determined. Using a difference in absolute change approach increases the power to discover an effect. A recommended method of analysis would be to adjust for the baseline attack rates [5-7].

Analyzing the attack rates as normally distributed also may have reduced the power of the study. According to table 2, mean and standard deviation of the number of attacks are of comparable size, implying a non-symmetric distribution with a long right tail. Transforming the data to achieve a more symmetric distribution may have resulted in more power (smaller confidence intervals) when an ANOVA or t-test is applied. Assuming a distribution for count response data would have been more appropriate [8].

The sample size calculation of this trial was designed to detect a difference in hearing loss, the most clinically relevant side effect in ablative gentamicin drug treatment and defined as secondary outcome. Thus, sample size calculation is not related to the main study objective (difference in change in attack rates), leaving uncertainties in terms of statistical power to detect a minimum clinically relevant difference in the primary endpoint [9].

The primary efficacy outcome was assessed during face-to-face interviews at the end of two six-month reporting periods. In confirmatory trials on chronic conditions such as MD a blinded, reliable and valid assessment of patient's subjective experience of symptoms together with real-time data collection is crucial to minimize recall bias. Disease-specific, standardized, validated electronic or paper-based, event-oriented or daily symptom diaries that are designed to be continuously completed by patients at home offer a substantial benefit over retrospective data collection based on face-to-face interviews at clinic visits [10,11].

The logic of the paper is that of a non-inferiority trial (which seeks to show the investigational product to be comparable) and does not fit a superiority trial (prove a difference): methylprednisolone is not inferior to gentamicin in terms of reducing the attack rate, but may have the potential to be superior in other (safety) outcomes (in particular hearing loss). A non-inferiority trial needs a non-inferiority margin, i.e., which maximal difference between active control and experimental drug is considered clinically irrelevant [12,13]. For this inferiority trial a per-protocol analysis would be performed to calculate the 95% confidence interval (CI) for the treatment difference - driven by the wish to construct conservative treatment effects. If the 95% CI is above the non-inferiority margin the null hypothesis 'The experimental treatment is inferior to the active treatment' would be rejected. A non-inferiority study might have avoided the aporia from not knowing how to interpret the non-significant trial results in a clinically relevant way. Many of those who read the paper [1] will probably use the non-inferiority approach to interpret the results. Since non-inferiority also implies that the experimental treatment may be slightly worse than the comparator, it is possible that its effect comes close to that of a placebo treatment [14]. Therefore, non-inferiority trials should include a third placebo arm to assess whether the comparator's and experimental's effects are superior to the placebo effect. This may be an issue since the attack reduction in the experimental group is on average 2.6 less compared to the comparator.

REFERENCES:
[1] Patel M, Agarwal K, Arshad Q, et al. Intratympanic methylprednisolone versus gentamicin in patients with unilateral Meniere's disease: a randomised, double-blind, comparative effectiveness trial. Lancet 2016; 388(10061): 2753-62.

[2] Pullens B, van Benthem PP. Intratympanic gentamicin for Meniere's disease or syndrome. Cochrane Database Syst Rev 2011; 3: CD008234.

[3] Phillips JS, Westerberg B. Intratympanic steroids for Meniere's disease or syndrome. Cochrane Database Syst Rev 2011; 7: CD008514.

[4] Altman DG, Bland JM. Absence of evidence is not evidence of absence. BMJ 1995; 311(7003): 485.

[5] Vickers AJ, Altman DG. Statistics notes: Analysing controlled trials with baseline and follow up measurements. BMJ 2001; 323(7321): 1123-4.

[6] Van Breukelen GJP. ANCOVA versus change from baseline had more power in randomized studies and more bias in nonrandomized studies. J Clin Epidemiol 2006; 59(9): 920-5.

[7] Egbewale BE, Lewis M, Sim J. Bias, precision and statistical power of analysis of covariance in the analysis of randomized trials with baseline imbalance: a simulation study. BMC Med Res Methodol 2014; 14: 49.

[8] Adrion C, Fischer CS, Wagner J, Gurkov R, Mansmann U, Strupp M. Efficacy and safety of betahistine treatment in patients with Meniere's disease: primary results of a long term, multicentre, double blind, randomised, placebo controlled, dose defining trial (BEMED trial). BMJ 2016; 352: h6816.

[9] Clark T, Berger U, Mansmann U. Sample size determinations in original research protocols for randomised clinical trials submitted to UK research ethics committees: review. BMJ 2013; 346: f1135.

[10] Gater A, Coon CD, Nelsen LM, Girman C. Unique Challenges in Development, Psychometric Evaluation, and Interpretation of Daily and Event Diaries as Endpoints in Clinical Trials. Ther Innov Regul Sci 2015; 49(6): 813-21.

[11] Dawson J, Doll H, Fitzpatrick R, Jenkinson C, Carr AJ. The routine use of patient reported outcome measures in healthcare settings. BMJ 2010; 340: c186.

[12] Kaul S, Diamond GA. Good enough: a primer on the analysis and interpretation of noninferiority trials. Ann Intern Med 2006; 145(1): 62-9.

[13] FDA Guidance for Industry. Non-Inferiority Clinical Trials to Establish Effectiveness. Food and Drug Administration, U.S. Department of Health and Human Services, Center for Drug Evaluation and Research, November 2016. URL: https://www.fda.gov/downloads/Drugs/GuidanceComplianceRegulatoryInformat...

[14] Pigeot I, Schafer J, Rohmel J, Hauschke D. Assessing non-inferiority of a new treatment in a three-arm clinical trial including a placebo. Stat Med 2003; 22(6): 883-99.

Competing interests: No competing interests

12 March 2017
Christine Adrion
biostatistician
Ulrich Mansmann, director and chair of biostatistics and bioinformatics; Michael Strupp, professor of neurology, University Hospital Munich
Institute for Medical Informatics, Biometry, and Epidemiology, University of Munich
Marchioninistr. 15, 81377 Munich, Germany