Re: Risk of cardiovascular serious adverse events associated with varenicline use for tobacco cessation: systematic review and meta-analysis
Singh’s posting raises the issue of power associated with the estimated risk difference and the need for calculating the optimal information size (OIS, that is, the required meta-analysis sample size to detect some clinically important difference). Unfortunately, Singh did not provide any OIS, and so, one is left pondering whether the meta-analysis by Prochaska and Hilton is sufficiently powered. In her reply, Prochaska observed that the upper limit of the 95% confidence interval for the pooled risk difference is 0.63%, and subsequently goes on to argue that their meta-analysis therefore ‘had sufficient powered to detect a difference as small as two-thirds of one percent’. This argument reflects a lacking understanding confidence intervals and significance testing.
The correct approach would be to estimate an OIS geared to demonstrate non-inferiority (of varenicline) at the conventional level of statistical significance (ie, 5%) with some desired power (eg, 90%).
To obtain the OIS we must first obtain realistic a priori estimates of the control group risk and the intervention (varenicline) group risk. The crude risk in the control group is 18/3801, or 0.47%. Excluding the zero-zero-event trials the crude risk is 18/3181, or 0.56%. We could thus plausibly assume that the control risk is about 0.5%. Prochaska and Hilton estimated a risk difference of 0.27%, so we can use this estimate as our a priori assumed risk difference. In terms of demonstrating non-inferiority, two non-inferiority bounds come to mind.
First Prochaska and Hilton emphasize that the 95% CI precludes 0.63%. However, the OIS for demonstrating that the risk difference is no larger than 0.63% with 90% power is 20462 patients (and 15285 with 80% power), which is more than twice as much as the current meta-analysis sample size.
Second, we can seek to test non-inferiority by first establishing the OIS required to detect some agreed upon minimally clinically important difference (MCID), and subsequently, if the OIS has not been reached, apply trial sequential futility boundaries.[2,3] Since no MCID has been established for cardiovascular events in smoking cessation, we may draw upon related fields. For the evaluation of cardiovascular risks in diabetes mellitus, FDA has recommended that non-inferiority could be inferred if the 95% CI of the relative risk excluded 1.8. Assuming a 0.5% control risk, the corresponding intervention group risk would be 0.9%, and we would therefore require that the 95%CI for the risk difference preclude 0.4%. The OIS for detecting a difference of 0.4% (superiority test) with 90% power is 18260 (and 13640 with 80% power). Since the OIS has not been reached, we can apply trial sequential futility boundaries for the cumulative Z-test to evaluate whether the evidence is in fact conclusive (see figure). Conclusive non-inferiority (or futility) is established when the cumulative z-curve enters the futility region. This has not happened with the current evidence – neither based on the OIS with 90% power or OIS with 80% power.
Considering the above information size and non-inferiority evaluations it stands to reason that the current meta-analytic evidence does not suffice to produce a conclusive answer, but that it most likely will suffice to inform a non-inferiority test once data from the 8000 patient CATS study becomes available.
1. Guyatt GH, Oxman AD, Kunz R, Brozek J, Alonso-Coello P, Rind D, et al. GRADE guidelines 6. Rating the quality of evidence – imprecision. J Clin Epi 2011; 64: 1283093.
2. van der Tweel I, Bollen C. Sequential meta-analysis: an efficient decision-making tool. Clin Trials 2010; 7:136-46.
3. Thorlund K, Engstrøm J, Wetterslev J, Brok J, Imberger G, Gluud C. User manual for trial sequential analysis (TSA). Copenhagen Trial Unit, Centre for Clinical Intervention Research, Copenhagen, Denmark. 2011. p. 1-115. Available from www.ctu.dk/tsa
4. Pogue J, Yusuf S. Overcoming the limitations of current meta-analysis of randomized controlled trials. Lancet 1998; 9095:47-52.
Competing interests: No competing interests