Risk of cardiovascular serious adverse events associated with varenicline use for tobacco cessation: systematic review and meta-analysis
BMJ 2012; 344 doi: https://doi.org/10.1136/bmj.e2856 (Published 04 May 2012) Cite this as: BMJ 2012;344:e2856
All rapid responses
Rapid responses are electronic comments to the editor. They enable our users to debate issues raised in articles published on bmj.com. A rapid response is first posted online. If you need the URL (web address) of an individual response, simply click on the response headline and copy the URL from the browser window. A proportion of responses will, after editing, be published online and in the print journal as letters, which are indexed in PubMed. Rapid responses are not indexed in PubMed and they are not journal articles. The BMJ reserves the right to remove responses which are being wilfully misrepresented as published articles or when it is brought to our attention that a response spreads misinformation.
From March 2022, the word limit for rapid responses will be 600 words not including references and author details. We will no longer post responses that exceed this limit.
The word limit for letters selected from posted responses remains 300 words.
Shinji Teramoto, MD, PhD, Department of Pulmonary Medicine, Hitachinaka Medical Education and Research Center, The University of Tsukuba
Prochaska JJ and Hilton JF have wisely answered the important clinical question whether varenicline causes serious cardiovascular complication in subjects attempting smoking cessation [1]. Is smoking cessation using varenicline with high cardiovascular risk superior to continuous smoking with less cardiovascular risk? However, Singh and colleagues have reported an increased risk of serious adverse cardiovascular events associated with the use of varenicline among tobacco users [2]. Although smoking cessation is always good for any health condition, the relatively expensive medicine varenicline used cessation program may not hold the merit for the tobacco users in terms of medical costs and future cardiovascular events [3]. Since their “fair” meta-analysis denied that varenicline use increased cardiovascular serious adverse events associated in the smoking cessation trial patients, we are very confident to continue smoking cessation program for many smokers using varenicline.
Because smoking is not a habit, but a disease, we should continue to help smoking cessation using all the useful techniques including varenicline for the patients with nicotine addiction as a therapy.
Preventive approach for the development of chronic obstructive pulmonary disease (COPD), which is 4-5th leading cause of death in worldwide, is urgently necessary. Smoking cessation is the most confident preventive strategy for the development of COPD [4].
Importantly, passive smoking is also a risk for the development of COPD in non-smoker families. Therefore, effective smoking cessation program implementation is rigorously necessary for smokers and smoker’s families. All the physician should do endeavor to introduce quitting smoking program irrespective to application of varenicline for young and adults’ smokers worldwide.
However, my experiences suggest that varenicline is the most helpful agent for quitting smoking in the smokers. The initial quitting ratio is greater than 80% in my experience. The absolute continuation of smoking cessation more than 6 months remains unsolved. To reduce the burden of smoking on health worldwide, further progress in smoking cessation programs is necessary.
The current paper clearly contributes to improvement of health status in many smokers’ circumstances. We are confident that an ounce of prevention is worth a pound of cure.
Reference List
[1] Prochaska JJ, Hilton JF. Risk of cardiovascular serious adverse events associated with varenicline use for tobacco cessation: systematic review and meta-analysis. BMJ 2012 May 4; 344:e2856.
[2] Singh S, Loke YK, Spangler JG, Furberg CD. Risk of serious adverse cardiovascular events associated with varenicline: a systematic review and meta-analysis. CMAJ 2011;183:1359-66.
[3] West R, Zatonski W, M CedzynskaM, et al. Placebo-controlled trial of cytisine for smoking cessation. New Engl J Med 365:193-1200, 011
[4] Teramoto S, Yamamoto H, Yamaguchi Y, Ouchi Y. Global burden of COPD in Japan and Asia. Lancet 362:1764-1765, 2003
Competing interests: None declared
Competing interests: No competing interests
Singh’s posting raises the issue of power associated with the estimated risk difference and the need for calculating the optimal information size (OIS, that is, the required meta-analysis sample size to detect some clinically important difference). Unfortunately, Singh did not provide any OIS, and so, one is left pondering whether the meta-analysis by Prochaska and Hilton is sufficiently powered. In her reply, Prochaska observed that the upper limit of the 95% confidence interval for the pooled risk difference is 0.63%, and subsequently goes on to argue that their meta-analysis therefore ‘had sufficient powered to detect a difference as small as two-thirds of one percent’. This argument reflects a lacking understanding confidence intervals and significance testing.
The correct approach would be to estimate an OIS geared to demonstrate non-inferiority (of varenicline) at the conventional level of statistical significance (ie, 5%) with some desired power (eg, 90%).[1]
To obtain the OIS we must first obtain realistic a priori estimates of the control group risk and the intervention (varenicline) group risk. The crude risk in the control group is 18/3801, or 0.47%. Excluding the zero-zero-event trials the crude risk is 18/3181, or 0.56%. We could thus plausibly assume that the control risk is about 0.5%. Prochaska and Hilton estimated a risk difference of 0.27%, so we can use this estimate as our a priori assumed risk difference. In terms of demonstrating non-inferiority, two non-inferiority bounds come to mind.
First Prochaska and Hilton emphasize that the 95% CI precludes 0.63%. However, the OIS for demonstrating that the risk difference is no larger than 0.63% with 90% power is 20462 patients (and 15285 with 80% power), which is more than twice as much as the current meta-analysis sample size.
Second, we can seek to test non-inferiority by first establishing the OIS required to detect some agreed upon minimally clinically important difference (MCID), and subsequently, if the OIS has not been reached, apply trial sequential futility boundaries.[2,3] Since no MCID has been established for cardiovascular events in smoking cessation, we may draw upon related fields.[4] For the evaluation of cardiovascular risks in diabetes mellitus, FDA has recommended that non-inferiority could be inferred if the 95% CI of the relative risk excluded 1.8. Assuming a 0.5% control risk, the corresponding intervention group risk would be 0.9%, and we would therefore require that the 95%CI for the risk difference preclude 0.4%. The OIS for detecting a difference of 0.4% (superiority test) with 90% power is 18260 (and 13640 with 80% power). Since the OIS has not been reached, we can apply trial sequential futility boundaries for the cumulative Z-test to evaluate whether the evidence is in fact conclusive (see figure). Conclusive non-inferiority (or futility) is established when the cumulative z-curve enters the futility region. This has not happened with the current evidence – neither based on the OIS with 90% power or OIS with 80% power.
Considering the above information size and non-inferiority evaluations it stands to reason that the current meta-analytic evidence does not suffice to produce a conclusive answer, but that it most likely will suffice to inform a non-inferiority test once data from the 8000 patient CATS study becomes available.
1. Guyatt GH, Oxman AD, Kunz R, Brozek J, Alonso-Coello P, Rind D, et al. GRADE guidelines 6. Rating the quality of evidence – imprecision. J Clin Epi 2011; 64: 1283093.
2. van der Tweel I, Bollen C. Sequential meta-analysis: an efficient decision-making tool. Clin Trials 2010; 7:136-46.
3. Thorlund K, Engstrøm J, Wetterslev J, Brok J, Imberger G, Gluud C. User manual for trial sequential analysis (TSA). Copenhagen Trial Unit, Centre for Clinical Intervention Research, Copenhagen, Denmark. 2011. p. 1-115. Available from www.ctu.dk/tsa
4. Pogue J, Yusuf S. Overcoming the limitations of current meta-analysis of randomized controlled trials. Lancet 1998; 9095:47-52.
Competing interests: No competing interests
Singh’s posting to BMJ suggesting our meta-analysis “has limitations in data, analysis and interpretation” is based on a number of misconceptions that warrant response.
First, citing the FDA label for Chantix, Singh criticized our analysis for excluding “cardiovascular adverse events from the varenicline arms that have been reported.” In fact, our coding of events directly and explicitly mirrored the coding strategy used by Singh et al. [1]. Both meta-analyses focused on cardiovascular serious adverse events. Hence, the myocardial infarction and stroke events reported in the FDA label were included as were the hospitalizations for angina, which occurred in both the varenicline (1 case) and placebo (2 cases) groups. The other 17 cases of stable angina, which occurred in both arms, were not serious (i.e., did not result in hospitalization) and were not included in either our analysis or that of Singh.
Second, Singh asserts that “the higher drop-out rate in the placebo group is irrelevant” and that his approach of counting events occurring at anytime during the trials, in many cases up to 52 weeks, “adhered to the intention-to-treat (ITT) analysis.” Yet, quoting an article from BMJ, “full application of intention to treat can only be performed where there is complete outcome data for all randomised subjects” [2]. In this case, not only was there incomplete data for randomized subjects, but the degree of incomplete data differed by condition, and in 13 of the 14 trials included by Singh, was biased in favor of detecting more events in the varenicline group. For participants who were lost to follow-up, Singh did not impute their data. For example, in the tobacco control field, missing subjects may be coded as treatment failures (i.e., coded as smokers), as a form of ITT analysis. Singh coded only the events as observed; we did the same in our analysis. The higher dropout rate in the placebo group is relevant. Singh’s analysis does not conform to ITT. Also relevant are the 8 trials with nearly 1600 participants that were excluded from Singh’s analysis because they did not have a single cardiovascular serious adverse event.
A difference in our methods was the time period of observation. Our observation period was the time participants were on drug or within 30 days of drug discontinuation. We chose this period because it is a biologically relevant time-frame given the half-life of varenicline, which leaves the body within 7 days, and it reduces the problem of differential dropout seen with Singh’s methods and the complexity of differentiating treatment versus disease (i.e., chronic tobacco exposure) effects on cardiovascular risk.
Third, Singh criticized our use of the risk difference for being “statistically underpowered at low event rates, and bias their estimates towards the null.” In fact, we presented the findings from three relative measures in addition to the risk difference and all four estimates were nonsignificant. We calculated and presented four different summary estimates to allow for transparent and direct comparison. Singh quotes the Cochrane Handbook (http://www.cochrane-handbook.org/), which actually does not recommend the Peto OR as the default approach for meta-analysis and specifically discourages its use when the studies have unequal allocation to experimental and control groups (section 9.4.4.2), as was the case for seven of the trials examining varenicline for tobacco cessation.
As an example, the trial by Bolliger observed 1 event in the treatment group with nearly 400 subjects versus no event in nearly 200 subjects in the placebo group. The Peto OR suggested a 4.5 fold greater difference. This level of inflation with the Peto OR was found in 8 of the 22 trials we reviewed and was not mirrored in the other relative summary estimates (Table 2). Researchers need to be mindful of their methods and question their findings even when they are in the direction that they favor. We are now working on a separate statistically-focused manuscript that will provide in greater detail the limitations of the Peto OR under conditions such as those seen with the analysis of cardiovascular serious adverse events and varenicline use.
Fourth, regarding power, in reference to our work, Singh states, “They conflate the lack of statistical significance in an underpowered meta-analysis as clinically insignificant.” Our meta-analysis of 22 trials with over 9200 individuals found that the 95% confidence interval on the risk difference excluded an increase larger than 0.63%. That is, our analysis had sufficient power to detect a difference as small as two-thirds of one percent. Inconsistently, Singh later goes on to emphasize the need for a clinical trial with 8000 patients, which is less than the 9200 included in our meta-analysis. Notably, our analysis demonstrated that the summary estimate has not changed over time with the addition of more trials (Figure 3).
Meta-analytic methods are specifically useful for testing hypotheses that would be underpowered in individual trials. By combining studies, a meta-analysis increases the sample size and by drawing from independent samples, increases generalizability of the findings. It also should be noted that the Singh analysis included fewer subjects (i.e., 8216 subjects) and concerns about power were not raised.
Singh cites a post-marketing study and states it found “an increase in cardiovascular events with varenicline” [3]. An increase over what? The study, which did not have a comparison condition, drew no conclusion regarding elevated risk. Similarly, Singh and Thomas Moore, an investigative journalist and expert witness in litigation against Chantix, published a post-marketing varenicline analysis that has been criticized for having an unknown denominator with no data on anticipated base rates [4].
In closing, Singh writes, “Despite the removal of cardiovascular events on varenicline, and a statistical approach that has limited powered to detect an effect there is an excess risk of cardiovascular events with varenicline in all five measures reported in their study.” This statement is wrong in multiple ways. As noted above, our coding strategy mirrored that used by Singh and was applied consistently for both the varenicline and placebo groups. The excess risk to which Singh refers is neither clinically nor statistically significant and again, is most likely due to the differential attrition by group.
We agree with Singh that clinicians need to weigh the risks and benefits of medications that they recommend to their patients. To aid clinicians, however, the empirical evidence needs to be transparent and attentive to bias. Our re-analysis was conducted to address problems with methods that resulted in misleading conclusions. It is unfortunate that Singh seems not to appreciate the problems with the statistical methods he chose.
1. Singh S, Loke YK, Spangler JG, Furberg CD. Risk of serious adverse cardiovascular events associated with varenicline: a systematic review and meta-analysis. CMAJ 2011; 183(12):1359-66.
2. Hollis S, Campbell F. What is meant by intention to treat analysis? Survey of published randomised controlled trials. BMJ 1999; 319(7211): 670–674.
3. Harrison-Woolrych M, Maggo S, Tan M, Savage R, Ashton J. Cardiovascular events in patients taking varenicline: a case series from intensive postmarketing surveillance in New Zealand. Drug Saf 2012; 35(1): 33-43.
4. Moore TJ, Furberg CD, Glenmullen J, Maltsberger JT, Singh S. Suicidal behavior and depression in smoking cessation treatments. PLoS One. 2011; 6(11):e27016.
Competing interests: Detailed disclosures included within our BMJ publication including grant funding from NIDA, NIMH, TRDRP, FAMRI, and a single investigator initiated research award from Pfizer, Inc. (#WS981308 registered online with www.clinicaltrials.gov, title is Varenicline Inpatient Study). The funding from Pfizer did not support the work of this meta-analysis.
In their meta-analytical investigation of rare adverse events associated with varenicline use for tobacco cessation, Prochaska and Hilton compared four different methods of meta-analysis (the Peto odds ratio, and the Mantel-Haenszel odds ratio, risk ratio and risk difference methods) and noted that they yielded different estimates of effect [1]. Such differences are not unexpected since the methods are all based on large sample asymptotic statistical theory, the assumptions of which are challenged when events are rare. The three Mantel-Haenszel methods also involve the use of an arbitrary numerical correction to avoid computational errors that occur when attempting to divide by zero.
Prochaska and Hilton conclude “for rare outcomes, summary estimates based on absolute effects are recommended and estimates based on the Peto odds ratio should be avoided”. They provide five arguments to support this statement, none of which is convincing, some of which are misleading, and some of which are seriously flawed. To demonstrate that a result is biased, it is necessary to know what the correct result should be. This cannot be achieved in a case-study such as this, which simply compares four different analytical methods. The authors’ conclusion is not justified and is potentially dangerous.
First, Prochaska and Hilton state “treatment effects based on relative risks always are as or less extreme than those based on odds ratios”. It is a mistake to compare the ‘extremeness’ of two different metrics in this way. Furthermore, when the outcomes are rare as they are in this example, odds ratios are in fact very close approximations to relative risks.
Second they argue “relative statistics cannot be calculated for trials with zero events and therefore can bias summaries against the null hypothesis of no effect”. This reflects a common intuitive feeling that it is wrong to exclude any studies from a meta-analysis. However, if no events occur at all in a trial, the trial in itself conveys no information about the relative odds or risks of events between the two groups. A meta-analysis may be viewed as a weighted average of trial results, with weights reflecting the amount of information each study contains about the summary statistic. Allocating a trial with no information a zero weight is entirely appropriate and does not introduce bias.
Third, they argue that relative statistics hide the impact of the effect, whereas risk differences most clearly convey the effect, using this to justify their meta-analysis of risk differences. We agree that absolute effect measures convey more useful information. However, the use of relative measures in a meta-analysis is not a barrier to re-expressing the treatment effect in absolute terms, e.g. as a number needed to treat to benefit or harm, or a risk difference.
Prochaska and Hilton also cite Vandermeer and colleagues’ analysis of 1613 meta-analyses [2], claiming that they showed the Peto odds ratio to be particularly biased. Vandermeer in fact compared results of asymptotic methods with meta-analytical techniques based on ‘permutation’ or ‘exact’ methods. As with the current study, Vandermeer did not know what the true result was, so was unable to indicate that one method was biased, only that methods gave different results.
Finally they imply that the Peto odds ratio method must be biased because it produces the most extreme values of odds ratios for the individual studies. This is a misunderstanding of the way in which the method should be applied – it is designed to compute an overall meta-analytical statistic and not for computation of estimates of odds ratios for individual studies. The strength of the Peto method is that it aggregates within-trial comparisons across trials in a way that avoids the need for the arbitrary addition of 0.5 events to some treatment groups in which no events were observed. It is this arbitrary addition that causes both the Mantel-Haenszel risk ratio and odds ratio to be similar, and the Mantel-Haenszel odds ratio to be smaller than the Peto odds ratio.
The rigorous approach to understanding bias in statistical methods is to undertake simulation studies, where the investigators create data with known true treatment effects, apply the alternative statistical methods and examine how the estimates compare with the truth. Such studies can investigate bias in the treatment effect, the coverage of confidence intervals, the correctness of P-values, and the power that different methods have to detect differences. One of us undertook and reported such a study of statistical methods for meta-analysis of rare events, and found evidence that the methods to be recommended in practice exactly are the opposite to the recommendation of Prochaska and Hilton, with the Peto method being the least biased and most powerful in situations where event rates were around 1% [3]. Notably, whilst the Mantel-Haenszel risk difference method produced relatively unbiased estimates of treatment effects, the method was shown to be seriously limited in its ability to detect real differences, as its confidence intervals are wide, giving poor statistical power. It is thus unsuitable for meta-analysis of rare events, as it reduces the chance of real increases in rare adverse events being detected, with the subsequent possibility that patients continue to be exposed to harmful effects of some medications.
[1] Prochaska J, Hilton J. Risk of serious adverse cardiovascular events associated with varenicline use for tobacco cessation: systematic review and meta-analysis. BMJ. 2012;344(e2856)
[2] Vandermeer B, Bialy L, Hooton N, Hartling L, Klassen TP, Johnston BC, et al. Meta-analyses of safety data: a comparison of exact versus asymptotic methods. Stat Methods Med Res 2009;18:421-32.
[3] Bradburn MJ, Deeks JJ, Berlin JA, Russell Localio A. Much ado about nothing: a comparison of the performance of meta-analytical methods with rare events. Stat Med 2007;26:53-77.
Competing interests: No competing interests
The meta-analysis by Proschaska and Hilton [1] found no significant association between varenicline use and serious cardiovascular events, reaching opposite conclusions from previous meta-analysis of Singh, et al [2]. Methodological differences in two different meta-analyses are responsible for these opposite conclusions.
As a rule of thumb, evidence of harm from a clinical intervention is more difficult to establish than evidence of benefit. While we may not know the truthful answer to this question until we have the results of well designed randomised controlled trials which are adequately powered to detect serious cardiovascular events as primary end points, clinicians must make therapeutic decisions before the results of such trials are readily available.
There are additional safety concerns on neuropsychiatric adverse effects. Serious psychiatric adverse effects including completed suicides, hostility/aggression, depression and psychosis, were reportedly associated with varenicline use in postmarketing surveillance data [3]. The benefit of varenicline in cessation of smoking is only modest with the approximate frequency of cessation rate of 25% [4]. Clinicians should not only weigh these risks against potential benefits, but also be mindful of the ethical principle of non-maleficence and the medical aphorism, Primum Non Nocere, when making prescribing decisions.
1. Prochaska J, Hilton J. Risk of serious adverse cardiovascular events associated with varenicline use for tobacco cessation: systematic review and meta-analysis. BMJ. 2012;344(e2856).
2. Singh S, Loke YK, Spangler JG, Furberg CD. Risk of serious adverse cardiovascular events associated with varenicline: a systematic review and meta-analysis. CMAJ2011;183:1359-66.
3. Varenicline (CHANTIX) Risks Underestimated . ISMP QuarterWatch 2011 May 19;2010(Q3):14
4. Cahill K, Stead LF, Lancaster T. Nicotine receptor partial agonists for smoking cessation. Cochrane Database Syst Rev. 2011;(2):CD006103.
Competing interests: No competing interests
It is interesting to note that the recent systematic review and meta-analysis by Proschaska and Hilton [1] found no significant increase in serious cardiovascular events associated with varenicline use. This conclusion is not consistent with the findings of a previous meta-analysis [2] and Sonal Singh has identified some reasons for this in his rapid response. A further question is which of the 22 trials included in the Prochaska review were company-funded studies? The largest study included [3] in this review was sponsored by Pfizer Inc (sponsor of Champix®) and pharmaceutical industry involvement in clinical trials may affect the outcomes of studies. It would be helpful if the authors of the recent meta-analysis could publish information about sponsorship of each study included in their review and discuss whether this may have had any effect on the results.
Similar to other safety issues with varenicline – for example psychiatric effects - the results from clinical trials appear to be at odds with post-marketing data. An early pooled analysis of trials (sponsored by Pfizer) concluded that “psychiatric events are uncommon and do not appear to be caused by varenicline per se” [4] but our post-marketing observational cohort study in New Zealand has shown psychiatric events are common and many are causally related to varenicline [5]. Regarding cardiovascular events, we recently published a case series of 172 events identified during this Intensive Medicines Monitoring Programme (IMMP) study [6]. There were 48 reports of myocardial ischaemia, with two key cases suggesting these events may have been induced by coronary artery spasm. There were another 50 reports of hypotensive events, of which two patients experienced chest pain/tightness in association with documented hypotension. Whilst there were confounding factors in some patients, these clinical observations from ‘real-life’ use & intensive monitoring of varenicline are important. We suggest there may be a plausible mechanism - of dysregulation of blood pressure leading to vasospasm – to explain some cardiovascular events in patients taking varenicline and further studies are needed to investigate this.
The Institute of Medicine recently recommended that the FDA should include data from observational studies in the evaluation of drug harm [BMJ 2012; 344:e3104]. We support this recommendation because clinical trials (and meta-analyses of these results) can only provide some of the data required for risk-benefit evaluation of medicines. The population of patients included in clinical trials may be very different to those using the same medicine in real-life post-marketing use. Ideally, data from all types of studies should be considered in the interest of patient safety.
References
1. Prochaska J, Hilton J. Risk of serious adverse cardiovascular events associated with varenicline use for tobacco cessation: systematic review and meta-analysis. BMJ. 2012;344(e2856).
2. Singh S, Loke YK, Spangler JG, Furberg CD. Risk of serious adverse cardiovascular events associated with varenicline: a systematic review and meta-analysis. CMAJ. 2011 Jul 4.
3. Rigotti NA, Pipe AL, Benowitz NL, Arteaga C, Garza D, Tonstad S. Efficacy and safety of varenicline for smoking cessation in patients with cardiovascular disease: a randomized trial. Circulation. Jan 19;121(2):221-9.
4. Tonstad S, Davies S, Flammer M, Russ C, Hughes J. Psychiatric adverse events in randomized, double-blind, placebo-controlled clinical trials of varenicline: a pooled analysis. Drug Saf. Apr 1;33(4):289-301.
5. Harrison-Woolrych M, Ashton J. Psychiatric adverse events associated with varenicline: an intensive postmarketing prospective cohort study in new zealand. Drug Saf. Sep 1;34(9):763-72.
6. Harrison-Woolrych M, Maggo S, Tan M, Savage R, Ashton J. Cardiovascular events in patients taking varenicline: A case series from intensive postmarketing surveillance in New Zealand. Drug Safety. 2011;35(1):33-43.
Competing interests: No competing interests
This meta-analysis by Prochaska et al regarding the cardiovascular risks of varenicline has limitations in data, analysis and interpretation which raises doubts about the veracity of their conclusions.
Firstly, the authors excluded a number of cardiovascular adverse events from the varenicline arms that have been reported.1For example, in the largest clinical trial, the Rigotti study the authors recorded 10 events in each treatment arm. In contrast, the updated FDA label, which contains warnings about an increase in cardiovascular risk associated with varenicline in patients with cardiovascular disease, notes several more treatment-emergent cardiovascular events during 30 days in varenicline than placebo “These include treatment emergent events (ontreatment or 30 days after treatment of angina pectoris (13 patients in varenicline arm vs 7 in placebo arm), and the serious cardiovascular events of non-fatal MI (4 vs 1) and nonfatal stroke (2 vs 0).” 2
Secondly, the authors prefer to analyse data by treatment level which would allow exclusion of events occurring in randomized patients. In contrast, we adhered to the intention-to-treat (ITT) analysis in accordance with FDA regulations and established and generally accepted scientific principles. In addition, by adhering to the ITT principles, noncompliance becomes a nonissue. Thus, the higher drop-out rate in the placebo group is irrelevant.
Thirdly, the authors recommend risk difference as the most appropriate model. However their approach contradicts advice from the Cochrane Handbook, “The Peto odds ratio method was ‘found to be the least biased and most powerful method’ and that risk difference analytical methods ‘tended to show conservative confidence interval coverage and low statistical power when risks of events were low”.3 As a result most regulatory meta-analysis of safety risks, including our meta-analysis are conducted on the relative scale. The authors’ models are statistically underpowered at low event rates, and bias their estimates towards the null.4
Fourthly, the authors provide no information on the optimal information size or the power of their meta-analysis. They conflate the lack of statistical significance in an underpowered meta-analysis as clinically insignificant. Despite the removal of cardiovascular events on varenicline, and a statistical approach that has limited powered to detect an effect there is an excess risk of cardiovascular events with varenicline in all five measures reported in their study. Another independent post-marketing study also reported an increase in cardiovascular events with varenicline. 5
Since none of the trials evaluated cardiovascular events as a primary outcome or were powered to detect individual differences in cardiovascular outcomes between varenicline and placebo, adequately powered randomized controlled trials are needed. The Institute of Medicine Report on the ethics of post-marketing safety studies has called for such adequately designed and powered post-marketing safety studies to address safety concerns be completed in a timely manner.6 A post-marketing Study to Evaluate Cardiac Assessment following different treatments of smoking cessation medications in subjects with and without psychiatric disorder (CATS) comparing varenicline, placebo, bupropion and nicotine replacement therapy of approximately 8000 patients with major adverse cardiovascular events as a primary outcome has been requested by the FDA.7 After 24 weeks of treatment it will collect data on cardiovascular outcomes for an additional 28 weeks of non-treatment, for a total of 52 weeks.8 Unfortunately, the results will only be available in 2017, just in time for the patent expiry of varenicline.
It is paradoxical that an efficacious drug used for smoking cessation increases cardiovascular risk in randomized clinical trials, when it should have presumably provided such a benefit.
However, until the results of such outcome trials are available, clinicians need to consider the overall risks of varenicline. i. e., the serious cardiovascular risk and other serious risks such as suicide, depression and violence noted in the drug label. 2 They need to balance these serious risks against its efficacy on abstinence, compared to placebo. 1
As a result of these safety concerns, the United States Veteran Administration, 9 does not recommend varenicline as a first line therapy for smoking cessation.
Sonal Singh, MD, MPH
Johns Hopkins University
Yoon K Loke, MD
University of East Anglia, UK
Financial conflicts of Interest: None
Reference
1. Singh S, Loke YK, Spangler JG, Furberg CD. Risk of serious adverse cardiovascular events associated with varenicline: a systematic review and meta-analysis. CMAJ 2011; 183:1359-66.
2. http://www.accessdata.fda.gov/drugsatfda_docs/label/2012/021928s028lbl.pdf
Accessed May 4, 2012
3. Higgins JPT, Deeks JJ, Altman DG (editors). Chapter 16: Special topics in statistics. In: Higgins JPT, Green S (editors), Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 (updated March 2011). The Cochrane Collaboration, 2011. Available from www.cochrane-handbook.org. Accessed May 4, 2012.
4. Bradburn MJ, Deeks JJ, Berlin JA, Russell Localio A. Much ado about nothing: a comparison of the performance of meta-analytical methods with rare events. Stat Med 2007; 26:53-77.
5. Harrison-Woolrych M, Maggo S, Tan M, Savage R, Ashton J. Cardiovascular events in patients taking varenicline: a case series from intensive postmarketing surveillance in New Zealand.Drug Saf 2012;35:33-43.
6 Institute of Medicine. (2012)Ethical and Scientific Issues in Studying the Safety of approved Drugs. Washington, DC : National Academies Press
7 http://clinicaltrials.gov/ct2/show/NCT01574703 Accessed May 2012
8. http://clinicaltrials.gov/ct2/show/NCT01456936 Accessed May 4, 2012
9. United States Department of Veterans Affairs Pharmacy Benefits Management Services (2011) Clinical Guidance: Varenicline Criteria for Prescribing. U.S. Department of Veterans Affairs Pharmacy Benefits Management Services. Available: http://www.pbm.va.gov/CriteriaForUse.aspx . Accessed May 4, 2012
Competing interests: No competing interests
Re: Risk of cardiovascular serious adverse events associated with varenicline use for tobacco cessation: systematic review and meta-analysis
Our BMJ 2012 meta-analysis of cardiovascular-related serious adverse events (CVD SAEs) associated to varenicline use has had had over 12,000 online full text viewings and drew attention in its critique of an earlier, widely publicized meta-analysis by Singh et al.(1) Two distinctions between our meta-analysis and the earlier work were the outcome definition and the choice of summary statistic. We restricted outcomes to treatment-emergent CVD SAEs in both active and placebo arms, whereas Singh et al.(1) allowed differential outcome follow-up by arm, which extended beyond the treatment-emergent period especially in the varenicline arm. In addition, Singh et al. used the Peto odds ratio (OR), while we summarised the study effects with the risk difference (RD), an estimate of absolute effect that usefully conveys the clinical effect of a treatment. The risk difference is particularly appropriate in examining rare event data because it accommodates trials with zero events.(2-4) Thus, our meta-analysis of 22 trials included 8 zero-event trials that were excluded from Singh’s analysis.
There is limited literature informing selection of an optimal summary statistic for fixed-effects meta-analyses in the setting of rare events. Rucker et al.(4) recommended that “if the observation times, and thus the numbers of events, vary considerably between studies, relative measures such as the RR and the OR are preferable,” and they further proposed that “the RD should be presented alongside the OR to put it in perspective, and thus allow ‘zero’ trials to contribute to the overall evidence and its interpretation.”(4) In our analysis, there was no advantage to using a relative measure because the observation periods and the number of events were comparable for all but 2 of the 22 trials. Consistent with the guidance of Rucker et al.(4), we reported Peto and Mantel-Haenszel ORs and the relative risk (RR) in addition to the risk difference.
The findings in this review do not imply that our methods should be applied to all reviews. We encourage further investigations of the best choice between and among relative and absolute measures in a wide range of event-rate and allocation-ratio settings.
References:
(1) Singh S, Loke YK, Spangler JG, Furberg CD. Risk of serious adverse cardiovascular events associated with varenicline: a systematic review and meta-analysis. CMAJ 2011;183:1359-66.
(2) Vandermeer B, Bialy L, Hooton N, Hartling L, Klassen TP, Johnston BC, et al. Meta-analyses of safety data: a comparison of exact versus asymptotic methods. Stat Methods Med Res 2009;18:421-32.
(3) Bradburn MJ, Deeks JJ, Berlin JA, Russell Localio A. Much ado about nothing: a comparison of the performance of meta-analytical methods with rare events. Stat Med 2007;26:53-77.
(4) Rucker G, Schwarzer G, Carpenter J, Olkin I. Why add anything to nothing? The arcsine difference as a measure of treatment effect in meta-analysis with zero cells. Statist Med 2009; 28:721–738.
Competing interests: Detailed disclosures included within our BMJ publication including grant funding from NIDA, NIMH, TRDRP, FAMRI, and a single investigator initiated research award from Pfizer, Inc. (#WS981308 registered online with www.clinicaltrials.gov, title is Varenicline Inpatient Study). The funding from Pfizer did not support the work of this meta-analysis.