Re: Risk of cardiovascular serious adverse events associated with varenicline use for tobacco cessation: systematic review and meta-analysis
Singh’s posting to BMJ suggesting our meta-analysis “has limitations in data, analysis and interpretation” is based on a number of misconceptions that warrant response.
First, citing the FDA label for Chantix, Singh criticized our analysis for excluding “cardiovascular adverse events from the varenicline arms that have been reported.” In fact, our coding of events directly and explicitly mirrored the coding strategy used by Singh et al. [1]. Both meta-analyses focused on cardiovascular serious adverse events. Hence, the myocardial infarction and stroke events reported in the FDA label were included as were the hospitalizations for angina, which occurred in both the varenicline (1 case) and placebo (2 cases) groups. The other 17 cases of stable angina, which occurred in both arms, were not serious (i.e., did not result in hospitalization) and were not included in either our analysis or that of Singh.
Second, Singh asserts that “the higher drop-out rate in the placebo group is irrelevant” and that his approach of counting events occurring at anytime during the trials, in many cases up to 52 weeks, “adhered to the intention-to-treat (ITT) analysis.” Yet, quoting an article from BMJ, “full application of intention to treat can only be performed where there is complete outcome data for all randomised subjects” [2]. In this case, not only was there incomplete data for randomized subjects, but the degree of incomplete data differed by condition, and in 13 of the 14 trials included by Singh, was biased in favor of detecting more events in the varenicline group. For participants who were lost to follow-up, Singh did not impute their data. For example, in the tobacco control field, missing subjects may be coded as treatment failures (i.e., coded as smokers), as a form of ITT analysis. Singh coded only the events as observed; we did the same in our analysis. The higher dropout rate in the placebo group is relevant. Singh’s analysis does not conform to ITT. Also relevant are the 8 trials with nearly 1600 participants that were excluded from Singh’s analysis because they did not have a single cardiovascular serious adverse event.
A difference in our methods was the time period of observation. Our observation period was the time participants were on drug or within 30 days of drug discontinuation. We chose this period because it is a biologically relevant time-frame given the half-life of varenicline, which leaves the body within 7 days, and it reduces the problem of differential dropout seen with Singh’s methods and the complexity of differentiating treatment versus disease (i.e., chronic tobacco exposure) effects on cardiovascular risk.
Third, Singh criticized our use of the risk difference for being “statistically underpowered at low event rates, and bias their estimates towards the null.” In fact, we presented the findings from three relative measures in addition to the risk difference and all four estimates were nonsignificant. We calculated and presented four different summary estimates to allow for transparent and direct comparison. Singh quotes the Cochrane Handbook (http://www.cochrane-handbook.org/), which actually does not recommend the Peto OR as the default approach for meta-analysis and specifically discourages its use when the studies have unequal allocation to experimental and control groups (section 9.4.4.2), as was the case for seven of the trials examining varenicline for tobacco cessation.
As an example, the trial by Bolliger observed 1 event in the treatment group with nearly 400 subjects versus no event in nearly 200 subjects in the placebo group. The Peto OR suggested a 4.5 fold greater difference. This level of inflation with the Peto OR was found in 8 of the 22 trials we reviewed and was not mirrored in the other relative summary estimates (Table 2). Researchers need to be mindful of their methods and question their findings even when they are in the direction that they favor. We are now working on a separate statistically-focused manuscript that will provide in greater detail the limitations of the Peto OR under conditions such as those seen with the analysis of cardiovascular serious adverse events and varenicline use.
Fourth, regarding power, in reference to our work, Singh states, “They conflate the lack of statistical significance in an underpowered meta-analysis as clinically insignificant.” Our meta-analysis of 22 trials with over 9200 individuals found that the 95% confidence interval on the risk difference excluded an increase larger than 0.63%. That is, our analysis had sufficient power to detect a difference as small as two-thirds of one percent. Inconsistently, Singh later goes on to emphasize the need for a clinical trial with 8000 patients, which is less than the 9200 included in our meta-analysis. Notably, our analysis demonstrated that the summary estimate has not changed over time with the addition of more trials (Figure 3).
Meta-analytic methods are specifically useful for testing hypotheses that would be underpowered in individual trials. By combining studies, a meta-analysis increases the sample size and by drawing from independent samples, increases generalizability of the findings. It also should be noted that the Singh analysis included fewer subjects (i.e., 8216 subjects) and concerns about power were not raised.
Singh cites a post-marketing study and states it found “an increase in cardiovascular events with varenicline” [3]. An increase over what? The study, which did not have a comparison condition, drew no conclusion regarding elevated risk. Similarly, Singh and Thomas Moore, an investigative journalist and expert witness in litigation against Chantix, published a post-marketing varenicline analysis that has been criticized for having an unknown denominator with no data on anticipated base rates [4].
In closing, Singh writes, “Despite the removal of cardiovascular events on varenicline, and a statistical approach that has limited powered to detect an effect there is an excess risk of cardiovascular events with varenicline in all five measures reported in their study.” This statement is wrong in multiple ways. As noted above, our coding strategy mirrored that used by Singh and was applied consistently for both the varenicline and placebo groups. The excess risk to which Singh refers is neither clinically nor statistically significant and again, is most likely due to the differential attrition by group.
We agree with Singh that clinicians need to weigh the risks and benefits of medications that they recommend to their patients. To aid clinicians, however, the empirical evidence needs to be transparent and attentive to bias. Our re-analysis was conducted to address problems with methods that resulted in misleading conclusions. It is unfortunate that Singh seems not to appreciate the problems with the statistical methods he chose.
1. Singh S, Loke YK, Spangler JG, Furberg CD. Risk of serious adverse cardiovascular events associated with varenicline: a systematic review and meta-analysis. CMAJ 2011; 183(12):1359-66.
2. Hollis S, Campbell F. What is meant by intention to treat analysis? Survey of published randomised controlled trials. BMJ 1999; 319(7211): 670–674.
3. Harrison-Woolrych M, Maggo S, Tan M, Savage R, Ashton J. Cardiovascular events in patients taking varenicline: a case series from intensive postmarketing surveillance in New Zealand. Drug Saf 2012; 35(1): 33-43.
4. Moore TJ, Furberg CD, Glenmullen J, Maltsberger JT, Singh S. Suicidal behavior and depression in smoking cessation treatments. PLoS One. 2011; 6(11):e27016.
Competing interests:
Detailed disclosures included within our BMJ publication including grant funding from NIDA, NIMH, TRDRP, FAMRI, and a single investigator initiated research award from Pfizer, Inc. (#WS981308 registered online with www.clinicaltrials.gov, title is Varenicline Inpatient Study). The funding from Pfizer did not support the work of this meta-analysis.
Rapid Response:
Re: Risk of cardiovascular serious adverse events associated with varenicline use for tobacco cessation: systematic review and meta-analysis
Singh’s posting to BMJ suggesting our meta-analysis “has limitations in data, analysis and interpretation” is based on a number of misconceptions that warrant response.
First, citing the FDA label for Chantix, Singh criticized our analysis for excluding “cardiovascular adverse events from the varenicline arms that have been reported.” In fact, our coding of events directly and explicitly mirrored the coding strategy used by Singh et al. [1]. Both meta-analyses focused on cardiovascular serious adverse events. Hence, the myocardial infarction and stroke events reported in the FDA label were included as were the hospitalizations for angina, which occurred in both the varenicline (1 case) and placebo (2 cases) groups. The other 17 cases of stable angina, which occurred in both arms, were not serious (i.e., did not result in hospitalization) and were not included in either our analysis or that of Singh.
Second, Singh asserts that “the higher drop-out rate in the placebo group is irrelevant” and that his approach of counting events occurring at anytime during the trials, in many cases up to 52 weeks, “adhered to the intention-to-treat (ITT) analysis.” Yet, quoting an article from BMJ, “full application of intention to treat can only be performed where there is complete outcome data for all randomised subjects” [2]. In this case, not only was there incomplete data for randomized subjects, but the degree of incomplete data differed by condition, and in 13 of the 14 trials included by Singh, was biased in favor of detecting more events in the varenicline group. For participants who were lost to follow-up, Singh did not impute their data. For example, in the tobacco control field, missing subjects may be coded as treatment failures (i.e., coded as smokers), as a form of ITT analysis. Singh coded only the events as observed; we did the same in our analysis. The higher dropout rate in the placebo group is relevant. Singh’s analysis does not conform to ITT. Also relevant are the 8 trials with nearly 1600 participants that were excluded from Singh’s analysis because they did not have a single cardiovascular serious adverse event.
A difference in our methods was the time period of observation. Our observation period was the time participants were on drug or within 30 days of drug discontinuation. We chose this period because it is a biologically relevant time-frame given the half-life of varenicline, which leaves the body within 7 days, and it reduces the problem of differential dropout seen with Singh’s methods and the complexity of differentiating treatment versus disease (i.e., chronic tobacco exposure) effects on cardiovascular risk.
Third, Singh criticized our use of the risk difference for being “statistically underpowered at low event rates, and bias their estimates towards the null.” In fact, we presented the findings from three relative measures in addition to the risk difference and all four estimates were nonsignificant. We calculated and presented four different summary estimates to allow for transparent and direct comparison. Singh quotes the Cochrane Handbook (http://www.cochrane-handbook.org/), which actually does not recommend the Peto OR as the default approach for meta-analysis and specifically discourages its use when the studies have unequal allocation to experimental and control groups (section 9.4.4.2), as was the case for seven of the trials examining varenicline for tobacco cessation.
As an example, the trial by Bolliger observed 1 event in the treatment group with nearly 400 subjects versus no event in nearly 200 subjects in the placebo group. The Peto OR suggested a 4.5 fold greater difference. This level of inflation with the Peto OR was found in 8 of the 22 trials we reviewed and was not mirrored in the other relative summary estimates (Table 2). Researchers need to be mindful of their methods and question their findings even when they are in the direction that they favor. We are now working on a separate statistically-focused manuscript that will provide in greater detail the limitations of the Peto OR under conditions such as those seen with the analysis of cardiovascular serious adverse events and varenicline use.
Fourth, regarding power, in reference to our work, Singh states, “They conflate the lack of statistical significance in an underpowered meta-analysis as clinically insignificant.” Our meta-analysis of 22 trials with over 9200 individuals found that the 95% confidence interval on the risk difference excluded an increase larger than 0.63%. That is, our analysis had sufficient power to detect a difference as small as two-thirds of one percent. Inconsistently, Singh later goes on to emphasize the need for a clinical trial with 8000 patients, which is less than the 9200 included in our meta-analysis. Notably, our analysis demonstrated that the summary estimate has not changed over time with the addition of more trials (Figure 3).
Meta-analytic methods are specifically useful for testing hypotheses that would be underpowered in individual trials. By combining studies, a meta-analysis increases the sample size and by drawing from independent samples, increases generalizability of the findings. It also should be noted that the Singh analysis included fewer subjects (i.e., 8216 subjects) and concerns about power were not raised.
Singh cites a post-marketing study and states it found “an increase in cardiovascular events with varenicline” [3]. An increase over what? The study, which did not have a comparison condition, drew no conclusion regarding elevated risk. Similarly, Singh and Thomas Moore, an investigative journalist and expert witness in litigation against Chantix, published a post-marketing varenicline analysis that has been criticized for having an unknown denominator with no data on anticipated base rates [4].
In closing, Singh writes, “Despite the removal of cardiovascular events on varenicline, and a statistical approach that has limited powered to detect an effect there is an excess risk of cardiovascular events with varenicline in all five measures reported in their study.” This statement is wrong in multiple ways. As noted above, our coding strategy mirrored that used by Singh and was applied consistently for both the varenicline and placebo groups. The excess risk to which Singh refers is neither clinically nor statistically significant and again, is most likely due to the differential attrition by group.
We agree with Singh that clinicians need to weigh the risks and benefits of medications that they recommend to their patients. To aid clinicians, however, the empirical evidence needs to be transparent and attentive to bias. Our re-analysis was conducted to address problems with methods that resulted in misleading conclusions. It is unfortunate that Singh seems not to appreciate the problems with the statistical methods he chose.
1. Singh S, Loke YK, Spangler JG, Furberg CD. Risk of serious adverse cardiovascular events associated with varenicline: a systematic review and meta-analysis. CMAJ 2011; 183(12):1359-66.
2. Hollis S, Campbell F. What is meant by intention to treat analysis? Survey of published randomised controlled trials. BMJ 1999; 319(7211): 670–674.
3. Harrison-Woolrych M, Maggo S, Tan M, Savage R, Ashton J. Cardiovascular events in patients taking varenicline: a case series from intensive postmarketing surveillance in New Zealand. Drug Saf 2012; 35(1): 33-43.
4. Moore TJ, Furberg CD, Glenmullen J, Maltsberger JT, Singh S. Suicidal behavior and depression in smoking cessation treatments. PLoS One. 2011; 6(11):e27016.
Competing interests: Detailed disclosures included within our BMJ publication including grant funding from NIDA, NIMH, TRDRP, FAMRI, and a single investigator initiated research award from Pfizer, Inc. (#WS981308 registered online with www.clinicaltrials.gov, title is Varenicline Inpatient Study). The funding from Pfizer did not support the work of this meta-analysis.