Influence of healthy candidate bias in assessing clinical effectiveness for implantable cardioverter-defibrillators: cohort study of older patients with heart failureBMJ 2014; 348 doi: https://doi.org/10.1136/bmj.g2866 (Published 08 May 2014) Cite this as: BMJ 2014;348:g2866
- Soko Setoguchi, associate professor12,
- Lynne Warner Stevenson, professor3,
- Garrick C Stewart, instructor3,
- Deepak L Bhatt, professor3,
- Andrew E Epstein, professor4,
- Manisha Desai, associate professor5,
- Lauren A Williams, research assistant1,
- Chih-Ying Chen, postdoctoral fellow1
- 1Division of Pharmacoepidemiology and Pharmacoeconomics, Brigham and Women’s Hospital and Harvard Medical School, Boston, MA, USA
- 2Duke Clinical Research Institute, Duke University School of Medicine, Durham, NC 27715, USA
- 3Division of Cardiovascular Medicine, Brigham and Women’s Hospital and Harvard Medical School, MA, USA
- 4Electrophysiology Section, Cardiovascular Division, University of Pennsylvania, Philadelphia, PA, USA
- 5Quantitative Sciences Unit, Stanford University School of Medicine, Palo Alto, CA, USA
- Correspondence to: S Setoguchi
- Accepted 25 March 2014
Objective To assess the potential contribution of unmeasured general health status to patient selection in assessments of the clinical effectiveness of implantable cardioverter-defibrillator (ICD) therapy.
Design Retrospective cohort study.
Setting Linked data from an ICD registry, heart failure registry, and Medicare claims data for ICDs implanted in 2005 through 2009.
Participants 29 426 patients admitted to hospital with heart failure aged 66 years or older and eligible for ICD therapy for primary prevention.
Main outcome measures Non-traumatic hip fracture, admission to a skilled nursing facility, and 30 day mortality—outcomes unlikely to be improved by ICD therapy.
Results Compared with 17 853 patients without ICD therapy, 11 573 patients with ICD therapy were younger and had lower ejection fraction and more cardiac admissions to hospital but fewer non-cardiac admissions to hospital and comorbid conditions. Patients with ICD therapy had greater freedom from unrelated events after adjusting for age and sex: hip fracture (hazard ratio 0.77, 95% confidence interval 0.64 to 0.92), skilled nursing facility admission (0.53, 0.50 to 0.55), and 30 day mortality (0.12, 0.10 to 0.15).
Conclusions Lower risks of measured outcomes likely reflect unmeasured differences in comorbidity and frailty. The findings highlight potential pitfalls of observational comparative effectiveness research and support physician consideration of general health status in selecting patients for ICD therapy.
After several landmark clinical trials showed the efficacy of implantable cardioverter-defibrillator (ICD) therapy,1 2 the US Centers for Medicare & Medicaid Services (CMS) expanded coverage of ICD therapy to include primary prevention of sudden cardiac death in Medicare beneficiaries with heart failure.3 Medicare beneficiaries mostly consist of those aged 65 years or older, in whom the risk of sudden cardiac death is lower than in the trial patients.4 5 6 None the less, one study projected that ICD therapy may be indicated for as many as 800 000 additional people with heart failure in the United States, most of whom are 65 years or older.7
In real world modern clinical practice, patients are selected to receive ICD therapy on the basis of multiple factors, including indications and contraindications, underlying disease severity, comorbid conditions, and overall prognosis. Guideline recommendations for both heart failure and device based therapy include “anticipated survival of at least a year with good functional capacity,” which is a crucial but subjective clinical assessment. In older patients, cognitive and functional capacity and social support can influence not only prognosis but also preferences for a device that prevents sudden cardiac death without improvement in quality of life. Information about these factors is often unavailable in large databases, including claims data, registries, and medical records, which affect the validity of assessing comparative effectiveness of ICD therapy in real world patients using these data.
To understand the contribution of unmeasured general health status to selection of patients for ICD therapy and potential bias in observational studies using registries and claims data, we compared three outcomes among patients with and without ICD therapy that can reflect general health status but are unlikely to be improved by ICD therapy: hip fracture, admission to a skilled nursing facility, and 30 day mortality.
We conducted a retrospective cohort study of older patients with heart failure using the CMS ICD registry (2005-08); the National Cardiovascular Database Registry’s ICD registry (2005-08); a national clinical registry of patients with heart failure aggregated from several quality improvement and accreditation programs, including the American Heart Association’s Get With the Guidelines program (2005-08); and data files on Medicare institutional and non-institutional claims (2004-09) (see web extra appendix I).
We linked the combined ICD registry and the heart failure registry to Medicare claims data using four non-unique identifiers: date of birth, sex, admission date for implantation of the ICD, and provider ID, which is described in detail elsewhere.8 Briefly, we validated this linkage among the subset of 196 923 patients who had a unique identifier in the ICD registry. Our linkage using non-unique identifiers yielded 98% specificity, 95% sensitivity, and 98% positive predictive value compared with the linkage method using both non-unique and unique identifiers.8
Study population, cohorts, and treatment status
Our study population represented patients with heart failure who were 66 years or older and eligible for ICD therapy for primary prevention (see web extra appendix II for detailed information on the study population). We identified two cohorts of patients with and without ICD therapy in the linked databases (fig 1⇓). Cohort 1 consisted of the population of Medicare beneficiaries who were linked to the ICD registry (areas A and B in fig 1) and the population of Medicare beneficiaries who were linked to the heart failure registry but not with the ICD registry (area C). Cohort 2, a smaller but conceptually more appropriate study population, consisted of Medicare beneficiaries who were linked to the heart failure registry with ICD therapy (area B) and without ICD therapy (area C). We conducted analyses in both cohorts to ensure that the findings were similar. All study patients had an ejection fraction of 35% or less. We excluded patients who had a history of cardiac arrest or sustained ventricular tachycardia. All patients were eligible for Medicare for one year before the index procedure or the index admission to hospital and were 66 years or older. We excluded patients who underwent both cardiac resynchronization therapy and ICD therapy, because biventricular pacing improves cardiac function and could improve patients’ functional status, potentially reducing the risk of the outcomes of interest.
We selected three outcomes that are not causally associated with ICD therapy, are markers for frailty and other unmeasured pre-existing patient characteristics, and can be accurately measured with high specificity in Medicare data. These included admission for non-traumatic hip fracture (that is, fracture not due to trauma other than a fall and often a consequence of osteoporosis and a fall), admission to a skilled nursing facility (a licensed facility that provides short term or long term skilled care, commonly called a “nursing home”), and 30 day mortality. We identified admissions to hospital for non-traumatic hip fracture on the basis of a diagnosis of hip fracture (international classification of diseases, 9th revision, clinical modification (ICD-9-CM) diagnosis code 820.xx or 733.14) and a related procedure code (ICD-9-CM procedure code 78.55, 79.05, 79.15, 79.25, 79.35, or 79.65 or current procedural terminology (CPT) code 27230-27248).9 We identified admissions to a skilled nursing facility in the Medicare institutional files. Because some such skilled nursing facility admissions identified in Medicare data are short term and may not be totally related to frailty, we also identified a subset of nursing home admissions that resulted in stays of more than 20 days (beyond the period for full Medicare coverage) or death within 20 days as a more accurate indicator of frailty. We chose mortality within 30 days, a timeframe during which prophylactic ICD therapy did not provide a benefit in the landmark clinical trials. For example, the Sudden Cardiac Death in Heart Failure Trial (SCD-HeFT) found that the mortality benefit started to appear approximately 1.5 years after implantation of an ICD.1 All patients were followed until the occurrence of the outcomes and death from the index date, which was the date of implantation for those who received ICD therapy and the discharge date from their admission to hospital for heart failure for those who did not receive ICD therapy.
In patients with or without ICD therapy, we described baseline characteristics, including use of several preventive care and screening tests as markers for health seeking behaviors. We reported counts and person time for each outcome and graphed cumulative outcome event risks over time using Kaplan-Meier estimators. We used proportional hazards regression to derive crude and adjusted hazard ratios comparing patients with and without ICD therapy.
Some values were missing for systolic blood pressure and serum levels of sodium, B type natriuretic peptide, and creatinine in some patients (see table 1). We handled missing values by using multiple imputation10 and assuming an underlying multivariate normal distribution. Our analysis was based on five imputed datasets where the imputation model included all variables in the outcome model (ICD use outcomes and potential confounders) as well as variables related to missingness (see web extra appendix IV).11
We repeated the same analyses after excluding patients older than 80 years and those with a previous admission to a skilled nursing facility, because the indication criterion of “expected survival with good functional capacity for >1 year” is less likely to be met for these patients.
We used SAS version 9.2 (SAS Institute, Cary, NC) for all analyses.
We identified 29 426 patients with heart failure (11 573 with ICD therapy and 17 853 without ICD therapy) in cohort 1 who met the eligibility criteria. Patients who received ICD therapy were younger and more likely to be men and white than patients who did not receive ICD therapy. Patients with ICD therapy also had lower ejection fractions, more previous admissions to hospital for cardiac diseases, more physician visits, and ischemic etiology of heart failure (table 1⇓). Patients without ICD therapy had a higher prevalence of non-cardiac admissions to hospital, previous admissions to a skilled nursing facility, more chronic kidney or lung disease, metastatic cancer, and other non-cardiovascular diseases. The prevalence of many preventive procedures and screening tests was higher in patients with ICD therapy. These findings were similar in cohort 2.
During 42 580 person years of follow-up (mean, 1.4 years) in cohort 1, we observed 676 admissions to hospital for hip fracture and 9475 admissions to a skilled nursing facility (table 2⇓). During the first 30 days, 2428 deaths occurred. Patients without ICD therapy had substantially higher incidence rates of non-traumatic hip fracture (17 v 9 per 1000 person years), admissions to a skilled nursing facility (354 v 112 per 1000 person years), and 30 day mortality (1699 v 165 per 1000 person years).
The cumulative risk of hip fracture at three years was 4.8% (95% confidence interval 4.3% to 5.3%) among patients without ICD therapy and 2.6% (2.3% to 3.1%) among patients with ICD therapy. The cumulative incidence curves diverged immediately after implantation of the ICD (fig 2⇓). The cumulative risk of admission to a skilled nursing facility at one year was 13.4% (12.8% to 14.0%) with ICD therapy and 35.7% (34.9% to 36.4%) without ICD therapy. These curves diverged immediately and paralleled within a few days after implantation of the ICD (fig 2). The pattern was similar for 30 day mortality (fig 2). The findings were similar in cohort 2 (see web extra appendix V).
Unadjusted rates of outcomes in patients with ICD therapy were lower than in patients without ICD therapy for admission for non-traumatic hip fracture (hazard ratio 0.51, 95% confidence interval 0.43 to 0.59), admission to a skilled nursing facility (0.39, 0.37 to 0.41), and 30 day mortality (0.10, 0.09 to 0.12) (table 3⇓). After adjustment for age and sex, these estimates moved toward 1. After adjustment for all patient measured characteristics, these estimates moved further but only moderately toward 1 and the risks remained substantially lower among patients who received ICD therapy (table 3).
The refined outcome for admission to a skilled nursing facility, which used a stay of 20 days or longer or death within 20 days, yielded similar results (tables 2 and 3). After adjustment for all measured factors, the risks remained significantly lower among patients with ICD therapy. Restricting the cohorts to patients 80 years or younger (see web extra appendix VI) or with no previous admission to a skilled nursing facility (see web extra appendix VII) did not change the results meaningfully.
We compared older patients with and without implantable converter-defibrillator (ICD) therapy for the risk of three outcomes that are unlikely to be directly improved by ICD therapy to assess the impact of unmeasured patient characteristics on patient selection for ICD therapy. We observed a 50% to 60% lower risk for admission for non-traumatic hip fracture, admission to a skilled nursing facility, and 30 day mortality in patients with ICD therapy compared with patients with heart failure without ICD therapy. These differences in the risks of unrelated outcomes were observed immediately after implantation of the ICD. The differences lessened after adjustment for measured patient characteristics, but risks remained lower in patients who received ICD therapy by 16% to 34%. Given the timing and size of the observed differences, these results likely reflects baseline differences between the candidates for ICD therapy and patients who did not receive ICD therapy that are not measured in Medicare or registry data, leading to selection bias or “healthy candidate bias.” From a clinical perspective, these differences reflect appropriate integration of multiple health dimensions into clinical decision making in modern clinical practice to select patients who are most likely to benefit.
The healthy candidate bias that we observed is analogous to biases described in other fields of epidemiology. Occupational epidemiologists have long recognized the “healthy worker survivor effect” when evaluating effects of environmental exposures. This bias arises in comparisons of workers exposed to work related toxins and workers not exposed because relatively healthy people are likely to gain employment and remain employed, whereas severely ill and chronically disabled people are ordinarily excluded from employment.12 Similar bias, known as “healthy user bias” in pharmacoepidemiology, has been observed among users of preventive medications and vaccines, such as hormone replacement therapy,13 14 statins,15 16 and influenza vaccine.17 Because implantable devices and surgical procedures typically pose short term risks in exchange for long term benefits, patients at high risk of complications or deemed too sick to benefit are less likely to be selected. In the Cardiac Arrhythmia Suppression Trial, a similar effect was observed.18 This healthy candidate effect is one of the biggest threats to validity in observational comparative effectiveness research, especially in comparisons of invasive interventions to less invasive alternatives.
Frailty reflected by low functional status and impaired cognitive function, lack of social support, unrecognized or untreated depression, and diminished quality of life are associated with decreased survival in patients with heart failure.19 20 21 22 23 Each of these factors may be associated with the decision to offer and receive ICD therapy. For example, preference for survival was strongly associated with longer survival in patients with heart failure in the Evaluation Study of Congestive Heart Failure and Pulmonary Artery Catheter Effectiveness.24 Among patients who survived fewer than 105 days, 31% indicated that they would trade more than 90% of their survival days to feel well for the time remaining, compared with 6% of patients who survived all 180 days (P<0.005), indicating that those most likely to die were least likely to have cared about prolonging survival.24 In another survey study of 105 consecutive patients with mean ejection fraction of 21%, 65% of whom received ICD therapy for primary prevention, recipients of ICD therapy expressed greater confidence that the device would save their own lives than those without ICD therapy (P<0.001).25 These data indicate that recipients of ICD therapy tend to have a strong preference for survival, which is likely to correlate directly with lower rates of death, admission to a skilled nursing facility, and even hip fracture indirectly. Finally, incomplete depiction of non-cardiovascular comorbidity is likely another source of unmeasured differences between patients with and without ICD therapy.25 26
Clinical and policy implications
Eligibility criteria for ICD therapy from both the heart failure guidelines26 27 28 and the device based therapy guidelines29 28 from the American College of Cardiology, the American Heart Association, and the European Society of Cardiology include anticipated survival with “good functional capacity” for at least one year. We found that 13% of patients with ICD therapy and 36% of patients without ICD therapy were admitted to skilled nursing facilities within a year if they did not die. Considering a composite outcome of admission to a skilled nursing facility and death, 55% of patients without ICD therapy would not meet the expectation of one year survival with good functional capacity, whereas a smaller fraction (21%) of patients with ICD therapy would meet the expectation.
A few previous studies have concluded that ICD therapy may be underused in general or certain Medicare patients without specific contraindications to implantation of a ICD for primary prevention of sudden cardiac death.29 30 31 While existing cardiovascular registries and Medicare data include information on skilled nursing facilities and ICD-9-CM diagnosis codes for comorbid conditions, they lack documentation of the severity of comorbidity, socioeconomic factors, functional status, cognitive function, and quality of life. Therefore, apparent gaps in ICD use assessed using information available in registries and Medicare data may arise, at least in part, from incomplete depiction of the appropriateness of patient selection for ICD therapy.
Our results indicate that a substantial fraction of patients who were potentially eligible for ICD therapy but did not receive it may have been at high risk for not “surviving with good quality of life.” However, our study had limited ability to identify patients who were specifically at higher risk for admission to a skilled nursing facility. There are without doubt patients who did not receive ICD therapy who would have derived meaningful life extension from prevention of sudden death, which are also not possible to identify in our analyses. Furthermore, some patients may have declined ICD therapy despite an appropriate recommendation.
Implications for observational comparative effectiveness research
Although observational studies provide data on real world patients in real world settings, they are subject to potential bias by selection, confounding, survivor bias, and inaccurate measurement of variables.31 32 Confounding by indication is described well in the pharmacoepidemiology literature32 33and can cause significant bias. However, registries typically include information on indications and contraindications for treatments. Therefore, confounding by indication was not a problem in our study using linked registry and claims data. Healthy candidate bias, a type of selection or confounding bias, can be the most challenging bias in observational comparative effectiveness research for devices and surgeries. We showed the existence of substantial healthy candidate bias in our ICD example, which was not controlled by multivariable adjustments for measured factors and might have affected, at least in part, previous ICD effectiveness studies.33 34 35
Various design-based and statistical approaches have been suggested to overcome healthy worker survivor effect or healthy user bias. These include restriction to survivors,35 36 lagged time or latency analysis,36 37 g-computation,37 38 39 and comparison to active users of similar preventive medications.39 40 The technique of restriction to a subset of survivors or to more comparable groups is a standard approach in epidemiology but one that is sometimes overlooked or intentionally avoided in favor of a larger sample size. Other methods, including latency analysis and g-computation, should be considered and used when appropriate. Finally, adjustment for proxies for unmeasured factors by maximizing available information in large databases using appropriate methods (for example, high dimensional propensity score methods) can potentially achieve better control for bias in some settings.40 41
This study suggests two important implications. Firstly, those responsible for the allocation of scarce healthcare resources in any country must carefully evaluate whether the efficacy evidence of expensive technologies such as ICD therapy apply to their use outside of clinical trials. Secondly, evaluations of effectiveness from clinical and administrative databases may overestimate the benefit of device or surgical therapies for which selection is influenced by healthy candidate bias.
Several limitations should be considered in interpreting these findings. Firstly, we identified patients who received and did not receive ICD therapy from different registries, which limited our ability to adjust for some factors such as New York Heart Association functional classification that were only measured in one of the registries. However, all patients were admitted to hospital with heart failure, both registries had equally limited information on non-cardiovascular comorbid conditions, and neither registry included information on frailty or social support. Also, all patients were Medicare beneficiaries, which was the target population of our study, and the findings were replicated in the secondary cohort (cohort 2) identified in the heart failure registry. In the comparison of 30 day mortality, it is possible that there was early excess non-sudden death in the ICD arm, as reported after myocardial infarction.41 42 However, this would only have served to diminish the noticeable excess of 30 day mortality observed among patients who did not receive ICD therapy. For the non-fatal outcomes, competing risks by death exist in the current study. However, if ICD survival benefit from the trials was translated into this population, our results would have been underestimates of the true differences in the occurrence of hip fracture and admissions to a skilled nursing facility, rendering the selection due to frailty and other unmeasured factors even larger than observed here.
Difference in survival between patients receiving and not receiving ICD therapy in this large observational study was influenced by unmeasured differences in baseline health status, as indicated by lower risk of non-traumatic hip fracture, admission to a skilled nursing facility, and 30 day mortality among recipients of ICD therapy. Our analyses of early mortality and non-fatal outcomes highlight potential pitfalls of observational comparative effectiveness analyses of outcomes associated with one treatment versus another due to healthy candidate bias. It is not possible to determine from this study the degree to which apparent underutilization of such therapies may instead reflect thoughtful clinical decisions in which doctors use factors such as frailty to allocate expensive therapies to patients who are most likely to benefit.
What is already known on this topic
Clinical trials of implantable cardioverter-defribrillators (ICDs) under-represent older patients with comorbidities, in whom the risk of sudden death might be lower, but many ICDs have been used in the United States and other countries
In clinical practice, treatments are not randomly allocated but selected based on indications and other considerations
Unmeasured selection factors can influence estimates of comparative effectiveness in observational studies, as shown by the healthy worker effect or healthy user bias in occupational research and pharmacoepidemiology
What this study adds
Evaluations of effectiveness from clinical and administrative databases may overestimate the benefit of device or surgical therapies for which selection is influenced by healthy candidate bias
Apparent underutilization of ICDs in older patients may instead reflect thoughtful clinical decisions in clinical practice in which clinicians use factors such as frailty to allocate expensive therapies to patients who are most likely to benefit
Cite this as: BMJ 2014;348:g2866
Harlan M Krumholz, Yale University, provided thoughtful comments on an early draft of the manuscript. Damon M Seils, Duke University, provided editorial assistance and helped prepare the manuscript for submission.
This work was presented in part at the Agency for Healthcare Research and Quality 2011 national conference, September 21, 2011, in Bethesda, Maryland, and at the 2013 scientific meeting for the American Heart Association, November 18, 2013, in Dallas, Texas.
Contributors: SS and LWS contributed to the acquisition of the data. SS and LWS developed the study design. SS, CYC, and MD contributed to the analysis of the data. SS and CYC drafted the manuscript. All authors contributed to the interpretation of the data and the revision of the work, and all approved the final version to be published. SS is guarantor. The authors of this report are responsible for its content. Statements in the report should not be construed as endorsement by the Agency for Healthcare Research and Quality or the US Department of Health and Human Services.
Funding: This project was funded under contract No HHSA29020050016I, task order 3, from the Agency for Healthcare Research and Quality, US Department of Health and Human Services, as part of the Developing Evidence to Inform Decisions about Effectiveness (DEcIDE) program; and contract No HHSM500201000001I, task orders 2 and 6, from the Centers for Medicare & Medicaid Services, US Department of Health and Human Services. SS was supported by midcareer development award K02HS017731 from the Agency for Healthcare Research and Quality.
Competing interests: All authors have completed the ICMJE uniform disclosure form at www.icmje.org/coi_disclosure.pdf (available on request from the corresponding author). SS reported receiving grants from Agency for Healthcare Research and Quality and from the Center for Medicare and Medicaid Services during the conduct of the study. She has made available online a detailed listing of financial disclosures (www.dcri.duke.edu/about-us/conflict-of-interest/). DLB reported receiving grants from Amarin, AstraZeneca, Bristol-Myers Squibb, Eisai, Ethicon, Medtronic, Roche, Sanofi Aventis, and The Medicines Company; conducting unfunded research for FlowCo, PLx Pharma, and Takeda; receiving personal fees from the Duke Clinical Research Institute, the Mayo Clinic, the Population Health Research Institute, the American College of Cardiology, Belvoir Publications, Slack Publications, WebMD, and Elsevier Practice Update Cardiology; serving on advisory boards for Elsevier Practice Update Cardiology, Medscape Cardiology, and Regado Biosciences; serving on the board of directors of the Boston VA Research Institute and the Society of Cardiovascular Patient Care; receiving non-financial support from the Society of Cardiovascular Patient Care and the American Heart Association; and serving as editor for the Journal of Invasive Cardiology, outside of the submitted work. AEE reported receiving honorariums from Biotronik, Boston Scientific, Medtronic, and St Jude Medical; receiving research funding from Biotronik, Boston Scientific, Medtronic, and St Jude Medical; and receiving fellowship program support from Biotronik, Boston Scientific, Medtronic, and St Jude Medical.
Ethical approval: This study was approved by the institutional review board of Brigham and Women’s Hospital.
Data sharing: No additional data available.
Transparency: The lead author affirms that this manuscript is an honest, accurate, and transparent account of the study being reported; that no important aspects of the study have been omitted; and that any discrepancies from the study as planned (and, if relevant, registered) have been explained.
This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 3.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/3.0/.