Original ArticleTime-dependent bias was common in survival analyses published in leading clinical journals
Introduction
Bias is the systematic deviation of study results or inferences from the truth. Because bias can lead to erroneous conclusions, its minimization is a critical goal of all good research [1]. Bias can arise during all study phases, including its design, conduct, or analysis [2]. Several studies have found extensive amounts of inappropriate statistical methodology in published papers [3], [4], [5]. Analytical bias is important because it often can be avoided with proper statistical methodology.
In a time-to-event or survival analysis [6], problems can occur when variables in the model change value after the start of patient observation. Such variables are called “time-dependent,” because their value can change over time. There are two general categories of time-dependent variables. “Baseline measurable” time-dependent variables, like systolic blood pressure and body mass index, can change over time but are measurable at baseline. These “baseline measurable” time-dependent variables are frequently analyzed as fixed, or unchanging, variables in survival analyses. In contrast, “baseline immeasurable” time-dependent variables cannot be measured at baseline and indicate what happened to patients during observation.
Biased estimates can occur if “baseline immeasurable” time-dependent variables are analyzed as fixed variables. Consider a hypothetic study determining prognosticators for patients who have a perforation of the sigmoid and undergo emergency hemicolectomy with colostomy. Patients who die in the first several months after the operation will never undergo closure of their colostomy. If this “baseline immeasurable” time-dependent factor (“Was colostomy closed?”) is analyzed in a survival analysis as a fixed variable, one would associate no colostomy closure with a worse survival. This association is erroneous, because death results in the colostomy not getting closed, rather than vice versa.
For a real example, consider a study in which we examined the effect of discharge summary dissemination upon readmission to hospital [7]. Our “baseline-immeasurable” time-dependent variable was “Did the patient have a follow-up visit with a physician who had received the discharge summary.” When it was analyzed as a fixed variable, we found a large reduction in readmission to hospital when patients saw physicians with the summary (adjusted hazard ratio [HR] 0.35, 95% CI 0.24–0.52). This is a biased association, because patients who are readmitted to the hospital early after discharge do not have a chance to see such physicians and are placed in the “no-summary” group. This makes outcomes for the “no-summary” group particularly poor, thereby making follow-up with physicians who had the summary look beneficial by comparison. When a time-dependent analysis is used, we found a much smaller effect of follow-up with a physician who had received the summary (adjusted HR 0.74, 95% CI 0.50–1.11).
In these two examples, biased conclusions occur because patient categorization with respect to the “baseline-immeasurable” time-dependent variable was partially due to their outcome. Glesby and Hoover termed this “survivor treatment selection bias” [8]. Because this bias is not limited to treatment variables or analyses in which survival is the outcome, we will use the more generic term “time-dependent bias” in this review. Time-dependent bias can be avoided with the proper use of a time-dependent covariate analysis [8], [9], [10], [11].
There are several reasons why time-dependent bias could be common in the medical literature. The use of time-to-event analyses has increased in the medical literature over the last 20 years [12]. In our experience, time-dependent analysis of survival data is infrequently taught during research training and is often given a cursory review in some epidemiologic textbooks. In many journals, articles infrequently undergo formal statistical review prior to publication [13].
We critically reviewed the survival analyses published in prominent general medical journals to determine whether time-dependent bias due to improper analytical methodology is common. We also wanted to determine if its prevalence has recently changed, and how often study conclusions would qualitatively change if the analysis had properly accounted for the time-dependent nature of the data.
Section snippets
Study selection and inclusion criteria
We reviewed studies containing survival analyses that were published in nine medical journals including the American Journal of Medicine, Annals of Internal Medicine, Archives of Internal Medicine, British Medical Journal (BMJ), Circulation, Chest, Journal of the American Medical Association (JAMA), Lancet, and New England Journal of Medicine. These journals were selected because they are general medical, respiratory, and cardiovascular journals with a broad readership, large circulation, and a
Results
Our search strategy resulted in 1,184 papers (see Appendix A). Four hundred seventy-seven papers did not have a survival analysis (Fig. 1). Of the 707 survival analyses, 25 were excluded because they did not determine the association of an outcome with any patient factor.
This left 682 survival analyses that determined the association of at least one variable with an outcome (Table 1). The number of survival analyses published per year increased during the study period. The Kaplan-Meier method,
Discussion
We found that time-dependent bias due to improper analytic methodology is surprisingly common in these medical journals. When present, time-dependent bias frequently affected the strength of a purported association that was highlighted in the abstract. The prevalence of time-dependent bias appears to be decreasing. In over half of the papers susceptible to time-dependent bias, its correction could have changed the studies' conclusion.
Time-dependent bias is important for several reasons. It can
Acknowledgements
Dr. van Walraven is an Ontario Ministry of Health Career Scientist. We greatly appreciate comments from Drs. Don Redelmeier, Dean Fergusson, Paul Hebert, and David Sackett regarding previous versions of this study.
References (97)
Bias in analytic research
J Chronic Dis
(1979)- et al.
Review of statistics usage in the American Journal of Obstetrics and Gynecology
Am J Obstet Gynecol
(1996) - et al.
Logistic regression in the medical literature: standards for use and reporting, with particular attention to one medical domain
J Clin Epidemiol
(2001) - et al.
High agreement but low kappa:II. Resolving the paradoxes
J Clin Epidemiol
(1990) - et al.
Steroids in idiopathic pulmonary fibrosis: a prospective assessment of adverse reactions, response to therapy, and survival
Am J Med
(2001) - et al.
Primary graft failure following lung transplantation
Chest
(1998) - et al.
Unfavorable prognosis of patients with stage II non-small cell lung cancer associated with macroscopic nodal metastases
Chest
(1999) - et al.
Association of tumour site and sex with survival benefit from adjuvant chemotherapy in colorectal cancer
Lancet
(2000) - et al.
Prognostic role of pericardial fluid cytology in cardiac tamponade associated with non-small cell lung cancer
Chest
(2000) - et al.
Primary intrathoracic malignant effusion: a descriptive study
Chest
(2001)
Counting alleles to predict recurrence of early-stage colorectal cancers
Lancet
Influence of infant-feeding patterns on early mother-to-child transmission of HIV-1 in Durban, South Africa: a prospective cohort study. South African Vitamin A Study Group
Lancet
Longevity of atomic-bomb survivors
Lancet
Prevention of cytomegalovirus infection and disease after lung transplantation : results using a unique regimen employing delayed ganciclovir
Chest
Survival following bilateral staple lung volume reduction surgery for emphysema
Chest
The association between CagA status and the development of esophagitis after the eradication of Helicobacter pylori
Am J Med
Rest-redistribution 201-Tl single-photon emission CT imaging for determination of myocardial viability: relationship among viability, mode of therapy, and long-term prognosis
Chest
Granulocyte colony-stimulating factor use is associated with decreased bacteremia and increased survival in neutropenic HIV-infected patients
Am J Med
Pulmonary metastases of endocrine origin: the role of surgery
Chest
Prognosis and recurrent patterns in bronchioloalveolar carcinoma
Chest
Pregnancy in cystic fibrosis. Fetal and maternal outcome
Chest
Rate of heart failure and 1-year survival for older people receiving low-dose beta-blocker therapy after myocardial infarction
Lancet
Poor bone quality or hip structure as risk factors affecting survival of total-hip arthroplasty
Lancet
Impact of body weight on long-term survival after lung transplantation
Chest
The impact of follow-up physician visits on emergency readmissions for patients with asthma and chronic obstructive pulmonary disease: a population-based study
Am J Med
Fecal incontinence in the institutionalized elderly: incidence, risk factors, and prognosis
Am J Med
Cytomegalovirus viremia in lung transplant recipients receiving ganciclovir and immune globulin
Chest
Interventional bronchoscopy for the management of airway complications following lung transplantation
Chest
Clinical progression and virological failure on highly active antiretroviral therapy in HIV-1 patients: a prospective cohort study. Swiss HIV Cohort Study
Lancet
Acute respiratory failure in the United States: incidence and 31-day survival
Chest
Persistent increases of BAL neutrophils as a predictor of mortality following lung transplant
Chest
An immunological algorithm to predict risk of high-grade rejection in cardiac transplant recipients
Lancet
Clinical and radiographic manifestations of uncommon pulmonary nontuberculous mycobacterial disease in AIDS patients
Chest
Survival in patients with cryptogenic fibrosing alveolitis: a population-based cohort study
Chest
Cisplatin, mitomycin, and interferon-alpha2a combination chemoimmunotherapy in the treatment of diffuse malignant pleural mesothelioma
Chest
Effects of smoking on survival and morbidity in patients randomized to medical or surgical therapy in the Coronary Artery Surgery Study (CASS): 10-year follow-up. CASS Investigators
J Am Coll Cardiol
Fundamentals of epidemiologic research
The use of statistics in the British Journal of Psychiatry
Br J Psychiatry
Time to event (survival) data
BMJ
The effect of discharge summary availability during post-discharge outpatient visits on readmission to hospital
J Gen Intern Med
Survivor treatment selection bias in observational studies: examples from the AIDS literature
Ann Intern Med
Regression models and life tables
J R Stat Soc
Estimating Cox-regression models with PROC PHREG. Survival analysis using the SAS system
Time-dependent covariates in the Cox proportional-hazards regression model
Annu Rev Public Health
Survival analysis in clinical trials: past developments and future directions
Biometrics
Statistical reviewing policies of medical journals: caveat lector?
J Gen Intern Med
Interferon therapy reduces the risk for hepatocellular carcinoma: national surveillance program of cirrhotic and noncirrhotic patients with chronic hepatitis C in Japan. IHIT Study Group. Inhibition of Hepatocarcinogenesis by Interferon Therapy
Ann Intern Med
Impact of aortic stiffness attenuation on survival of patients in end-stage renal failure
Circulation
Cited by (272)
Comparative effectiveness of individual pneumococcal vaccines with dual pneumococcal vaccination in older United States Veterans
2022, VaccineCitation Excerpt :We identified three exposure groups: PCV 13, only PPSV 23 and dual vaccination (combined PCV 13 and PPSV 23). As our exposure groups are based on a time-varying covariate – pneumococcal vaccination status, simply assigning individuals to their initial vaccination status would introduce time-dependent bias.[22] Hence, we employed a within-patient crossover design to appropriately assign exposure time in person-days to the appropriate exposure groups.[23,24]
Letter to the Editor: Caution warranted when interpreting registry studies of medication. A response to Boland et al. (2020)
2022, Journal of Psychiatric ResearchImpact of biologics on health-related quality of life in patients with Ankylosing spondylitis: A systematic review and meta-analysis of randomized controlled trials
2022, Seminars in Arthritis and RheumatismA Cautionary Tale: Grouping Patients on Late Events
2022, Mayo Clinic ProceedingsAssociation between antibiotic resistance in intensive care unit (ICU)-acquired infections and excess resource utilization: Evidence from Spain, Italy, and Portugal
2022, Infection Control and Hospital Epidemiology