Association between physician US News & World Report medical school ranking and patient outcomes and costs of care: observational study

Data

We linked the 20% Medicare Inpatient Carrier and Medicare Beneficiary Summary Files from 2011 to 2015 to a comprehensive physician database from Doximity. Doximity is an online professional network for physicians that has assembled data on all US physicians—both those who are registered members of the service as well as those who are not—through multiple sources and data partnerships: the National Plan and Provider Enumeration System National Provider Identifier Registry, state medical boards, specialty societies such as the American Board of Medical Specialties, and collaborating hospitals and medical schools. The database includes information on physician age, sex, year of medical school completion, credentials (allopathic versus osteopathic training), specialty, and board certification.19202122 Prior studies have validated data for a random sample of physicians in the Doximity database using manual audits.1920 We were able to match approximately 95% of physicians in the Medicare database to the Doximity database. Details of the Doximity database are described elsewhere.11920222324

Patient population

We analysed Medicare fee-for-service beneficiaries aged 65 years or older admitted to hospital with a medical condition (as defined by the presence of a medical Medicare Severity Diagnosis Related Group (MS-DRG) on admission) between 1 January 2011 and 31 December 2015. To avoid comparing patient outcomes across physicians of different specialties, we focused on patients treated by general internists. We restricted our sample to patients treated in acute care hospitals, and excluded hospital admissions where a patient left against medical advice. To minimize the influence of patients selecting their physicians or physicians selecting their patients, we focused our analyses on emergency hospital admissions, defined as either emergency or urgent admissions identified in Claim Source Inpatient Admission Code of Medicare data (although we were not only interested in patients admitted as an emergency, this approach was necessary to reduce the impact of unmeasured confounding). To allow sufficient follow-up, we excluded patients admitted in December 2015 from 30 day mortality analyses and patients discharged in December 2015 from readmission analyses.

Medical school rankings

Data on medical school attended were available for approximately 80% of physicians in our data. We restricted analyses to those who graduated from medical schools in the US, excluding 259 (0.6%) who self reported as graduates of “other medical schools.” For those physicians for whom information on medical school was available, we matched schools to the rankings of the US News & World Report (USNWR) in research and primary care (see supplemental eTable 1). The USNWR has published research rankings of medical schools since 1983, and it added a primary care ranking in 1995.25 USNWR uses four attributes to rank medical schools: reputation, research activity, student selectivity, and faculty resources. The rankings are commonly used as a metric for assessing the quality of medical schools151617 (although other less commonly used ranking schemas exist). Rankings are based on a weighted average of indicators, including peer assessment by school deans, evaluation by residency directors, selectivity of student admission (medical college admission test scores, student grade point averages, and acceptance rate), and faculty-student ratio.26 In addition, research rankings also take into account research activity of the faculty; primary care rankings include a measure of the proportion of graduates entering primary care specialties.

To allow for a non-linear relation we categorized medical schools into groups on the basis of USNWR ranking: 1-10, 11-20, 21-30, 31-40, 41-50, and ≥50 (only the top 50 medical schools are ranked, and therefore, we put unranked schools into the last category). We considered these six categories as the ranking categories. To measure the USNWR ranking of a physician’s medical school during the approximate period of school attendance, we used rankings published in 2002 as opposed to current rankings, and we examined patient outcomes of these physicians in 2011-15. Previous studies have found a high correlation between USNWR school rankings across years27 and relatively stable rankings over time for the top 20 medical schools in primary care rankings.28

Outcome variables

Our outcomes of interest included 30 day mortality, 30 day readmissions, and costs of care. Information on dates of death, including deaths out of hospital, was available in the Medicare Beneficiary Summary Files, which have been verified by death certificate.29 We excluded less than 1% of patients with non-validated death dates. We defined costs of care as total Medicare Part B spending (physician fee-for-service spending, including visits, procedures, and interpretations of tests or images) for each hospital admission, because Part A spending (hospital spending) is largely invariant as it is determined by MS-DRGs.

Attribution of patient outcomes to physicians

Based on prior studies,121222430 we defined the physician responsible for patient outcomes and spending as the physician who billed the most Medicare Part B spending costs during that hospital admission. On average, 51%, 22%, and 11% of total Part B spending was accounted for by the first, second, and third highest spending physicians, respectively. We restricted our analyses to hospital admissions in which the assigned physician was a general internist. For patients transferred to other acute care hospitals (1.2% of hospital admissions), we attributed the multi-hospital episode of care and associated outcomes to the assigned physician of the initial hospital admission.3132

Adjustment variables

We adjusted for patient and physician characteristics and hospital fixed effects. Patient characteristics included age (as a continuous variable, with quadratic and cubic terms to allow for a non-linear relation), sex, race or ethnic group (non-Hispanic white American, non-Hispanic black American, Hispanic American, other), primary diagnosis (indicator variables for MS-DRG), indicators for 27 comorbid conditions (from the Chronic Conditions Data Warehouse developed by the Centers for Medicare and Medicaid Services33), median household income by zip code (in 10ths), an indicator for dual Medicare-Medicaid coverage, day of the week on which the admission occurred, and the indicator variable for year. Physician characteristics included age (as a continuous variable, plus quadratic and cubic terms), sex, credentials (allopathic versus osteopathic training), and patient volume (as a continuous variable, with quadratic and cubic terms). We also adjusted for hospital fixed effects—indicator variables for each hospital, which account for both measured and unmeasured characteristics of hospitals that do not vary over time, including unmeasured differences in patient populations. Therefore, our models effectively compared patient outcomes between physicians who graduated from medical schools of varying USNWR rank, practicing within the same hospital.343536 This method allowed us to circumvent the potential concern that physicians from highly ranked medical school may appear to have better (or worse) patient outcomes because they are differentially employed by hospitals with better support systems or whose patients have, on average, lower severity of illness (or alternatively, worse support systems and patients with, on average, higher severity of illness).

Statistical analysis

We examined the association between the USNWR ranking category of the medical school a physician attended and the physician’s 30 day patient mortality rate using a multivariable linear probability model, adjusting for patient and physician characteristics and hospital fixed effects. We used cluster robust standard errors to account for the possibility that outcomes among patients treated by the same physician may be correlated with each other.37 After fitting the regression model, we calculated adjusted 30 day mortality rates using marginal standardization (also known as predictive margins or margins of responses); for each hospital admission we calculated predicted probabilities of patient mortality with physician medical school ranking fixed at each category and then averaged over the distribution of covariates in our national sample.38 In addition, to test whether mortality rates changed monotonically across USNWR medical school ranking categories, we conducted a trend analysis (P for trend) by refitting the regression model using the ranking categories as a continuous variable.

We then evaluated the relation between the ranking of the medical school a physician attended and 30 day readmission rates and costs of care using a similar method to the analysis of mortality. To estimate these associations, we used multivariable linear probability models adjusting for patient and physician characteristics and hospital fixed effects.

Secondary analyses

We conducted several additional analyses. Firstly, to address the possibility that the USNWR ranking of a physician’s medical school in 2002 may not reflect the ranking at the time the physician attended medical school, we restricted our analysis to physicians who graduated from medical school within four years (from 1998 through 2006) of when the rankings were created (given that the typical duration of medical school education is four years). We also conducted an analysis that used USNWR medical school rankings from 2009 instead of 2002.

Secondly, to address the possibility that physicians who graduated from more highly ranked medical schools may treat patients with greater or lesser unmeasured severity of illness, we repeated our analyses focusing on hospitalists instead of general internists. Hospitalists typically work in scheduled shifts or blocks (eg, one week on and one week off) and in general do not treat patients in the outpatient setting. Therefore, within the same hospital, patients treated by hospitalists may be considered to be plausibly randomised to a particular hospitalist based only on the time of the patient’s admission and the hospitalist’s work schedule.212239 We assessed the validity of this assumption by testing the balance of a broad range of patient characteristics between physicians who graduated from lower ranked versus higher ranked medical schools. We defined hospitalists as general internists who filed at least 90% of their total evaluation-and-management billings in an inpatient setting, a claims based approach that has been previously validated (sensitivity of 84.2%, specificity of 96.5%, and positive predictive value of 88.9%).40 To address the possibility that patients who are admitted multiple times may be assigned to the hospitalist who treated the patient previously, we restricted our analyses to patients’ first admission to a given hospital during the study period.

Thirdly, to evaluate whether our findings were sensitive to how we attributed patients to physicians, we tested two alternative attribution methods: attributing patients to physicians with the largest number of evaluation-and-management claims, and attributing patients to physicians who billed the first evaluation-and-management claim for a given hospital admission (the “admitting physician”).

Fourthly, we used multivariable logistic regression models, instead of multivariable linear probability models, to test whether our findings were sensitive to the model specification for the analyses of binary outcomes (mortality and readmissions). To overcome complete or quasi-complete separation problems (ie, perfect or nearly perfect prediction of the outcome by the model), we combined medical diagnosis related group codes with no outcome event (30 day mortality or readmission) into clinically similar categories.41

Fifthly, because cost data were right skewed we conducted an additional sensitivity analysis using a generalised linear model with a log-link and normal distribution for our cost analyses.42

Sixthly, we investigated whether the association between a physician’s USNWR medical school ranking and subsequent patient outcomes and costs of care was modified by years of experience. We hypothesised that medical school may play a greater role, if any, as a signal of quality for physicians who recently completed residency training as opposed to older physicians for whom any signal of quality owing to medical school may dissipate over time as physician practice norms conform to those of other peers or hospital standards.

Seventhly, to evaluate the influence of hospitals where physicians practice, instead of comparing physicians who graduated from highly ranked versus lower ranked schools within the same hospital, we compared physicians across hospitals, by removing hospital fixed effects from our regression models.

Finally, to address important concerns about the methods used for the USNWR rankings, we repeated the analyses using alternative rankings. Alternative to the primary care ranking, we used the ranking based on social mission score developed by Mullan and colleagues, which is calculated based on the percentages of graduates who work in primary care, graduates who work in health professional shortage areas, and underrepresented minorities.43 Alternative to the research ranking, we used a ranking based on the amount of NIH (National Institute of Health) funding to medical schools. Importantly, our baseline analysis focused on USNWR rankings, rather than these possibly more objective measures of medical school quality, because the key empirical question of interest in this study was whether the commonly used USNWR rankings bear any predictive signal for downstream patient outcomes and costs of care for physicians who graduated from high ranked versus lower ranked schools in USNWR rankings.

Data preparation was conducted using SAS, version 9.4 (SAS Institute), and analyses were performed using Stata, version 14 (Stata Corp).

Patient and public involvement

No patients were involved in setting the research question or the outcome measures, nor were they involved in developing plans for design or implementation of the study. No patients were asked to advise on interpretation or writing up of results. There are no plans to disseminate the results of the research to study participants or the relevant patient community.