Are Physicians' USMLE Scores Associated With Patient Mortality? A Multilevel Modeling Approach to USMLE Predictive Validity Evidence

Abstract

Background and Aims

Validity evidence for high-stakes licensing examinations, particularly the association between exam scores and clinical practice outcomes, is essential for supporting valid score-based inferences. However, a gap existed in understanding how variability in patient in-hospital mortality is partitioned between physicians and hospitals and how physicians' USMLE composite scores are associated with patient mortality using multilevel modeling. Thus, we examined variability in patient in-hospital mortality between physicians and hospitals and investigated the association between physicians USMLE composite scores and patient mortality, controlling for patient, physician, and hospital characteristics.

Methods

Hospitalization data were sourced from the American Medical Association, USMLE program, and Pennsylvania Health Care Cost Containment Council. Deidentified data included 150,907 patients nested within 1744 physicians in 170 hospitals. Multilevel logistic regression, including a random intercept and six conditional hierarchical models, was used to account for the nested structure and control for covariates. The seven fitted models were compared descriptively and inferentially.

Results

Approximately 5% and 12% of the variability in patient mortality was attributable to physicians and hospitals, respectively. A one standard deviation increase in the USMLE composite score significantly predicted a 6% reduction in patient mortality (odds ratio 0.94; 95% CI = 0.89, 0.99), controlling for patient, physician, and hospital covariates.

Conclusion

Results highlight variability in mortality attributable to physicians and hospitals and underscore the predictive value of USMLE scores. Higher USMLE composite scores were associated with a lower likelihood of patient in-hospital mortality, providing additional validity evidence for the exam and supporting its use for licensure decisions.