Comparisons of Validity of the New and Prior MCAT Exams in Predicting Performances on Steps 1, 2, and 3 of the United States Medical Licensing Examinations.

Abstract

This study examined the validity of the new MCAT exam (administered since 2015) for predicting medical students' performance on United States Medical Licensing Examinations (USMLE) and compared the findings with those of the prior MCAT version. Participants comprised two samples of students who entered Sidney Kimmel Medical College at Thomas Jefferson University between 2012 and 2020. One sample included 1,111 students (559 men, 552 women) with new MCAT scores who matriculated between 2016 and 2020, and the other comprised 1,312 students (668 men, 644 women) with prior MCAT scores who matriculated between 2012 and 2015. We used students' MCAT scores as predictors of performance on Steps 1, 2, and 3 of the USMLE. Bivariate correlations and path analysis were used for statistical analyses. Path analysis showed new MCAT total scores resulted in R² values of 0.14, 0.11, and 0.16 for predicting performance on Steps 1, 2, and 3 of the USMLE, respectively. The new MCAT total scores demonstrated levels of validity comparable to the prior MCAT for predicting students' performances on the criterion measures. Additional path analyses showed an impact of gender on the predictive validities for some section scores of the new (but not prior) MCAT exam. Replication of this study is recommended in other medical schools to examine generalizability of our findings regarding predictive validities of section scores of the new MCAT exam, particularly regarding gender and section.