Contemporary Step 1 Predictive Methods Across 12 US MD-Granting Medical Schools.

Abstract

Since the United States Medical Licensing Examination (USMLE) Step 1 scoring transitioned from reporting numerical scores to reporting only pass or fail in 2022, the national first-time Step 1 failure rate has increased. In this shifting educational landscape, students require more detailed guidance to reach their Step 1 goals. Ideally, this would be based on data-driven methods for identifying students at-risk for failing Step 1, leveraging data to support student success. However, developing methods to identify at-risk students early enough for personalized interventions to be put in place can be difficult as every medical school has unique structures and assessment methods. To bridge this gap, the current study compared 12 US MD-granting medical schools' Step 1 predictive models used to identify students at-risk for Step 1 failure. Each institution reported their Step 1 predictive method via virtual meetings and a brief survey. We assessed each model in terms of sensitivity, specificity, and timing of risk identification with respect to when students take the Step 1 exam. Six institutions identified students at-risk through categories of risk (scoring below a threshold, having a combination of risk factors). Three institutions identified risk using multiple regression models. Two institutions used a combination of categorical risk assessment and regression methods. One institution used growth mixture modeling to track student performance. Across the 12 participating institutions, performance on a National Board of Medical Examiners (NBME), Comprehensive Basic Science Examination (CBSE), or Comprehensive Basic Science Self-Assessment (CBSSA) exam were the most meaningful predictors. Most institutions identified at-risk students at the onset of a dedicated Step 1 study period. Although each predictive method had strengths, each also had limitations in identifying students at-risk for failing Step 1. Schools had varying thresholds for identifying and/or intervening with at-risk students. All institutions noted they were continually striving to improve their predictive model performance to enhance student learning outcomes. Overall, the current study describes and compares 12 contemporary Step 1 predictive methods, noting key features and recommendations for the development, implementation, and refinement of Step 1 predictive methodology.