Beyond admissions: Using situational judgment test scores to predict learner performance throughout undergraduate medical education.

Abstract

Introduction: Situational judgment tests (SJTs) have been introduced in medical school admissions across multiple national contexts over the past three decades. However, ongoing research is needed to understand SJTs' role in predicting outcomes across undergraduate medical education (UME), particularly in the context of institution-specific approaches to medical student assessment. This study investigated the role of the Casper SJT for understanding student performance in professionalism, communication and self-awareness at three key timepoints in admissions and UME: interview day scores, the preclinical phase, and the clinical phase.

Method: We conducted a retrospective exploratory analysis of associations between the Casper score, interview day scores, and medical student assessment data from 508 students at the University of Michigan Medical School. Using sequential logistic regression models, we linked student data across admissions and UME phases, using milestone assessments and professionalism referrals as outcome variables. To better understand the role of the composite SJT score in understanding construct-specific UME assessments, we mapped Casper constructs to institution-specific assessments of professionalism, communication skills, and self-awareness.

Results: When modeling the relationship between Casper SJT scores and communication competency domains in UME, we found that including the SJT score in the communication and self-awareness domains models modestly improved the model outcomes. However, our modeling analysis demonstrated limited utility for predicting professionalism lapses during medical school using SJT scores.

Discussion: As SJTs are adopted in selection processes, it is crucial to investigate whether SJT scores predict future student success in the context of real-world institutional variation in applicant pool, admissions processes, and UME assessment programs. Our analysis contributes to efforts to establish the effectiveness of the Casper SJT for predicting temporally distant and competency-specific outcomes by demonstrating the utility of sequential logistic regression modeling methods. Our discussion critically considers the use of composite SJT scores to predict for construct-specific UME assessments.