Underrepresented in Medicine and International Graduate Bias in the CASPer Situational Judgment Test for Ophthalmology Residency Applicants in 2022-2023.

Purpose: Situational Judgment Tests (SJTs) are increasingly utilized in medical trainee recruitment decisions yet little is known of their potential biases. This study was designed to evaluate the Computer-Based Assessment for Sampling Personal Characteristics (CASPer) SJT scores of ophthalmology residency applicants and compare these to Step 1 and Step 2 scores in different demographic groups.

Methods: All applicants to the ophthalmology residency match completed CASPer during the 2022-2023 recruitment cycle. Applicant scores were cross-referenced within the SF Match Residency Application System to collect data on applicant gender, self-identified Under-Represented in Medicine (URiM) status, and International Medical Graduate (IMG) status, as well as Step 1 and Step 2 scores.

Results: A total of 760 of 765 (99.3%) of applicants with CASPer scores were identified in SF Match, 41% of which were female, 15% URiM, and 11% IMG. Males scored slightly higher on all three tests, with the mean difference in Step 1 (4.3) reaching a statistical significance (Cohen's d = 0.29, p < 0.001). When comparing URiM and IMG applicants, these groups scored significantly lower than their counterparts on all three tests (p all <0.001), with the magnitude of mean difference greater for CASPer than Step 1 or 2. There were no differences between genders in CASPer percentile scores; however, URiM applicants performed on average 17% and IMG applicants 32% lower than their counterparts.

Conclusions: The CASPer SJT demonstrated significant bias against URiM and IMG ophthalmology residency applicants. This bias was even greater than Step 1 and Step 2 performance, suggesting the continued need for caution when utilizing any standardized testing in recruitment decisions.