How Well Do Accreditation Council for Graduate Medical Education Milestones Track Readiness for Anesthesiology Certifying Examinations in a National Resident Cohort?

BACKGROUND Milestones evaluation is mandated by the Accreditation Council for Graduate Medical Education (ACGME) to help training programs measure residents' progress toward competency and identify specific areas for trainee improvement. Data from training programs raised concerns that Milestones ratings may reflect the year of training more than resident progress toward competency. We examined the relationship between residents' Milestones ratings toward the end of residency training and their performance on the American Board of Anesthesiology (ABA) examinations, widely considered as the gold standard of competency. METHODS We compared Milestones 2.0 ratings and board scores of all anesthesiologists who completed an ACGME-accredited residency program between July 2021 and June 2022 (AY22) and had their first-time ABA ADVANCED Examination (written), Standardized Oral Examination (SOE) and Objective Structured Clinical Examination (OSCE) performance available by 2023. We first assessed the correlation between the average rating achieved across all 23 Milestones during the last 6 months of residency training and the Z-scores of these three examinations among first-time takers. Then, we evaluated the correlations between 9 specific Milestones and their conceptually related domains tested by the ADVANCED, the SOE, and the OSCE; we calculated Pearson and polychoric correlation coefficients for continuous and ordinal data, respectively. RESULTS All 23 Milestones 2.0 AY22 ratings were available for 1849 Post Graduate Year (PGY)-4, Clinical Anesthesia Year 3 (CA-3) residents. These were matched to 1799 first-time ADVANCED and 1383 first-time SOE and OSCE takers. The average ACGME Milestones ratings across all competencies were significantly correlated with examination Z-scores (all P < .001)-the ADVANCED (r = 0.135 [95% confidence interval {CI}, 0.089-0.180], the SOE (r = 0.117 [0.065-0.169]), and the OSCE (r = 0.112 [0.060-0.164]). For the domain-specific comparisons, scores on the ADVANCED Examination correlated modestly with the Medical Knowledge Milestone domain (r = 0.289, P < .001), but there were no statistically significant associations between SOE task ratings and their related Milestones domains (ρ = 0.027 to 0.091, P = .31 to 0.81). In comparisons of similar domains evaluated by OSCE stations and the Milestones, 2 were statistically significantly correlated with a weak magnitude (Interpretation of Monitors and Echocardiograms [ρ = 0.093, P = .031] and Ethical Issues [ρ = 0.049, P = .003]) while 2 others were not statistically significant (Application of Ultrasonography [ρ = 0.052, P = .775] and Communication with other Professionals [ρ = 0.052, P = .086]). CONCLUSIONS There was a modest correlation between the last Medical Knowledge Milestone achieved and the ADVANCED Examination. However, the weak correlations between residency Milestones and the SOE or OSCE performance suggest that the Milestones system, as currently implemented by anesthesiology training programs, does not predict certifying examination performance.