Evaluating the 2023 Standardized Letter of Recommendation Form in Applications to Orthopaedic Surgery Residency.

Background: Successfully matching into orthopaedic surgery residency is a challenging endeavor due to the limited number of available positions. Implementation of the Standard Letter of Recommendation (SLOR) in 2017 was intended to allow better comparisons between applicants; however, the form suffered from notable rank inflation ultimately requiring introduction of an updated SLOR for the 2023 to 2024 application cycle. The purpose of this study was to evaluate whether the 2023 SLOR improves on deficiencies of the 2017 SLOR.

Methods: Applications submitted to our institution during the 2023 to 2024 orthopaedic surgery residency application cycle were analyzed. Applicant objective metrics were collected, including US Medical Licensing Exam scores, clerkship grades, AΩA status, and research productivity. Scores for each domain of the 2023 SLOR forms were recorded. Mean scores were calculated for each question, and cumulative scores per SLOR were determined. Intraclass correlation coefficients were determined per applicant and evaluator. The Spearman rank correlation coefficient was calculated for comparison of SLOR cumulative scores and objective metrics.

Results: Three hundred sixty-eight applications were reviewed, and 743 2023 SLORs were submitted in conjunction with these applications. Most of the 2023 SLORs (n = 399, 53.70%) had cumulative scores placing applicants in at least the 90th percentile. Furthermore, 267 evaluators (72.36%) consistently gave scores of 99s and 10s for each question. Intraclass correlation coefficients were rated as fair to moderate agreement between evaluators for each applicant. Objective applicant metrics did not meaningfully correlate with SLOR cumulative scores.

Discussion: The 2023 SLOR demonstrated notable rank inflation with only 22.7% of applicants receiving a score of less than 7 on any domain. By contrast, 69.9% of applicants received a score of "rare, perfect gem" on at least one domain of the form. We encourage the American Orthopaedic Association Council of Orthopaedic Residency Director to continue to improve the form and provide guidance to residency program leadership on proper use of the grading system.