Leveraging Large Language Models to Summarize Data and Free-Text Comments From Resident Assessments.

Abstract

Background: Summarizing resident assessments for Clinical Competency Committee or mentoring meetings is challenging. This especially applies to raters' free-text comments, which often provide valuable feedback.

Objective: To assess feasibility and accuracy of using a large language model (LLM) to label free-text comments as "Strengths" or "Areas for Improvement" from end-of-rotation assessments of residents.

Methods: We performed a mixed-methods study of residents' end-of-rotation assessments completed between July 1, 2024 and June 30, 2025. Following data de-identification, we used a private (paid) LLM account (ChatGPT-4o) to compile numerical and label free-text ("Strengths," "Areas for Improvement") feedback. We assessed feasibility using time spent de-identifying data, time for LLMs to return labeled free-text comments, and LLM recall (ability to accurately extract/label free-text feedback). We performed 2 confirmatory trials (free LLMs: MS CoPilot, ChatGPT-4) to ascertain reproducibility. Two subject matter experts labeled free-text comments to determine LLM labeling accuracy. The study was performed in 2025.

Results: Our dataset represented 277 individual assessments for 25 unique residents. Feasibility data demonstrated data de-identification time of ∼60 minutes and LLM processing/labeling times of <7 minutes per LLM. All LLMs consistently compiled all free-text entries, accurately labeling 59.7% of free-text comments. Human analysis of comments (476) identified 388 (81.5%) "Strengths" and 88 (18.5%) "Areas for Improvement." Of these, LLM recall was 67.8% for "Strengths" and 23.9% for "Areas for Improvement."

Conclusions: We designed prompts that feasibly labeled end-of-rotation assessment free-text comments across 3 LLMs with recall for "Strengths" 2.5 times higher than recall for "Areas for Improvement."