Comparative analysis of human-generated versus Artificial Intelligence-drafted summary paragraphs for medical student performance evaluations.

Abstract

Purpose: This study evaluated the efficiency and effectiveness of using Generative Artificial Intelligence (GenAI) to draft Medical Student Performance Evaluation (MSPE) summary paragraphs for medical students.

Materials and methods: Evaluations on the pediatrics clerkship were used to develop MSPE summary paragraphs. Time to completion was noted for paragraphs drafted by GenAI, created using Microsoft 365 Copilot, and compared to human-generated. Undergraduate Medical Education (UME) leaders were recruited to evaluate 10 randomized pairs of paragraphs through a blinded survey.

Results: Copilot-drafted paragraphs required significantly less time to completion compared to human-generated paragraphs (median 6 vs. 12.5 min, p = 0.002). UME leaders showed no significant preference and were unable to consistently identify Copilot vs human authorship. When stratified by perception of authorship, human-generated paragraphs were significantly less likely to be preferred if they were perceived as being Copilot-drafted than if they were perceived as being human-generated (p = 0.017), suggesting an element of anti-AI bias. Competencies were highlighted to a similar degree, and Copilot-drafted paragraphs were perceived as having significantly less biased language by both UME leaders (p = 0.004) and an independent analysis using a validated gender bias calculator (p = 0.029).

Conclusions: Copilot-drafted MSPE summaries are efficient, comparable in quality, and may reduce the introduction of bias.