Henry Avetisian, Marc A Abdou, William J Karakash, Daniel Rusu, Aidan Lindgren, Matthew C Gallo, Joseph Patterson, Jeffrey C Wang, Raymond J Hah, Ram K Alluri
{"title":"Indistinguishable Narratives: A Blinded Comparison of AI- Versus Human-Written Personal Statements in Application to Orthopaedic Surgery Residency.","authors":"Henry Avetisian, Marc A Abdou, William J Karakash, Daniel Rusu, Aidan Lindgren, Matthew C Gallo, Joseph Patterson, Jeffrey C Wang, Raymond J Hah, Ram K Alluri","doi":"10.4300/JGME-D-25-00418.1","DOIUrl":null,"url":null,"abstract":"<p><strong>Background: </strong>As residency program applications become more homogenized due to pass-fail United States Medical Licensing Examination Step 1 scores, pre-clinical grading, and standardized letters of recommendation, personal statements remain a key subjective tool to assess applicant fit. With the rise of large language models like ChatGPT, applicants may increasingly use AI to draft these statements.</p><p><strong>Objective: </strong>To determine whether ChatGPT can generate orthopaedic surgery personal statements that are indistinguishable from and comparable in quality to human-written statements.</p><p><strong>Methods: </strong>In 2025 at a single urban academic university hospital, ChatGPT-4.0 was given a standardized prompt to generate 5 personal statements across distinct themes. Five de-identified statements from applicants who interviewed or matched at our institution from 2020 to 2021 were randomly selected. The 10 essays were randomized and independently reviewed by 8 blinded assessors, including 6 attending physicians and 2 resident physicians, all of whom serve on the resident admissions committee. Using a 9-question REDCap survey, evaluators rated each statement on readability, originality, authenticity, and overall quality on a 100-point scale; guessed whether each was AI- or human-generated (Yes/No); and indicated whether they would offer an interview (Yes/No). Chi-square tests were used for analysis.</p><p><strong>Results: </strong>There were no significant differences between ChatGPT and human-written statements in readability (<i>P</i>=.63), originality (<i>P</i>=.48), authenticity (<i>P</i>=.79), or overall quality (<i>P</i>=.84). Assessors could not reliably identify AI- versus human-written statements (19 of 40 vs 22 of 39, <i>P</i>=.43) and interview offers were similarly comparable (29 of 40 vs 29 of 39, <i>P</i>=.85).</p><p><strong>Conclusions: </strong>ChatGPT can produce orthopaedic surgery personal statements that are both indistinguishable from and comparable in quality to human-written ones.</p>","PeriodicalId":37886,"journal":{"name":"Journal of graduate medical education","volume":"18 2","pages":"160-163"},"PeriodicalIF":0.0000,"publicationDate":"2026-04-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC13086132/pdf/","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Journal of graduate medical education","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/10.4300/JGME-D-25-00418.1","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"2026/4/15 0:00:00","PubModel":"Epub","JCR":"","JCRName":"","Score":null,"Total":0}
引用次数: 0
Abstract
Background: As residency program applications become more homogenized due to pass-fail United States Medical Licensing Examination Step 1 scores, pre-clinical grading, and standardized letters of recommendation, personal statements remain a key subjective tool to assess applicant fit. With the rise of large language models like ChatGPT, applicants may increasingly use AI to draft these statements.
Objective: To determine whether ChatGPT can generate orthopaedic surgery personal statements that are indistinguishable from and comparable in quality to human-written statements.
Methods: In 2025 at a single urban academic university hospital, ChatGPT-4.0 was given a standardized prompt to generate 5 personal statements across distinct themes. Five de-identified statements from applicants who interviewed or matched at our institution from 2020 to 2021 were randomly selected. The 10 essays were randomized and independently reviewed by 8 blinded assessors, including 6 attending physicians and 2 resident physicians, all of whom serve on the resident admissions committee. Using a 9-question REDCap survey, evaluators rated each statement on readability, originality, authenticity, and overall quality on a 100-point scale; guessed whether each was AI- or human-generated (Yes/No); and indicated whether they would offer an interview (Yes/No). Chi-square tests were used for analysis.
Results: There were no significant differences between ChatGPT and human-written statements in readability (P=.63), originality (P=.48), authenticity (P=.79), or overall quality (P=.84). Assessors could not reliably identify AI- versus human-written statements (19 of 40 vs 22 of 39, P=.43) and interview offers were similarly comparable (29 of 40 vs 29 of 39, P=.85).
Conclusions: ChatGPT can produce orthopaedic surgery personal statements that are both indistinguishable from and comparable in quality to human-written ones.
期刊介绍：
- Be the leading peer-reviewed journal in graduate medical education; - Promote scholarship and enhance the quality of research in the field; - Disseminate evidence-based approaches for teaching, assessment, and improving the learning environment; and - Generate new knowledge that enhances graduates'' ability to provide high-quality, cost-effective care.
求助内容：
应助结果提醒方式：
京公网安备 11010802042870号
