Leveraging a Large Language Model to Generate Quality Improvement Feedback for Clinical Notes.

IF 2.7 2区医学 Q4 MEDICAL INFORMATICS

Applied Clinical Informatics Pub Date : 2026-03-01 Epub Date: 2026-04-15 DOI:10.1055/a-2851-0739

Christopher J Kim, Joseph Gelfinbein, Nihan Gencerliler, Nusrat Jahan, Jahnavi Udaikumar, Lauren M Heery, Adam Goodman, Sarah Ng, Joel Attard, Sharmin Asha, Jesse Burk-Rafel, Benedict V Guzman, Katherine A Hochman, Paul A Testa, Jonah Feldman

{"title":"Leveraging a Large Language Model to Generate Quality Improvement Feedback for Clinical Notes.","authors":"Christopher J Kim, Joseph Gelfinbein, Nihan Gencerliler, Nusrat Jahan, Jahnavi Udaikumar, Lauren M Heery, Adam Goodman, Sarah Ng, Joel Attard, Sharmin Asha, Jesse Burk-Rafel, Benedict V Guzman, Katherine A Hochman, Paul A Testa, Jonah Feldman","doi":"10.1055/a-2851-0739","DOIUrl":null,"url":null,"abstract":"Background: Poor documentation quality can significantly affect health care operations, but the feedback process for clinicians to improve clinical notes is time-consuming and often insufficient. Large language models (LLMs) such as Generative Pre-trained Transformer 4 (GPT-4) have the potential to streamline this process.Objectives: This study aimed to determine whether an LLM can generate feedback to improve the medical contingency and discharge planning (MCDP) component of clinical documentation that is non-inferior to feedback by physicians.Methods: A cross-sectional study of GPT-4 feedback and physician feedback on inpatient progress notes was conducted. A random sample of 64 inpatient progress notes identified by the validated artificial intelligence (AI) Audit Tool as having a low likelihood of containing MCDP was included from adult general medicine patients hospitalized at New York University Langone Health (NYULH) in December 2023. Both the GPT-4 model and attending physicians generated feedback on these inpatient progress notes. A/B testing was then conducted on the measures of understandability, usefulness, acceptability, and impartiality. Evaluations employed 5-point Likert scales that were converted to 10-point bidirectional interval scales for interpretability, ranging from -10 (human suggestions significantly better) to +10 (GPT-4 suggestions significantly better), with a non-inferiority threshold set to -1 for the primary endpoint.Results: Sixty-four inpatient progress notes were included, representing 55% female patients with a median age of 73 years. GPT-4 feedback was non-inferior to physician feedback in all measures: Understandability (mean, 1.27, 95% CI: 0.73-1.8, p < 0.001), usefulness (mean, 2.09, 95% CI: 1.27-2.91, p < 0.001), acceptability (mean, 2.07, 95% CI: 1.33-2.81, p < 0.001), and impartiality (mean, -0.20, 95% CI: -0.52 to 0.12, p < 0.001).Conclusion: This study shows that an LLM can be leveraged to generate note quality feedback that is non-inferior to expert clinician feedback.","PeriodicalId":48956,"journal":{"name":"Applied Clinical Informatics","volume":" ","pages":"252-258"},"PeriodicalIF":2.7000,"publicationDate":"2026-03-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC13287944/pdf/","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Applied Clinical Informatics","FirstCategoryId":"3","ListUrlMain":"https://doi.org/10.1055/a-2851-0739","RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"2026/4/15 0:00:00","PubModel":"Epub","JCR":"Q4","JCRName":"MEDICAL INFORMATICS","Score":null,"Total":0}

引用次数: 0

Abstract

Background: Poor documentation quality can significantly affect health care operations, but the feedback process for clinicians to improve clinical notes is time-consuming and often insufficient. Large language models (LLMs) such as Generative Pre-trained Transformer 4 (GPT-4) have the potential to streamline this process.

Objectives: This study aimed to determine whether an LLM can generate feedback to improve the medical contingency and discharge planning (MCDP) component of clinical documentation that is non-inferior to feedback by physicians.

Methods: A cross-sectional study of GPT-4 feedback and physician feedback on inpatient progress notes was conducted. A random sample of 64 inpatient progress notes identified by the validated artificial intelligence (AI) Audit Tool as having a low likelihood of containing MCDP was included from adult general medicine patients hospitalized at New York University Langone Health (NYULH) in December 2023. Both the GPT-4 model and attending physicians generated feedback on these inpatient progress notes. A/B testing was then conducted on the measures of understandability, usefulness, acceptability, and impartiality. Evaluations employed 5-point Likert scales that were converted to 10-point bidirectional interval scales for interpretability, ranging from -10 (human suggestions significantly better) to +10 (GPT-4 suggestions significantly better), with a non-inferiority threshold set to -1 for the primary endpoint.

Results: Sixty-four inpatient progress notes were included, representing 55% female patients with a median age of 73 years. GPT-4 feedback was non-inferior to physician feedback in all measures: Understandability (mean, 1.27, 95% CI: 0.73-1.8, p < 0.001), usefulness (mean, 2.09, 95% CI: 1.27-2.91, p < 0.001), acceptability (mean, 2.07, 95% CI: 1.33-2.81, p < 0.001), and impartiality (mean, -0.20, 95% CI: -0.52 to 0.12, p < 0.001).

Conclusion: This study shows that an LLM can be leveraged to generate note quality feedback that is non-inferior to expert clinician feedback.

查看原文本刊更多论文

利用大型语言模型为临床记录生成质量改进反馈。

背景：较差的文档质量会严重影响医疗保健操作，但临床医生改进临床记录的反馈过程既耗时又不充分。大型语言模型（llm），如生成预训练变压器4 （GPT-4）有可能简化这一过程。目的：确定LLM是否可以产生反馈，以改善临床文件的医疗应急和出院计划（MCDP）部分，而不逊色于医生的反馈。方法：采用横断面研究GPT-4反馈与住院病历医师反馈。2023年12月，从纽约大学朗格尼健康中心（NYULH）住院的成人普通医学患者中随机抽取64份住院病历，经验证的人工智能审计工具确定其含有MCDP的可能性较低。GPT-4模型和主治医生都会对这些住院病人的进展记录进行反馈。然后根据可理解性、有用性、可接受性和公正性进行A/B测试。评估采用5点李克特量表，转换为10点双向间隔量表的可解释性，范围从-10（人类建议显着更好）到+10 （GPT-4建议显着更好），主要终点的非劣效阈值设置为-1。结果：纳入64份住院病程记录，其中55%为女性患者，中位年龄为73岁。GPT-4反馈在所有指标上均不逊于医生反馈：可理解性（平均1.27,95% CI 0.73 ~ 1.8, P < 0.001）、有用性（平均2.09,95% CI 1.27 ~ 2.91, P < 0.001）、可接受性（平均2.07,95% CI 1.33 ~ 2.81, P < 0.001）和公正性（平均-0.20,95% CI -0.52 ~ 0.12, P < 0.001）。结论：本研究表明，LLM可以用来产生笔记质量反馈，而不逊色于专家临床医生的反馈。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

Applied Clinical Informatics MEDICAL INFORMATICS-

CiteScore

4.60

自引率

24.10%

发文量

132

期刊介绍： ACI is the third Schattauer journal dealing with biomedical and health informatics. It perfectly complements our other journals Öffnet internen Link im aktuellen FensterMethods of Information in Medicine and the Öffnet internen Link im aktuellen FensterYearbook of Medical Informatics. The Yearbook of Medical Informatics being the “Milestone” or state-of-the-art journal and Methods of Information in Medicine being the “Science and Research” journal of IMIA, ACI intends to be the “Practical” journal of IMIA.