Miaomiao Cheng, Huiru Wang, Qi Zhang, Jun Qin, Lei Gong, Ling Guo, Xiaoyan Xiao
{"title":"Large Language Model-Assisted Discharge Instructions for Diabetic Kidney Disease: Comparative Evaluation Study.","authors":"Miaomiao Cheng, Huiru Wang, Qi Zhang, Jun Qin, Lei Gong, Ling Guo, Xiaoyan Xiao","doi":"10.2196/100792","DOIUrl":null,"url":null,"abstract":"<p><strong>Background: </strong>Discharge instructions are an important component of the transition from hospital to home for patients with diabetic kidney disease (DKD), whose care often involves multiple comorbidities, medications, and stage-specific management needs. However, producing complete, accurate, clear, and personalized instructions can be challenging for junior physicians.</p><p><strong>Objective: </strong>This study aimed to evaluate whether large language model (LLM) assistance improves the quality of DKD discharge instructions produced by junior physicians and whether its benefits vary according to chronic kidney disease (CKD) stage.</p><p><strong>Methods: </strong>The study included 101 hospitalized patients with DKD. Discharge instructions for each patient were initially drafted by a group of junior residents (resident group); subsequently, another group of residents at the same level, with the assistance of DeepSeek-V3.2 (AI-assisted group), generated a separate set of discharge instructions for the same 101 patients, resulting in a total of 202 documents. Four clinical experts, including 2 nephrologists and 2 endocrinologists, independently scored both versions of the discharge instructions using a 5-point Likert scale across 5 dimensions: completeness, accuracy, safety, clarity, and personalization. Interrater reliability was assessed using the intraclass correlation coefficient (ICC).</p><p><strong>Results: </strong>Interrater reliability was excellent in both the resident group (ICC=0.817, 95% CI 0.763-0.864; <i>P</i><.001) and the AI-assisted group (ICC=0.860, 95% CI 0.816-0.897; <i>P</i><.001). Compared with resident-only instructions, AI-assisted instructions received significantly higher scores across 5 evaluation dimensions and the overall evaluation (all Bonferroni-adjusted <i>P</i><.001), with large effect sizes (<i>r</i>=0.741-0.927). In the resident-only group, CKD stage was negatively correlated with accuracy, safety, clarity, and overall quality (Spearman ρ=-0.33 to -0.37; all <i>P</i><.001), whereas no significant correlations were observed for most dimensions in the AI-assisted group (|ρ|<0.15; <i>P</i>>.05). The proportion of cases achieving a ≥1-point improvement in overall quality was 99% (100/101; 95% CI 94.6%-100.0%).</p><p><strong>Conclusions: </strong>LLM assistance improved the completeness, clarity, and personalization of DKD discharge instructions generated by junior physicians, with advantages greatest in complex cases. A human-AI collaborative model is a promising approach to reduce documentation burden.</p>","PeriodicalId":56334,"journal":{"name":"JMIR Medical Informatics","volume":"14 ","pages":"e100792"},"PeriodicalIF":5.0000,"publicationDate":"2026-11-13","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC13766588/pdf/","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"JMIR Medical Informatics","FirstCategoryId":"3","ListUrlMain":"https://doi.org/10.2196/100792","RegionNum":3,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q2","JCRName":"MEDICAL INFORMATICS","Score":null,"Total":0}
引用次数: 0
Abstract
Background: Discharge instructions are an important component of the transition from hospital to home for patients with diabetic kidney disease (DKD), whose care often involves multiple comorbidities, medications, and stage-specific management needs. However, producing complete, accurate, clear, and personalized instructions can be challenging for junior physicians.
Objective: This study aimed to evaluate whether large language model (LLM) assistance improves the quality of DKD discharge instructions produced by junior physicians and whether its benefits vary according to chronic kidney disease (CKD) stage.
Methods: The study included 101 hospitalized patients with DKD. Discharge instructions for each patient were initially drafted by a group of junior residents (resident group); subsequently, another group of residents at the same level, with the assistance of DeepSeek-V3.2 (AI-assisted group), generated a separate set of discharge instructions for the same 101 patients, resulting in a total of 202 documents. Four clinical experts, including 2 nephrologists and 2 endocrinologists, independently scored both versions of the discharge instructions using a 5-point Likert scale across 5 dimensions: completeness, accuracy, safety, clarity, and personalization. Interrater reliability was assessed using the intraclass correlation coefficient (ICC).
Results: Interrater reliability was excellent in both the resident group (ICC=0.817, 95% CI 0.763-0.864; P<.001) and the AI-assisted group (ICC=0.860, 95% CI 0.816-0.897; P<.001). Compared with resident-only instructions, AI-assisted instructions received significantly higher scores across 5 evaluation dimensions and the overall evaluation (all Bonferroni-adjusted P<.001), with large effect sizes (r=0.741-0.927). In the resident-only group, CKD stage was negatively correlated with accuracy, safety, clarity, and overall quality (Spearman ρ=-0.33 to -0.37; all P<.001), whereas no significant correlations were observed for most dimensions in the AI-assisted group (|ρ|<0.15; P>.05). The proportion of cases achieving a ≥1-point improvement in overall quality was 99% (100/101; 95% CI 94.6%-100.0%).
Conclusions: LLM assistance improved the completeness, clarity, and personalization of DKD discharge instructions generated by junior physicians, with advantages greatest in complex cases. A human-AI collaborative model is a promising approach to reduce documentation burden.
背景:出院指示是糖尿病肾病(DKD)患者从医院转到家庭的重要组成部分,其护理通常涉及多种合并症、药物治疗和阶段特异性管理需求。然而,制定完整、准确、清晰和个性化的指导对初级医生来说是一个挑战。目的:本研究旨在评估大语言模型(LLM)辅助是否能提高初级医生制作的DKD出院说明书的质量,以及其益处是否因慢性肾脏疾病(CKD)分期而异。方法:纳入101例住院DKD患者。每位患者的出院指示最初由一组初级住院医师(住院医师组)起草;随后,同一级别的另一组住院医生在DeepSeek-V3.2(人工智能辅助组)的帮助下,为同样的101名患者生成了一套单独的出院说明,共计202份文件。4名临床专家,包括2名肾病学家和2名内分泌学家,分别对两种版本的出院说明进行了独立评分,采用李克特5分量表,分为5个维度:完整性、准确性、安全性、清晰度和个性化。用类内相关系数(ICC)评估组间信度。结果:两组居民的量表间信度均极好(ICC=0.817, 95% CI 0.763-0.864; PPPr=0.741-0.927)。在住院治疗组,CKD分期与准确性、安全性、清晰度和整体质量呈负相关(Spearman ρ=-0.33至-0.37;所有PP ρ均为0.05)。总体质量改善≥1点的病例比例为99% (100/101;95% CI 94.6%-100.0%)。结论:LLM辅助提高了初级医生生成的DKD出院指示的完整性、清晰度和个性化,在复杂病例中优势最大。人类-人工智能协作模型是一种很有前途的减少文档负担的方法。
期刊介绍:
JMIR Medical Informatics (JMI, ISSN 2291-9694) is a top-rated, tier A journal which focuses on clinical informatics, big data in health and health care, decision support for health professionals, electronic health records, ehealth infrastructures and implementation. It has a focus on applied, translational research, with a broad readership including clinicians, CIOs, engineers, industry and health informatics professionals.
Published by JMIR Publications, publisher of the Journal of Medical Internet Research (JMIR), the leading eHealth/mHealth journal (Impact Factor 2016: 5.175), JMIR Med Inform has a slightly different scope (emphasizing more on applications for clinicians and health professionals rather than consumers/citizens, which is the focus of JMIR), publishes even faster, and also allows papers which are more technical or more formative than what would be published in the Journal of Medical Internet Research.