Artificial Intelligence-Generated Patient Education Materials for Helicobacter pylori Infection: A Comparative Analysis

IF 4.3 2区医学 Q1 GASTROENTEROLOGY & HEPATOLOGY

Helicobacter Pub Date : 2024-08-04 DOI:10.1111/hel.13115

Shuyan Zeng, Qingzhou Kong, Xiaoqi Wu, Tian Ma, Limei Wang, Leiqi Xu, Guanjun Kou, Mingming Zhang, Xiaoyun Yang, Xiuli Zuo, Yueyue Li, Yanqing Li

{"title":"Artificial Intelligence-Generated Patient Education Materials for Helicobacter pylori Infection: A Comparative Analysis","authors":"Shuyan Zeng, Qingzhou Kong, Xiaoqi Wu, Tian Ma, Limei Wang, Leiqi Xu, Guanjun Kou, Mingming Zhang, Xiaoyun Yang, Xiuli Zuo, Yueyue Li, Yanqing Li","doi":"10.1111/hel.13115","DOIUrl":null,"url":null,"abstract":"<div>\n \n \n <section>\n \n <h3> Background</h3>\n \n <p>Patient education contributes to improve public awareness of <i>Helicobacter pylori</i>. Large language models (LLMs) offer opportunities to revolutionize patient education transformatively. This study aimed to assess the quality of patient educational materials (PEMs) generated by LLMs and compared with physician sourced.</p>\n </section>\n \n <section>\n \n <h3> Materials and Methods</h3>\n \n <p>Unified instruction about composing a PEM about <i>H. pylori</i> at a sixth-grade reading level in both English and Chinese were given to physician and five LLMs (Bing Copilot, Claude 3 Opus, Gemini Pro, ChatGPT-4, and ERNIE Bot 4.0). The assessments of the completeness and comprehensibility of the Chinese PEMs were conducted by five gastroenterologists and 50 patients according to three-point Likert scale. Gastroenterologists were asked to evaluate both English and Chinese PEMs and determine the accuracy and safety. The accuracy was assessed by six-point Likert scale. The minimum acceptable scores were 4, 2, and 2 for accuracy, completeness, and comprehensibility, respectively. The Flesch–Kincaid and Simple Measure of Gobbledygook scoring systems were employed as readability assessment tools.</p>\n </section>\n \n <section>\n \n <h3> Results</h3>\n \n <p>Accuracy and comprehensibility were acceptable for English PEMs of all sources, while completence was not satisfactory. Physician-sourced PEM had the highest accuracy mean score of 5.60 and LLM-generated English PEMs ranged from 4.00 to 5.40. The completeness score was comparable between physician-sourced PEM and LLM-generated PEMs in English. Chinese PEMs from LLMs proned to have lower score in accuracy and completeness assessment than English PEMs. The mean score for completeness of five LLM-generated Chinese PEMs was 1.82–2.70 in patients' perspective, which was higher than gastroenterologists' assessment. Comprehensibility was satisfactory for all PEMs. No PEM met the recommended sixth-grade reading level.</p>\n </section>\n \n <section>\n \n <h3> Conclusion</h3>\n \n <p>LLMs have potential in assisting patient education. The accuracy and comprehensibility of LLM-generated PEMs were acceptable, but further optimization on improving completeness and accounting for a variety of linguistic contexts are essential for enhancing the feasibility.</p>\n </section>\n </div>","PeriodicalId":13223,"journal":{"name":"Helicobacter","volume":"29 4","pages":""},"PeriodicalIF":4.3000,"publicationDate":"2024-08-04","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Helicobacter","FirstCategoryId":"3","ListUrlMain":"https://onlinelibrary.wiley.com/doi/10.1111/hel.13115","RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"GASTROENTEROLOGY & HEPATOLOGY","Score":null,"Total":0}

引用次数: 0

Abstract

Background

Patient education contributes to improve public awareness of Helicobacter pylori. Large language models (LLMs) offer opportunities to revolutionize patient education transformatively. This study aimed to assess the quality of patient educational materials (PEMs) generated by LLMs and compared with physician sourced.

Materials and Methods

Unified instruction about composing a PEM about H. pylori at a sixth-grade reading level in both English and Chinese were given to physician and five LLMs (Bing Copilot, Claude 3 Opus, Gemini Pro, ChatGPT-4, and ERNIE Bot 4.0). The assessments of the completeness and comprehensibility of the Chinese PEMs were conducted by five gastroenterologists and 50 patients according to three-point Likert scale. Gastroenterologists were asked to evaluate both English and Chinese PEMs and determine the accuracy and safety. The accuracy was assessed by six-point Likert scale. The minimum acceptable scores were 4, 2, and 2 for accuracy, completeness, and comprehensibility, respectively. The Flesch–Kincaid and Simple Measure of Gobbledygook scoring systems were employed as readability assessment tools.

Results

Accuracy and comprehensibility were acceptable for English PEMs of all sources, while completence was not satisfactory. Physician-sourced PEM had the highest accuracy mean score of 5.60 and LLM-generated English PEMs ranged from 4.00 to 5.40. The completeness score was comparable between physician-sourced PEM and LLM-generated PEMs in English. Chinese PEMs from LLMs proned to have lower score in accuracy and completeness assessment than English PEMs. The mean score for completeness of five LLM-generated Chinese PEMs was 1.82–2.70 in patients' perspective, which was higher than gastroenterologists' assessment. Comprehensibility was satisfactory for all PEMs. No PEM met the recommended sixth-grade reading level.

Conclusion

LLMs have potential in assisting patient education. The accuracy and comprehensibility of LLM-generated PEMs were acceptable, but further optimization on improving completeness and accounting for a variety of linguistic contexts are essential for enhancing the feasibility.

查看原文本刊更多论文

人工智能生成的幽门螺旋杆菌感染患者教育材料：比较分析。

背景：患者教育有助于提高公众对幽门螺旋杆菌的认识。大语言模型（LLM）为患者教育的变革提供了机会。本研究旨在评估大语言模型生成的患者教育材料（PEM）的质量，并与医生提供的材料进行比较：研究人员向医生和五种LLM（Bing Copilot、Claude 3 Opus、Gemini Pro、ChatGPT-4和ERNIE Bot 4.0）提供了统一的指导，让他们以六年级的阅读水平用中英文撰写关于幽门螺杆菌的PEM。五位胃肠病学专家和 50 位患者按照李克特三点量表对中文 PEM 的完整性和可理解性进行了评估。胃肠病专家被要求对中英文 PEM 进行评估，并确定其准确性和安全性。准确性采用六点李克特量表进行评估。准确性、完整性和可理解性的最低可接受分数分别为 4 分、2 分和 2 分。采用 Flesch-Kincaid 和 Simple Measure of Gobbledygook 评分系统作为可读性评估工具：结果：所有来源的英文 PEM 的准确性和可理解性均可接受，而完整性则不尽人意。由医生提供的 PEM 的准确性平均得分最高，为 5.60 分，而由 LLM 生成的英文 PEM 的准确性平均得分在 4.00 到 5.40 之间。医生来源的 PEM 和 LLM 生成的英文 PEM 的完整性得分相当。在准确性和完整性评估中，来自法律硕士的中文 PEM 的得分明显低于英文 PEM。从患者的角度来看，5份由法律硕士生成的中文PEM在完整性方面的平均得分为1.82-2.70分，高于消化内科医生的评估。所有简明医疗表的可理解性均令人满意。结论：结论：LLM 在协助患者教育方面具有潜力。LLM 生成的患者教育信息的准确性和可理解性均可接受，但要提高可行性，必须进一步优化信息的完整性并考虑到各种语言环境。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

Helicobacter 医学-微生物学

CiteScore

8.40

自引率

9.10%

发文量

审稿时长

2 months

期刊介绍： Helicobacter is edited by Professor David Y Graham. The editorial and peer review process is an independent process. Whenever there is a conflict of interest, the editor and editorial board will declare their interests and affiliations. Helicobacter recognises the critical role that has been established for Helicobacter pylori in peptic ulcer, gastric adenocarcinoma, and primary gastric lymphoma. As new helicobacter species are now regularly being discovered, Helicobacter covers the entire range of helicobacter research, increasing communication among the fields of gastroenterology; microbiology; vaccine development; laboratory animal science.