Communicative competence of generative artificial intelligence in responding to patient queries about colorectal cancer surgery.

IF 2.5 3区医学 Q2 GASTROENTEROLOGY & HEPATOLOGY

International Journal of Colorectal Disease Pub Date : 2024-06-20 DOI:10.1007/s00384-024-04670-3

Min Hyeong Jo, Min-Jun Kim, Heung-Kwon Oh, Mi Jeong Choi, Hye-Rim Shin, Tae-Gyun Lee, Hong-Min Ahn, Duck-Woo Kim, Sung-Bum Kang

{"title":"Communicative competence of generative artificial intelligence in responding to patient queries about colorectal cancer surgery.","authors":"Min Hyeong Jo, Min-Jun Kim, Heung-Kwon Oh, Mi Jeong Choi, Hye-Rim Shin, Tae-Gyun Lee, Hong-Min Ahn, Duck-Woo Kim, Sung-Bum Kang","doi":"10.1007/s00384-024-04670-3","DOIUrl":null,"url":null,"abstract":"Purpose: To examine the ability of generative artificial intelligence (GAI) to answer patients' questions regarding colorectal cancer (CRC).Methods: Ten clinically relevant questions about CRC were selected from top-rated hospitals' websites and patient surveys and presented to three GAI tools (Chatbot Generative Pre-Trained Transformer [GPT-4], Google Bard, and CLOVA X). Their responses were compared with answers from the CRC information book. Response evaluation was performed by two groups, each consisting of five healthcare professionals (HCP) and patients. Each question was scored on a 1-5 Likert scale based on four evaluation criteria (maximum score, 20 points/question).Results: In an analysis including only HCPs, the information book scored 11.8 ± 1.2, GPT-4 scored 13.5 ± 1.1, Google Bard scored 11.5 ± 0.7, and CLOVA X scored 12.2 ± 1.4 (P = 0.001). The score of GPT-4 was significantly higher than those of the information book (P = 0.020) and Google Bard (P = 0.001). In an analysis including only patients, the information book scored 14.1 ± 1.4, GPT-4 scored 15.2 ± 1.8, Google Bard scored 15.5 ± 1.8, and CLOVA X scored 14.4 ± 1.8, without significant differences (P = 0.234). When both groups of evaluators were included, the information book scored 13.0 ± 0.9, GPT-4 scored 14.4 ± 1.2, Google Bard scored 13.5 ± 1.0, and CLOVA X scored 13.3 ± 1.5 (P = 0.070).Conclusion: The three GAIs demonstrated similar or better communicative competence than the information book regarding questions related to CRC surgery in Korean. If high-quality medical information provided by GAI is supervised properly by HCPs and published as an information book, it could be helpful for patients to obtain accurate information and make informed decisions.","PeriodicalId":13789,"journal":{"name":"International Journal of Colorectal Disease","volume":null,"pages":null},"PeriodicalIF":2.5000,"publicationDate":"2024-06-20","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11189990/pdf/","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"International Journal of Colorectal Disease","FirstCategoryId":"3","ListUrlMain":"https://doi.org/10.1007/s00384-024-04670-3","RegionNum":3,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q2","JCRName":"GASTROENTEROLOGY & HEPATOLOGY","Score":null,"Total":0}

引用次数: 0

Abstract

Purpose: To examine the ability of generative artificial intelligence (GAI) to answer patients' questions regarding colorectal cancer (CRC).

Methods: Ten clinically relevant questions about CRC were selected from top-rated hospitals' websites and patient surveys and presented to three GAI tools (Chatbot Generative Pre-Trained Transformer [GPT-4], Google Bard, and CLOVA X). Their responses were compared with answers from the CRC information book. Response evaluation was performed by two groups, each consisting of five healthcare professionals (HCP) and patients. Each question was scored on a 1-5 Likert scale based on four evaluation criteria (maximum score, 20 points/question).

Results: In an analysis including only HCPs, the information book scored 11.8 ± 1.2, GPT-4 scored 13.5 ± 1.1, Google Bard scored 11.5 ± 0.7, and CLOVA X scored 12.2 ± 1.4 (P = 0.001). The score of GPT-4 was significantly higher than those of the information book (P = 0.020) and Google Bard (P = 0.001). In an analysis including only patients, the information book scored 14.1 ± 1.4, GPT-4 scored 15.2 ± 1.8, Google Bard scored 15.5 ± 1.8, and CLOVA X scored 14.4 ± 1.8, without significant differences (P = 0.234). When both groups of evaluators were included, the information book scored 13.0 ± 0.9, GPT-4 scored 14.4 ± 1.2, Google Bard scored 13.5 ± 1.0, and CLOVA X scored 13.3 ± 1.5 (P = 0.070).

Conclusion: The three GAIs demonstrated similar or better communicative competence than the information book regarding questions related to CRC surgery in Korean. If high-quality medical information provided by GAI is supervised properly by HCPs and published as an information book, it could be helpful for patients to obtain accurate information and make informed decisions.

Abstract Image

查看原文本刊更多论文

生成式人工智能在回答患者有关结直肠癌手术的询问时的交流能力。

目的：研究生成式人工智能（GAI）回答患者有关结直肠癌（CRC）问题的能力：从排名靠前的医院网站和患者调查中选取了十个与 CRC 相关的临床问题，并将其提交给三个 GAI 工具（聊天机器人生成预训练转换器 [GPT-4]、谷歌巴德和 CLOVA X）。他们的回答与 CRC 信息手册中的答案进行了比较。回答评估由两组人员进行，每组由五名医疗保健专业人员（HCP）和患者组成。每个问题都根据四项评估标准以 1-5 分的李克特量表进行评分（最高分，20 分/问题）：在仅包括医护人员的分析中，信息手册得分为 11.8 ± 1.2，GPT-4 得分为 13.5 ± 1.1，Google Bard 得分为 11.5 ± 0.7，CLOVA X 得分为 12.2 ± 1.4（P = 0.001）。GPT-4 的得分明显高于信息手册（P = 0.020）和 Google Bard（P = 0.001）。在仅包括患者的分析中，信息手册得分为 14.1 ± 1.4，GPT-4 得分为 15.2 ± 1.8，Google Bard 得分为 15.5 ± 1.8，CLOVA X 得分为 14.4 ± 1.8，无显著差异（P = 0.234）。如果将两组评估者都包括在内，信息手册得分 13.0 ± 0.9，GPT-4 得分 14.4 ± 1.2，Google Bard 得分 13.5 ± 1.0，CLOVA X 得分 13.3 ± 1.5（P = 0.070）：结论：在用韩语回答与 CRC 手术相关的问题时，三个 GAI 所表现出的沟通能力与信息手册相似或更好。如果 GAI 提供的高质量医疗信息能得到 HCP 的适当监督，并以信息手册的形式出版，将有助于患者获得准确的信息并做出明智的决定。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

International Journal of Colorectal Disease 医学-外科

CiteScore

4.90

自引率

3.60%

发文量

206

审稿时长

3-8 weeks

期刊介绍： The International Journal of Colorectal Disease, Clinical and Molecular Gastroenterology and Surgery aims to publish novel and state-of-the-art papers which deal with the physiology and pathophysiology of diseases involving the entire gastrointestinal tract. In addition to original research articles, the following categories will be included: reviews (usually commissioned but may also be submitted), case reports, letters to the editor, and protocols on clinical studies. The journal offers its readers an interdisciplinary forum for clinical science and molecular research related to gastrointestinal disease.