Assessment of ChatGPT generated educational material for head and neck surgery counseling

IF 1.8 4区医学 Q2 OTORHINOLARYNGOLOGY

American Journal of Otolaryngology Pub Date : 2024-07-21 DOI:10.1016/j.amjoto.2024.104410

Lana Mnajjed , Rusha J. Patel

{"title":"Assessment of ChatGPT generated educational material for head and neck surgery counseling","authors":"Lana Mnajjed , Rusha J. Patel","doi":"10.1016/j.amjoto.2024.104410","DOIUrl":null,"url":null,"abstract":"<div><h3>Background</h3><p>ChatGPT is becoming very popular as an information source for the public. The adequacy of ChatGPT generated patient counseling material has not yet been extensively assessed.</p></div><div><h3>Methods</h3><p>ChatGPT was presented with perioperative counseling and complication questions regarding five different procedure, and accuracy of responses was assessed. The chat was then asked to present an explanation of each procedure, and quality of the responses were compared to online educational material.</p></div><div><h3>Results</h3><p>ChatGPT responses were comprehensive when discussing counseling points commonly discussed by a provider prior to a procedure. Responses to questions on surgical complications were less accurate and comprehensive. In comparison to online educational material, ChatGPT scored at or above the median SAM and PEMAT scores for all procedures.</p></div><div><h3>Conclusions</h3><p>ChatGPT did well addressing basic counseling points during the perioperative period, although it did not perform as well when addressing surgical complications. Chat response quality was comparable to currently available online educational material.</p></div>","PeriodicalId":7591,"journal":{"name":"American Journal of Otolaryngology","volume":"45 5","pages":"Article 104410"},"PeriodicalIF":1.8000,"publicationDate":"2024-07-21","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"American Journal of Otolaryngology","FirstCategoryId":"3","ListUrlMain":"https://www.sciencedirect.com/science/article/pii/S0196070924001960","RegionNum":4,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q2","JCRName":"OTORHINOLARYNGOLOGY","Score":null,"Total":0}

引用次数: 0

Abstract

Background

ChatGPT is becoming very popular as an information source for the public. The adequacy of ChatGPT generated patient counseling material has not yet been extensively assessed.

Methods

ChatGPT was presented with perioperative counseling and complication questions regarding five different procedure, and accuracy of responses was assessed. The chat was then asked to present an explanation of each procedure, and quality of the responses were compared to online educational material.

Results

ChatGPT responses were comprehensive when discussing counseling points commonly discussed by a provider prior to a procedure. Responses to questions on surgical complications were less accurate and comprehensive. In comparison to online educational material, ChatGPT scored at or above the median SAM and PEMAT scores for all procedures.

Conclusions

ChatGPT did well addressing basic counseling points during the perioperative period, although it did not perform as well when addressing surgical complications. Chat response quality was comparable to currently available online educational material.

查看原文本刊更多论文

对 ChatGPT 生成的头颈外科手术咨询教育材料进行评估。

背景：作为公众的信息来源，ChatGPT 越来越受欢迎。目前尚未对 ChatGPT 生成的患者咨询材料的充分性进行广泛评估：方法：向 ChatGPT 提出了有关五种不同手术的围手术期咨询和并发症问题，并对回答的准确性进行了评估。然后要求聊天者对每种手术进行解释，并将回答的质量与在线教育材料进行比较：结果：在讨论医疗服务提供者在手术前通常会讨论的咨询要点时，ChatGPT 的回答非常全面。对手术并发症问题的回答则不够准确和全面。与在线教材相比，ChatGPT 在所有手术中的得分都达到或超过了 SAM 和 PEMAT 分数的中位数：结论：ChatGPT 在解决围术期基本咨询要点方面表现良好，但在解决手术并发症方面表现不佳。聊天回复质量与目前可用的在线教育材料相当。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

American Journal of Otolaryngology 医学-耳鼻喉科学

CiteScore

4.40

自引率

4.00%

发文量

378

审稿时长

41 days

期刊介绍： Be fully informed about developments in otology, neurotology, audiology, rhinology, allergy, laryngology, speech science, bronchoesophagology, facial plastic surgery, and head and neck surgery. Featured sections include original contributions, grand rounds, current reviews, case reports and socioeconomics.