Assessing chatbots ability to produce leaflets on cataract surgery: Bing AI, chatGPT 3.5, chatGPT 4o, ChatSonic, Google Bard, Perplexity, and Pi.

IF 2.6 3区医学 Q2 OPHTHALMOLOGY

Journal of cataract and refractive surgery Pub Date : 2025-05-01 DOI:10.1097/j.jcrs.0000000000001622

Polly Thompson, Richard Thornton, Conor M Ramsden

{"title":"Assessing chatbots ability to produce leaflets on cataract surgery: Bing AI, chatGPT 3.5, chatGPT 4o, ChatSonic, Google Bard, Perplexity, and Pi.","authors":"Polly Thompson, Richard Thornton, Conor M Ramsden","doi":"10.1097/j.jcrs.0000000000001622","DOIUrl":null,"url":null,"abstract":"Purpose: To evaluate leaflets on cataract surgery produced by 7 common free chatbots.Setting: UK-based ophthalmologists carrying out online research.Design: Data were collected from the responses of 7 freely available online chatbots.Methods: Analysis of answers given by 7 chatbots (Bing AI, chatGPT 3.5, chatGPT 4o, ChatSonic, Google Bard, Perplexity, and Pi) was prompted to \"make a patient information leaflet on cataract surgery.\" Answers were evaluated using the DISCERN instrument, Patient Education Materials Assessment Tool (PEMAT), presence of misinformation, the Flesch-Kincaid Grade level readability score, and material reliability.Results: The highest overall scored response was from ChatSonic, followed by Bing AI and then Perplexity. The lowest scoring was ChatGPT 3.5. ChatSonic achieved the highest DISCERN and PEMAT scores, and had the highest Flesch-Kincaid Grade level. The lowest DISCERN and PEMAT scores were for Pi. Only ChatGPT 3.5 included some misinformation in its response. Bing AI, ChatSonic, and Perplexity included reliable references; the other chatbots provided no references.Conclusions: This study demonstrates a range of answers given by chatbots creating a cataract surgery leaflet, suggesting variation in their development and reliability. ChatGPT 3.5 scored the most poorly. However, ChatSonic indicated promise in how technology may be used to assist information giving in ophthalmology.","PeriodicalId":15214,"journal":{"name":"Journal of cataract and refractive surgery","volume":" ","pages":"371-375"},"PeriodicalIF":2.6000,"publicationDate":"2025-05-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Journal of cataract and refractive surgery","FirstCategoryId":"3","ListUrlMain":"https://doi.org/10.1097/j.jcrs.0000000000001622","RegionNum":3,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q2","JCRName":"OPHTHALMOLOGY","Score":null,"Total":0}

引用次数: 0

Abstract

Purpose: To evaluate leaflets on cataract surgery produced by 7 common free chatbots.

Setting: UK-based ophthalmologists carrying out online research.

Design: Data were collected from the responses of 7 freely available online chatbots.

Methods: Analysis of answers given by 7 chatbots (Bing AI, chatGPT 3.5, chatGPT 4o, ChatSonic, Google Bard, Perplexity, and Pi) was prompted to "make a patient information leaflet on cataract surgery." Answers were evaluated using the DISCERN instrument, Patient Education Materials Assessment Tool (PEMAT), presence of misinformation, the Flesch-Kincaid Grade level readability score, and material reliability.

Results: The highest overall scored response was from ChatSonic, followed by Bing AI and then Perplexity. The lowest scoring was ChatGPT 3.5. ChatSonic achieved the highest DISCERN and PEMAT scores, and had the highest Flesch-Kincaid Grade level. The lowest DISCERN and PEMAT scores were for Pi. Only ChatGPT 3.5 included some misinformation in its response. Bing AI, ChatSonic, and Perplexity included reliable references; the other chatbots provided no references.

Conclusions: This study demonstrates a range of answers given by chatbots creating a cataract surgery leaflet, suggesting variation in their development and reliability. ChatGPT 3.5 scored the most poorly. However, ChatSonic indicated promise in how technology may be used to assist information giving in ophthalmology.

查看原文本刊更多论文

评估聊天机器人制作白内障手术传单的能力：Bing AI、chatGPT 3.5、chatGPT 40、ChatSonic、b谷歌Bard、Perplexity和Pi。

目的：本研究旨在评估七种常见的免费聊天机器人制作的白内障手术宣传单。环境：会话式人工智能服务（聊天机器人）在生活的各个方面变得越来越普遍，包括医疗保健。白内障手术是世界上最常见的手术，而且数量还在增加。聊天机器人可能的应用包括信息传递和教育，让临床医生更有效地分配时间。设计：分析7个聊天机器人（Bing AI、chatGPT 3.5、chatGPT 40、ChatSonic、b谷歌Bard、Perplexity、Pi）给出的答案，提示“制作白内障手术患者信息手册”。方法：采用DISCERN仪器、患者教育材料评估工具（PEMAT）、错误信息的存在、Flesch-Kincaid Grade Level可读性评分和材料信度对答案进行评估。结果：ChatSonic的综合得分最高，其次是Bing AI，然后是Perplexity。最低的得分是ChatGPT 3.5。ChatSonic取得了最高的DISCERN和PEMAT分数，并具有最高的flesch - kinkaid等级。Pi的辨明和PEMAT分数最低。只有ChatGPT 3.5在其响应中包含了一些错误信息。Bing AI、ChatSonic和Perplexity都包含可靠的参考文献；其他聊天机器人没有提供参考。结论：这项研究展示了聊天机器人在制作白内障手术传单时给出的一系列答案，表明它们的发展和可靠性存在差异。ChatGPT 3.5得分最低。然而，ChatSonic在如何使用技术来辅助眼科信息提供方面表现出了希望。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

$Journal of cataract and refractive surgery$

Journal of cataract and refractive surgery 医学-外科

CiteScore

5.60

自引率

14.30%

发文量

259

审稿时长

8.5 weeks

期刊介绍： The Journal of Cataract & Refractive Surgery (JCRS), a preeminent peer-reviewed monthly ophthalmology publication, is the official journal of the American Society of Cataract and Refractive Surgery (ASCRS) and the European Society of Cataract and Refractive Surgeons (ESCRS). JCRS publishes high quality articles on all aspects of anterior segment surgery. In addition to original clinical studies, the journal features a consultation section, practical techniques, important cases, and reviews as well as basic science articles.