Dr Polly Thompson, Dr Richard Thornton, Mr Conor Ramsden
{"title":"Assessing chatbots ability to produce leaflets on cataract surgery: Bing AI, chatGPT 3.5, chatGPT 4o, ChatSonic, Google Bard, Perplexity and Pi.","authors":"Dr Polly Thompson, Dr Richard Thornton, Mr Conor Ramsden","doi":"10.1097/j.jcrs.0000000000001622","DOIUrl":null,"url":null,"abstract":"<p><strong>Purpose: </strong>This study aimed to evaluate leaflets on cataract surgery produced by seven common free chatbots.</p><p><strong>Setting: </strong>Usage of conversational artificial intelligence services (chatbots) is becoming more prevalent in all aspects of life, including healthcare. Cataract surgery is the most commonly performed operation in the world, with numbers set to increase. Possible applications for chatbots include information giving and education, allowing clinicians to allocate their time more efficiently.</p><p><strong>Design: </strong>Analysis of answers given by seven chatbots (Bing AI, chatGPT 3.5, chatGPT 4o, ChatSonic, Google Bard, Perplexity and Pi) were prompted to \"make a patient information leaflet on cataract surgery\".</p><p><strong>Methods: </strong>Answers were evaluated using the DISCERN instrument, Patient Education Materials Assessment Tool (PEMAT), presence of misinformation, the Flesch-Kincaid Grade Level readability score and material reliability.</p><p><strong>Results: </strong>The highest overall scored response was from ChatSonic, followed by Bing AI and then Perplexity. The lowest scoring was ChatGPT 3.5.ChatSonic achieved the highest DISCERN and PEMAT scores, and had the highest Flesch-Kincaid Grade level. The lowest DISCERN and PEMAT scores were for Pi. Only ChatGPT 3.5 included some misinformation in its response. Bing AI, ChatSonic and Perplexity included reliable references; the other chatbots provided no references.</p><p><strong>Conclusions: </strong>This study demonstrates a range of answers given by chatbots creating a cataract surgery leaflet, suggesting variation in their development and reliability. ChatGPT 3.5 scored the most poorly. However, ChatSonic indicated promise in how technology may be used to assist information giving in ophthalmology.</p>","PeriodicalId":15214,"journal":{"name":"Journal of cataract and refractive surgery","volume":" ","pages":""},"PeriodicalIF":2.6000,"publicationDate":"2025-01-31","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Journal of cataract and refractive surgery","FirstCategoryId":"3","ListUrlMain":"https://doi.org/10.1097/j.jcrs.0000000000001622","RegionNum":3,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q2","JCRName":"OPHTHALMOLOGY","Score":null,"Total":0}
引用次数: 0
Abstract
Purpose: This study aimed to evaluate leaflets on cataract surgery produced by seven common free chatbots.
Setting: Usage of conversational artificial intelligence services (chatbots) is becoming more prevalent in all aspects of life, including healthcare. Cataract surgery is the most commonly performed operation in the world, with numbers set to increase. Possible applications for chatbots include information giving and education, allowing clinicians to allocate their time more efficiently.
Design: Analysis of answers given by seven chatbots (Bing AI, chatGPT 3.5, chatGPT 4o, ChatSonic, Google Bard, Perplexity and Pi) were prompted to "make a patient information leaflet on cataract surgery".
Methods: Answers were evaluated using the DISCERN instrument, Patient Education Materials Assessment Tool (PEMAT), presence of misinformation, the Flesch-Kincaid Grade Level readability score and material reliability.
Results: The highest overall scored response was from ChatSonic, followed by Bing AI and then Perplexity. The lowest scoring was ChatGPT 3.5.ChatSonic achieved the highest DISCERN and PEMAT scores, and had the highest Flesch-Kincaid Grade level. The lowest DISCERN and PEMAT scores were for Pi. Only ChatGPT 3.5 included some misinformation in its response. Bing AI, ChatSonic and Perplexity included reliable references; the other chatbots provided no references.
Conclusions: This study demonstrates a range of answers given by chatbots creating a cataract surgery leaflet, suggesting variation in their development and reliability. ChatGPT 3.5 scored the most poorly. However, ChatSonic indicated promise in how technology may be used to assist information giving in ophthalmology.
期刊介绍：
The Journal of Cataract & Refractive Surgery (JCRS), a preeminent peer-reviewed monthly ophthalmology publication, is the official journal of the American Society of Cataract and Refractive Surgery (ASCRS) and the European Society of Cataract and Refractive Surgeons (ESCRS).
JCRS publishes high quality articles on all aspects of anterior segment surgery. In addition to original clinical studies, the journal features a consultation section, practical techniques, important cases, and reviews as well as basic science articles.
求助内容：
应助结果提醒方式：
京公网安备 11010802042870号
