The performance of ChatGPT-4 and Bing Chat in frequently asked questions about glaucoma.

IF 1.4 4区医学 Q3 OPHTHALMOLOGY

European Journal of Ophthalmology Pub Date : 2025-02-19 DOI:10.1177/11206721251321197

Levent Doğan, İbrahim Edhem Yılmaz

{"title":"The performance of ChatGPT-4 and Bing Chat in frequently asked questions about glaucoma.","authors":"Levent Doğan, İbrahim Edhem Yılmaz","doi":"10.1177/11206721251321197","DOIUrl":null,"url":null,"abstract":"Purpose: To evaluate the appropriateness and readability of the responses generated by ChatGPT-4 and Bing Chat to frequently asked questions about glaucoma.Method: Thirty-four questions were generated for this study. Each question was directed three times to a fresh ChatGPT-4 and Bing Chat interface. The obtained responses were categorised by two glaucoma specialists in terms of their appropriateness. Accuracy of the responses was evaluated using the Structure of the Observed Learning Outcome (SOLO) taxonomy. Readability of the responses was assessed using Flesch Reading Ease (FRE), Flesch Kincaid Grade Level (FKGL), Coleman-Liau Index (CLI), Simple Measure of Gobbledygook (SMOG), and Gunning- Fog Index (GFI).Results: The percentage of appropriate responses was 88.2% (30/34) and 79.2% (27/34) in ChatGPT-4 and Bing Chat, respectively. Both the ChatGPT-4 and Bing Chat interfaces provided at least one inappropriate response to 1 of the 34 questions. The SOLO test results for ChatGPT-3.5 and Bing Chat were 3.86 ± 0.41 and 3.70 ± 0.52, respectively. No statistically significant difference in performance was observed between both LLMs (p = 0.101). The mean count of words used when generating responses was 316.5 (± 85.1) and 61.6 (± 25.8) in ChatGPT-4 and Bing Chat, respectively (p < 0.05). According to FRE scores, the generated responses were suitable for only 4.5% and 33% of U.S. adults in ChatGPT-4 and Bing Chat, respectively (p < 0.05).Conclusions: ChatGPT-4 and Bing Chat consistently provided appropriate responses to the questions. Both LLMs had low readability scores, but ChatGPT-4 provided more difficult responses in terms of readability.","PeriodicalId":12000,"journal":{"name":"European Journal of Ophthalmology","volume":" ","pages":"11206721251321197"},"PeriodicalIF":1.4000,"publicationDate":"2025-02-19","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"European Journal of Ophthalmology","FirstCategoryId":"3","ListUrlMain":"https://doi.org/10.1177/11206721251321197","RegionNum":4,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q3","JCRName":"OPHTHALMOLOGY","Score":null,"Total":0}

引用次数: 0

Abstract

Purpose: To evaluate the appropriateness and readability of the responses generated by ChatGPT-4 and Bing Chat to frequently asked questions about glaucoma.

Method: Thirty-four questions were generated for this study. Each question was directed three times to a fresh ChatGPT-4 and Bing Chat interface. The obtained responses were categorised by two glaucoma specialists in terms of their appropriateness. Accuracy of the responses was evaluated using the Structure of the Observed Learning Outcome (SOLO) taxonomy. Readability of the responses was assessed using Flesch Reading Ease (FRE), Flesch Kincaid Grade Level (FKGL), Coleman-Liau Index (CLI), Simple Measure of Gobbledygook (SMOG), and Gunning- Fog Index (GFI).

Results: The percentage of appropriate responses was 88.2% (30/34) and 79.2% (27/34) in ChatGPT-4 and Bing Chat, respectively. Both the ChatGPT-4 and Bing Chat interfaces provided at least one inappropriate response to 1 of the 34 questions. The SOLO test results for ChatGPT-3.5 and Bing Chat were 3.86 ± 0.41 and 3.70 ± 0.52, respectively. No statistically significant difference in performance was observed between both LLMs (p = 0.101). The mean count of words used when generating responses was 316.5 (± 85.1) and 61.6 (± 25.8) in ChatGPT-4 and Bing Chat, respectively (p < 0.05). According to FRE scores, the generated responses were suitable for only 4.5% and 33% of U.S. adults in ChatGPT-4 and Bing Chat, respectively (p < 0.05).

Conclusions: ChatGPT-4 and Bing Chat consistently provided appropriate responses to the questions. Both LLMs had low readability scores, but ChatGPT-4 provided more difficult responses in terms of readability.

查看原文本刊更多论文

ChatGPT-4和必应聊天在青光眼常见问题中的表现。

目的：评价ChatGPT-4和Bing Chat对青光眼常见问题回复的适宜性和可读性。方法：本研究共设置34个问题。每个问题都被三次引导到一个新的ChatGPT-4和必应聊天界面。获得的答复由两位青光眼专家根据其适当性进行分类。使用观察学习结果的结构（SOLO）分类法评估反应的准确性。采用Flesch Reading Ease （FRE）、Flesch kinkaid Grade Level （FKGL）、Coleman-Liau Index （CLI）、Simple Measure of Gobbledygook （SMOG）和Gunning- Fog Index （GFI）评估问卷的可读性。结果：ChatGPT-4和Bing Chat的正确应答率分别为88.2%（30/34）和79.2%（27/34）。ChatGPT-4和必应聊天界面对34个问题中的1个都至少提供了一个不恰当的回答。ChatGPT-3.5和Bing Chat的SOLO测试结果分别为3.86±0.41和3.70±0.52。两种llm之间的性能差异无统计学意义（p = 0.101）。ChatGPT-4和Bing Chat在生成回答时的平均字数分别为316.5（±85.1）和61.6（±25.8）(p p结论：ChatGPT-4和Bing Chat对问题的回答一致。两个法学硕士的可读性得分都很低，但ChatGPT-4在可读性方面提供了更困难的回答。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

European Journal of Ophthalmology 医学-眼科学

CiteScore

3.60

自引率

0.00%

发文量

372

审稿时长

3-8 weeks

期刊介绍： The European Journal of Ophthalmology was founded in 1991 and is issued in print bi-monthly. It publishes only peer-reviewed original research reporting clinical observations and laboratory investigations with clinical relevance focusing on new diagnostic and surgical techniques, instrument and therapy updates, results of clinical trials and research findings.