Evaluation of ChatGPT-4 in detecting referable diabetic retinopathy using single fundus images

AJO International Pub Date : 2025-03-18 DOI:10.1016/j.ajoint.2025.100111

Owais Aftab , Hamza Khan , Brian L. VanderBeek , Drew Scoles , Benjamin J. Kim , Jonathan C. Tsui

{"title":"Evaluation of ChatGPT-4 in detecting referable diabetic retinopathy using single fundus images","authors":"Owais Aftab , Hamza Khan , Brian L. VanderBeek , Drew Scoles , Benjamin J. Kim , Jonathan C. Tsui","doi":"10.1016/j.ajoint.2025.100111","DOIUrl":null,"url":null,"abstract":"<div><h3>Purpose</h3><div>Evaluate ChatGPT-4′s ability to identify referable diabetic retinopathy (DR) from single fundus images.</div></div><div><h3>Design</h3><div>A cross-sectional study comparing ChatGPT-4′s versus retina specialists’ identification of more than mild DR (mtmDR) and vision-threatening DR (VTDR).</div></div><div><h3>Methods</h3><div>Images in equal proportions of normal, mild, moderate, and severe nonproliferative DR (NPDR), proliferative DR (PDR), and blurry images with and without suspected PDR were presented to a panel of blinded retina specialists who identified images as readable or unreadable, and potentially as mtmDR or VTDR. These images were also submitted to ChatGPT-4 three times with a standardized prompt regarding mtmDR and VTDR. Sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) were calculated for ChatGPT-4′s responses regarding mtmDR and VTDR as compared to the retina specialists majority determination.</div></div><div><h3>Results</h3><div>Retina specialists read 158/180 prompts (87.7 %) with excellent interrater reliability while ChatGPT-4 read 132/180 (73.33 %) of the image prompts. For mtmDR, ChatGPT-4 demonstrated a sensitivity of 96.2 %, specificity of 19.1 %, PPV of 69.1 %, and NPV of 72.7 %. Overall, 90.9 % of prompts read by ChatGPT-4 were labeled as mtmDR. For VTDR, ChatGPT-4 demonstrated a 63.0 % sensitivity, 62.5 % specificity, 71.9 % PPV, and 52.6 % NPV compared to retina specialists. ChatGPT-4 labeled 51.5 % of read images as VTDR. Overall referability was 66.6 % for retina specialists and 93.3 % for ChatGPT-4.</div></div><div><h3>Conclusion</h3><div>While ChatGPT-4 demonstrates promise in identifying moderate-to-severe DR, its limited specificity and tendency to overcall disease reduce its current utility as a screening tool.</div></div>","PeriodicalId":100071,"journal":{"name":"AJO International","volume":"2 2","pages":"Article 100111"},"PeriodicalIF":0.0000,"publicationDate":"2025-03-18","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"AJO International","FirstCategoryId":"1085","ListUrlMain":"https://www.sciencedirect.com/science/article/pii/S2950253525000140","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"","JCRName":"","Score":null,"Total":0}

引用次数: 0

Abstract

Purpose

Evaluate ChatGPT-4′s ability to identify referable diabetic retinopathy (DR) from single fundus images.

Design

A cross-sectional study comparing ChatGPT-4′s versus retina specialists’ identification of more than mild DR (mtmDR) and vision-threatening DR (VTDR).

Methods

Images in equal proportions of normal, mild, moderate, and severe nonproliferative DR (NPDR), proliferative DR (PDR), and blurry images with and without suspected PDR were presented to a panel of blinded retina specialists who identified images as readable or unreadable, and potentially as mtmDR or VTDR. These images were also submitted to ChatGPT-4 three times with a standardized prompt regarding mtmDR and VTDR. Sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) were calculated for ChatGPT-4′s responses regarding mtmDR and VTDR as compared to the retina specialists majority determination.

Results

Retina specialists read 158/180 prompts (87.7 %) with excellent interrater reliability while ChatGPT-4 read 132/180 (73.33 %) of the image prompts. For mtmDR, ChatGPT-4 demonstrated a sensitivity of 96.2 %, specificity of 19.1 %, PPV of 69.1 %, and NPV of 72.7 %. Overall, 90.9 % of prompts read by ChatGPT-4 were labeled as mtmDR. For VTDR, ChatGPT-4 demonstrated a 63.0 % sensitivity, 62.5 % specificity, 71.9 % PPV, and 52.6 % NPV compared to retina specialists. ChatGPT-4 labeled 51.5 % of read images as VTDR. Overall referability was 66.6 % for retina specialists and 93.3 % for ChatGPT-4.

Conclusion

While ChatGPT-4 demonstrates promise in identifying moderate-to-severe DR, its limited specificity and tendency to overcall disease reduce its current utility as a screening tool.

查看原文本刊更多论文

ChatGPT-4在单张眼底图像检测糖尿病视网膜病变中的应用价值

目的评估ChatGPT-4从单张眼底图像中识别糖尿病视网膜病变（DR）的能力。设计一项横断面研究，比较ChatGPT-4与视网膜专家对轻度以上DR （mtmDR）和视力威胁DR （VTDR）的识别。方法将正常、轻度、中度和重度非增殖性DR （NPDR）、增殖性DR （PDR）以及有或没有可疑PDR的模糊图像等比例的图像提交给盲眼视网膜专家小组，他们将图像识别为可读或不可读，可能是mtmDR或VTDR。这些图像也被提交给ChatGPT-4三次，并有关于mtdr和VTDR的标准化提示。计算ChatGPT-4对mtdr和VTDR的敏感性、特异性、阳性预测值（PPV）和阴性预测值（NPV），并与视网膜专家的多数检测结果进行比较。结果视网膜专家读取158/180条提示符（87.7%），具有良好的互读可靠性，而ChatGPT-4读取132/180条提示符（73.33%）。对于mtmDR， ChatGPT-4的敏感性为96.2%，特异性为19.1%，PPV为69.1%，NPV为72.7%。总体而言，ChatGPT-4读取的90.9%的提示被标记为mtmDR。与视网膜专家相比，ChatGPT-4对VTDR的敏感性为63.0%，特异性为62.5%，PPV为71.9%，NPV为52.6%。ChatGPT-4将51.5%的读取图像标记为VTDR。视网膜专家的总体转诊率为66.6%，ChatGPT-4的总体转诊率为93.3%。结论：虽然ChatGPT-4在识别中度至重度DR方面表现出希望，但其有限的特异性和过度诊断疾病的倾向降低了其目前作为筛查工具的实用性。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

AJO International

自引率

0.00%

发文量