Development and Validation of a Deep Learning Algorithm for Differentiation of Choroidal Nevi from Small Melanoma in Fundus Photographs

IF 3.2 Q1 OPHTHALMOLOGY
Shiva Sabazade MD , Marco A. Lumia Michalski MD , Jakub Bartoszek MD , Maria Fili MD, PhD , Mats Holmström MS , Gustav Stålhammar MD, PhD
{"title":"Development and Validation of a Deep Learning Algorithm for Differentiation of Choroidal Nevi from Small Melanoma in Fundus Photographs","authors":"Shiva Sabazade MD ,&nbsp;Marco A. Lumia Michalski MD ,&nbsp;Jakub Bartoszek MD ,&nbsp;Maria Fili MD, PhD ,&nbsp;Mats Holmström MS ,&nbsp;Gustav Stålhammar MD, PhD","doi":"10.1016/j.xops.2024.100613","DOIUrl":null,"url":null,"abstract":"<div><h3>Purpose</h3><div>To develop and validate a deep learning algorithm capable of differentiating small choroidal melanomas from nevi.</div></div><div><h3>Design</h3><div>Retrospective multicenter cohort study.</div></div><div><h3>Participants</h3><div>A total of 802 images from 688 patients diagnosed with choroidal nevi or melanoma.</div></div><div><h3>Methods</h3><div>Wide field and standard field fundus photographs were collected from patients diagnosed with choroidal nevi or melanoma by ocular oncologists during clinical examinations. A lesion was classified as a nevus if it was followed for at least 5 years without being rediagnosed as melanoma. A neural network optimized for image classification was trained and validated on cohorts of 495 and 168 images and subsequently tested on independent sets of 86 and 53 images.</div></div><div><h3>Main Outcome Measures</h3><div>Area under the curve (AUC) in receiver operating characteristic analysis for differentiating small choroidal melanomas from nevi.</div></div><div><h3>Results</h3><div>The algorithm achieved an AUC of 0.88 in both test cohorts, outperforming ophthalmologists using the Mushroom shape, Orange pigment, Large size, Enlargement, and Subretinal fluid (AUC 0.77) and To Find Small Ocular Melanoma Using Helpful Hints Daily (AUC 0.67) risk factors (DeLong’s test, <em>P</em> &lt; 0.001). The algorithm performed equally well for wide field and standard field photos (AUC 0.89 for both when analyzed separately). Using an optimal operating point of 0.63 (on a scale from 0.00 to 1.00) determined from the training and validation datasets, the algorithm achieved 100% sensitivity and 74% specificity in the first test cohort (F-score 0.72), and 80% sensitivity and 81% specificity in the second (F-score 0.71), which consisted of images from external clinics nationwide. It outperformed 12 ophthalmologists in sensitivity (Mann–Whitney <em>U</em>, <em>P</em> = 0.006) but not specificity (<em>P</em> = 0.54). The algorithm showed higher sensitivity than both resident and consultant ophthalmologists (Dunn's test, <em>P</em> = 0.04 and <em>P</em> = 0.006, respectively) but not ocular oncologists (<em>P</em> &gt; 0.99, all <em>P</em> values Bonferroni corrected).</div></div><div><h3>Conclusions</h3><div>This study develops and validates a deep learning algorithm for differentiating small choroidal melanomas from nevi, matching or surpassing the discriminatory performance of experienced human ophthalmologists. Further research will aim to validate its utility in clinical settings.</div></div><div><h3>Financial Disclosure(s)</h3><div>Financial DisclosuresProprietary or commercial disclosure may be found in the Footnotes and Disclosures at the end of this article.</div></div>","PeriodicalId":74363,"journal":{"name":"Ophthalmology science","volume":"5 1","pages":"Article 100613"},"PeriodicalIF":3.2000,"publicationDate":"2024-08-30","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Ophthalmology science","FirstCategoryId":"1085","ListUrlMain":"https://www.sciencedirect.com/science/article/pii/S2666914524001490","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"OPHTHALMOLOGY","Score":null,"Total":0}
引用次数: 0

Abstract

Purpose

To develop and validate a deep learning algorithm capable of differentiating small choroidal melanomas from nevi.

Design

Retrospective multicenter cohort study.

Participants

A total of 802 images from 688 patients diagnosed with choroidal nevi or melanoma.

Methods

Wide field and standard field fundus photographs were collected from patients diagnosed with choroidal nevi or melanoma by ocular oncologists during clinical examinations. A lesion was classified as a nevus if it was followed for at least 5 years without being rediagnosed as melanoma. A neural network optimized for image classification was trained and validated on cohorts of 495 and 168 images and subsequently tested on independent sets of 86 and 53 images.

Main Outcome Measures

Area under the curve (AUC) in receiver operating characteristic analysis for differentiating small choroidal melanomas from nevi.

Results

The algorithm achieved an AUC of 0.88 in both test cohorts, outperforming ophthalmologists using the Mushroom shape, Orange pigment, Large size, Enlargement, and Subretinal fluid (AUC 0.77) and To Find Small Ocular Melanoma Using Helpful Hints Daily (AUC 0.67) risk factors (DeLong’s test, P < 0.001). The algorithm performed equally well for wide field and standard field photos (AUC 0.89 for both when analyzed separately). Using an optimal operating point of 0.63 (on a scale from 0.00 to 1.00) determined from the training and validation datasets, the algorithm achieved 100% sensitivity and 74% specificity in the first test cohort (F-score 0.72), and 80% sensitivity and 81% specificity in the second (F-score 0.71), which consisted of images from external clinics nationwide. It outperformed 12 ophthalmologists in sensitivity (Mann–Whitney U, P = 0.006) but not specificity (P = 0.54). The algorithm showed higher sensitivity than both resident and consultant ophthalmologists (Dunn's test, P = 0.04 and P = 0.006, respectively) but not ocular oncologists (P > 0.99, all P values Bonferroni corrected).

Conclusions

This study develops and validates a deep learning algorithm for differentiating small choroidal melanomas from nevi, matching or surpassing the discriminatory performance of experienced human ophthalmologists. Further research will aim to validate its utility in clinical settings.

Financial Disclosure(s)

Financial DisclosuresProprietary or commercial disclosure may be found in the Footnotes and Disclosures at the end of this article.
开发和验证用于区分眼底照片中脉络膜痣和小黑色素瘤的深度学习算法
目的开发并验证一种能够将小型脉络膜黑色素瘤与痣区分开来的深度学习算法。方法收集眼部肿瘤专家在临床检查中诊断出脉络膜痣或黑色素瘤患者的广视野和标准视野眼底照片。如果病变经过至少 5 年的跟踪观察而未被再次诊断为黑色素瘤,则该病变被归类为痣。主要结果测量用于区分小型脉络膜黑色素瘤和痣的接收者操作特征分析曲线下面积(AUC)。结果该算法在两个测试组中的AUC都达到了0.88,优于使用蘑菇形、橙色色素、大尺寸、增大和视网膜下积液(AUC 0.77)和每日使用帮助提示查找小型眼部黑色素瘤(AUC 0.67)风险因素的眼科医生(DeLong检验,P <0.001)。该算法在宽视野和标准视野照片上的表现同样出色(单独分析时,两者的 AUC 均为 0.89)。根据训练和验证数据集确定的最佳操作点为 0.63(从 0.00 到 1.00),该算法在第一个测试群组(F-score 0.72)中的灵敏度和特异性分别达到了 100%和 74%,在第二个测试群组(F-score 0.71)中的灵敏度和特异性分别达到了 80%和 81%。该算法在灵敏度(Mann-Whitney U,P = 0.006)和特异性(P = 0.54)方面均优于 12 位眼科医生。该算法的灵敏度高于住院医生和眼科顾问医生(Dunn's 检验,P = 0.04 和 P = 0.006),但不高于眼科肿瘤医生(P > 0.99,所有 P 值均经 Bonferroni 校正)。结论本研究开发并验证了一种用于区分小型脉络膜黑色素瘤和痣的深度学习算法,其判别性能可与经验丰富的人类眼科医生相媲美,甚至更胜一筹。进一步的研究将旨在验证其在临床环境中的实用性。财务披露专利或商业披露可参见本文末尾的脚注和披露。
本文章由计算机程序翻译,如有差异,请以英文原文为准。
求助全文
约1分钟内获得全文 求助全文
来源期刊
Ophthalmology science
Ophthalmology science Ophthalmology
CiteScore
3.40
自引率
0.00%
发文量
0
审稿时长
89 days
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
确定
请完成安全验证×
copy
已复制链接
快去分享给好友吧!
我知道了
右上角分享
点击右上角分享
0
联系我们:info@booksci.cn Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。 Copyright © 2023 布克学术 All rights reserved.
京ICP备2023020795号-1
ghs 京公网安备 11010802042870号
Book学术文献互助
Book学术文献互助群
群 号:481959085
Book学术官方微信