Effect of patient-contextual skin images in human- and artificial intelligence-based diagnosis of melanoma: Results from the 2020 SIIM-ISIC melanoma classification challenge.

IF 8.4 2区 医学 Q1 DERMATOLOGY
Nicholas R Kurtansky, Clare A Primiero, Brigid Betz-Stablein, Marc Combalia, Pascale Guitera, Allan Halpern, Jonathan Kentley, Harald Kittler, Konstantinos Liopyris, Josep Malvehy, Christoph Rinner, Philipp Tschandl, Jochen Weber, Veronica Rotemberg, H Peter Soyer
{"title":"Effect of patient-contextual skin images in human- and artificial intelligence-based diagnosis of melanoma: Results from the 2020 SIIM-ISIC melanoma classification challenge.","authors":"Nicholas R Kurtansky, Clare A Primiero, Brigid Betz-Stablein, Marc Combalia, Pascale Guitera, Allan Halpern, Jonathan Kentley, Harald Kittler, Konstantinos Liopyris, Josep Malvehy, Christoph Rinner, Philipp Tschandl, Jochen Weber, Veronica Rotemberg, H Peter Soyer","doi":"10.1111/jdv.20479","DOIUrl":null,"url":null,"abstract":"<p><strong>Background: </strong>While the high accuracy of reported AI tools for melanoma detection is promising, the lack of holistic consideration of the patient is often criticized. Along with medical history, a dermatologist would also consider intra-patient nevi patterns, such that nevi that are different from others on a given patient are treated with suspicion.</p><p><strong>Objective: </strong>To evaluate whether patient-contextual lesion-images improves diagnostic accuracy for melanoma in a dermoscopic image-based AI competition and a human reader study.</p><p><strong>Methods: </strong>An international online AI competition was held in 2020. The task was to classify dermoscopy images as melanoma or benign lesions. A multi-source dataset of dermoscopy images grouped by patient were provided, and additional use of public datasets was permitted. Competitors were judged on area under the receiver operating characteristic (AUROC) on a private leaderboard. Concurrently, a human reader study was hosted using a subset of the test data. Participants gave their initial diagnosis of an index case (melanoma vs. benign) and were then presented with seven additional lesion-images of that patient before giving a second prediction of the index case. Outcome measures were sensitivity and specificity.</p><p><strong>Results: </strong>The top 50 of 3308 AI competition entries achieved AUROC scores ranging from 0.943 to 0.949. Few algorithms considered intra-patient lesion patterns and instead most evaluated images independently. The median sensitivity and specificity of human readers before receiving contextual images were 60.0% and 86.7%, and after were 60.0% and 85.7%. Human and AI algorithm performance varied by image source.</p><p><strong>Conclusion: </strong>This study provided an open-source state-of-the-art algorithm for melanoma detection that has been evaluated at multiple centres. Patient-contextual images did not positively impact performance of AI algorithms or human readers. Providing seven contextual images and no total body image may have been insufficient to test the applicability of the intra-patient lesion patterns.</p>","PeriodicalId":17351,"journal":{"name":"Journal of the European Academy of Dermatology and Venereology","volume":" ","pages":""},"PeriodicalIF":8.4000,"publicationDate":"2024-12-08","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Journal of the European Academy of Dermatology and Venereology","FirstCategoryId":"3","ListUrlMain":"https://doi.org/10.1111/jdv.20479","RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"DERMATOLOGY","Score":null,"Total":0}
引用次数: 0

Abstract

Background: While the high accuracy of reported AI tools for melanoma detection is promising, the lack of holistic consideration of the patient is often criticized. Along with medical history, a dermatologist would also consider intra-patient nevi patterns, such that nevi that are different from others on a given patient are treated with suspicion.

Objective: To evaluate whether patient-contextual lesion-images improves diagnostic accuracy for melanoma in a dermoscopic image-based AI competition and a human reader study.

Methods: An international online AI competition was held in 2020. The task was to classify dermoscopy images as melanoma or benign lesions. A multi-source dataset of dermoscopy images grouped by patient were provided, and additional use of public datasets was permitted. Competitors were judged on area under the receiver operating characteristic (AUROC) on a private leaderboard. Concurrently, a human reader study was hosted using a subset of the test data. Participants gave their initial diagnosis of an index case (melanoma vs. benign) and were then presented with seven additional lesion-images of that patient before giving a second prediction of the index case. Outcome measures were sensitivity and specificity.

Results: The top 50 of 3308 AI competition entries achieved AUROC scores ranging from 0.943 to 0.949. Few algorithms considered intra-patient lesion patterns and instead most evaluated images independently. The median sensitivity and specificity of human readers before receiving contextual images were 60.0% and 86.7%, and after were 60.0% and 85.7%. Human and AI algorithm performance varied by image source.

Conclusion: This study provided an open-source state-of-the-art algorithm for melanoma detection that has been evaluated at multiple centres. Patient-contextual images did not positively impact performance of AI algorithms or human readers. Providing seven contextual images and no total body image may have been insufficient to test the applicability of the intra-patient lesion patterns.

求助全文
约1分钟内获得全文 求助全文
来源期刊
CiteScore
10.70
自引率
8.70%
发文量
874
审稿时长
3-6 weeks
期刊介绍: The Journal of the European Academy of Dermatology and Venereology (JEADV) is a publication that focuses on dermatology and venereology. It covers various topics within these fields, including both clinical and basic science subjects. The journal publishes articles in different formats, such as editorials, review articles, practice articles, original papers, short reports, letters to the editor, features, and announcements from the European Academy of Dermatology and Venereology (EADV). The journal covers a wide range of keywords, including allergy, cancer, clinical medicine, cytokines, dermatology, drug reactions, hair disease, laser therapy, nail disease, oncology, skin cancer, skin disease, therapeutics, tumors, virus infections, and venereology. The JEADV is indexed and abstracted by various databases and resources, including Abstracts on Hygiene & Communicable Diseases, Academic Search, AgBiotech News & Information, Botanical Pesticides, CAB Abstracts®, Embase, Global Health, InfoTrac, Ingenta Select, MEDLINE/PubMed, Science Citation Index Expanded, and others.
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
确定
请完成安全验证×
copy
已复制链接
快去分享给好友吧!
我知道了
右上角分享
点击右上角分享
0
联系我们:info@booksci.cn Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。 Copyright © 2023 布克学术 All rights reserved.
京ICP备2023020795号-1
ghs 京公网安备 11010802042870号
Book学术文献互助
Book学术文献互助群
群 号:481959085
Book学术官方微信