{"title":"Development of a Transfer Learning-Based, Multimodal Neural Network for Identifying Malignant Dermatological Lesions From Smartphone Images.","authors":"Jiawen Deng, Eddie Guo, Heather Jianbo Zhao, Kaden Venugopal, Myron Moskalyk","doi":"10.1177/11769351251349891","DOIUrl":null,"url":null,"abstract":"<p><strong>Objectives: </strong>Early skin cancer detection in primary care settings is crucial for prognosis, yet clinicians often lack relevant training. Machine learning (ML) methods may offer a potential solution for this dilemma. This study aimed to develop a neural network for the binary classification of skin lesions into malignant and benign categories using smartphone images and clinical data via a multimodal and transfer learning-based approach.</p><p><strong>Methods: </strong>We used the PAD-UFES-20 dataset, which included 2298 sets of lesion images. Three neural network models were developed: (1) a clinical data-based network, (2) an image-based network using a pre-trained DenseNet-121 and (3) a multimodal network combining clinical and image data. Models were tuned using Bayesian Optimisation HyperBand across 5-fold cross-validation. Model performance was evaluated using AUC-ROC, average precision, Brier score, calibration curve metrics, Matthews correlation coefficient (MCC), sensitivity and specificity. Model explainability was explored using permutation importance and Grad-CAM.</p><p><strong>Results: </strong>During cross-validation, the multimodal network achieved an AUC-ROC of 0.91 (95% confidence interval [CI] 0.88-0.93) and a Brier score of 0.15 (95% CI 0.11-0.19). During internal validation, it retained an AUC-ROC of 0.91 and a Brier score of 0.12. The multimodal network outperformed the unimodal models on threshold-independent metrics and at MCC-optimised threshold, but it had similar classification performance as the image-only model at high-sensitivity thresholds. Analysis of permutation importance showed that key clinical features influential for the clinical data-based network included bleeding, lesion elevation, patient age and recent lesion growth. Grad-CAM visualisations showed that the image-based network focused on lesioned regions during classification rather than background artefacts.</p><p><strong>Conclusions: </strong>A transfer learning-based, multimodal neural network can accurately identify malignant skin lesions from smartphone images and clinical data. External validation with larger, more diverse datasets is needed to assess the model's generalisability and support clinical adoption.</p>","PeriodicalId":35418,"journal":{"name":"Cancer Informatics","volume":"24 ","pages":"11769351251349891"},"PeriodicalIF":2.4000,"publicationDate":"2025-06-24","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC12188081/pdf/","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Cancer Informatics","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/10.1177/11769351251349891","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"2025/1/1 0:00:00","PubModel":"eCollection","JCR":"Q2","JCRName":"MATHEMATICAL & COMPUTATIONAL BIOLOGY","Score":null,"Total":0}
引用次数: 0
Abstract
Objectives: Early skin cancer detection in primary care settings is crucial for prognosis, yet clinicians often lack relevant training. Machine learning (ML) methods may offer a potential solution for this dilemma. This study aimed to develop a neural network for the binary classification of skin lesions into malignant and benign categories using smartphone images and clinical data via a multimodal and transfer learning-based approach.
Methods: We used the PAD-UFES-20 dataset, which included 2298 sets of lesion images. Three neural network models were developed: (1) a clinical data-based network, (2) an image-based network using a pre-trained DenseNet-121 and (3) a multimodal network combining clinical and image data. Models were tuned using Bayesian Optimisation HyperBand across 5-fold cross-validation. Model performance was evaluated using AUC-ROC, average precision, Brier score, calibration curve metrics, Matthews correlation coefficient (MCC), sensitivity and specificity. Model explainability was explored using permutation importance and Grad-CAM.
Results: During cross-validation, the multimodal network achieved an AUC-ROC of 0.91 (95% confidence interval [CI] 0.88-0.93) and a Brier score of 0.15 (95% CI 0.11-0.19). During internal validation, it retained an AUC-ROC of 0.91 and a Brier score of 0.12. The multimodal network outperformed the unimodal models on threshold-independent metrics and at MCC-optimised threshold, but it had similar classification performance as the image-only model at high-sensitivity thresholds. Analysis of permutation importance showed that key clinical features influential for the clinical data-based network included bleeding, lesion elevation, patient age and recent lesion growth. Grad-CAM visualisations showed that the image-based network focused on lesioned regions during classification rather than background artefacts.
Conclusions: A transfer learning-based, multimodal neural network can accurately identify malignant skin lesions from smartphone images and clinical data. External validation with larger, more diverse datasets is needed to assess the model's generalisability and support clinical adoption.
目的:初级保健机构的早期皮肤癌检测对预后至关重要,但临床医生往往缺乏相关培训。机器学习(ML)方法可能为这种困境提供一个潜在的解决方案。本研究旨在通过基于多模态和迁移学习的方法,利用智能手机图像和临床数据开发一个神经网络,将皮肤病变分为恶性和良性两类。方法:使用pad - upes -20数据集,该数据集包含2298组病变图像。开发了三种神经网络模型:(1)基于临床数据的网络;(2)使用预训练的DenseNet-121的基于图像的网络;(3)结合临床和图像数据的多模态网络。通过5倍交叉验证,使用贝叶斯优化HyperBand对模型进行了调整。采用AUC-ROC、平均精密度、Brier评分、校准曲线指标、Matthews相关系数(MCC)、敏感性和特异性评价模型的性能。利用排列重要性和Grad-CAM方法探讨了模型的可解释性。结果:在交叉验证中,多模式网络的AUC-ROC为0.91(95%可信区间[CI] 0.88-0.93), Brier评分为0.15 (95% CI 0.11-0.19)。在内部验证中,AUC-ROC为0.91,Brier评分为0.12。在阈值无关度量和mcc优化阈值上,多模态网络优于单模态模型,但在高灵敏度阈值下,它的分类性能与仅图像模型相似。排列重要性分析显示,影响临床数据网络的关键临床特征包括出血、病变升高、患者年龄和近期病变生长。Grad-CAM可视化显示,基于图像的网络在分类过程中专注于损伤区域,而不是背景伪像。结论:基于迁移学习的多模态神经网络可以从智能手机图像和临床数据中准确识别皮肤恶性病变。需要使用更大、更多样化的数据集进行外部验证,以评估模型的通用性并支持临床采用。
期刊介绍:
The field of cancer research relies on advances in many other disciplines, including omics technology, mass spectrometry, radio imaging, computer science, and biostatistics. Cancer Informatics provides open access to peer-reviewed high-quality manuscripts reporting bioinformatics analysis of molecular genetics and/or clinical data pertaining to cancer, emphasizing the use of machine learning, artificial intelligence, statistical algorithms, advanced imaging techniques, data visualization, and high-throughput technologies. As the leading journal dedicated exclusively to the report of the use of computational methods in cancer research and practice, Cancer Informatics leverages methodological improvements in systems biology, genomics, proteomics, metabolomics, and molecular biochemistry into the fields of cancer detection, treatment, classification, risk-prediction, prevention, outcome, and modeling.