CerviFusionNet: A multi-modal, hybrid CNN-transformer-GRU model for enhanced cervical lesion multi-classification

IF 4.6 2区综合性期刊 Q1 MULTIDISCIPLINARY SCIENCES

iScience Pub Date : 2024-11-02 DOI:10.1016/j.isci.2024.111313

Yuyang Sha , Qingyue Zhang , Xiaobing Zhai , Menghui Hou , Jingtao Lu , Weiyu Meng , Yuefei Wang , Kefeng Li , Jing Ma

{"title":"CerviFusionNet: A multi-modal, hybrid CNN-transformer-GRU model for enhanced cervical lesion multi-classification","authors":"Yuyang Sha , Qingyue Zhang , Xiaobing Zhai , Menghui Hou , Jingtao Lu , Weiyu Meng , Yuefei Wang , Kefeng Li , Jing Ma","doi":"10.1016/j.isci.2024.111313","DOIUrl":null,"url":null,"abstract":"<div><div>Cervical lesions pose a significant threat to women’s health worldwide. Colposcopy is essential for screening and treating cervical lesions, but its effectiveness depends on the doctor’s experience. Artificial intelligence-based solutions via colposcopy images have shown great potential in cervical lesions screening. However, some challenges still need to be addressed, such as low algorithm performance and lack of high-quality multi-modal datasets. Here, we established a multi-modal colposcopy dataset of 2,273 HPV+ patients, comprising original colposcopy images, acetic acid reactions at 60s and 120s, iodine staining, diagnostic reports, and pathological results. Utilizing this dataset, we developed CerviFusionNet, a hybrid architecture that merges convolutional neural networks and vision transformers to learn robust representations. We designed a temporal module to capture dynamic changes in acetic acid sequences, which can boost the model performance without sacrificing inference speed. Compared with several existing methods, CerviFusionNet demonstrated excellent accuracy and efficiency.</div></div>","PeriodicalId":342,"journal":{"name":"iScience","volume":"27 12","pages":"Article 111313"},"PeriodicalIF":4.6000,"publicationDate":"2024-11-02","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"iScience","FirstCategoryId":"103","ListUrlMain":"https://www.sciencedirect.com/science/article/pii/S2589004224025380","RegionNum":2,"RegionCategory":"综合性期刊","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"MULTIDISCIPLINARY SCIENCES","Score":null,"Total":0}

引用次数: 0

Abstract

Cervical lesions pose a significant threat to women’s health worldwide. Colposcopy is essential for screening and treating cervical lesions, but its effectiveness depends on the doctor’s experience. Artificial intelligence-based solutions via colposcopy images have shown great potential in cervical lesions screening. However, some challenges still need to be addressed, such as low algorithm performance and lack of high-quality multi-modal datasets. Here, we established a multi-modal colposcopy dataset of 2,273 HPV+ patients, comprising original colposcopy images, acetic acid reactions at 60s and 120s, iodine staining, diagnostic reports, and pathological results. Utilizing this dataset, we developed CerviFusionNet, a hybrid architecture that merges convolutional neural networks and vision transformers to learn robust representations. We designed a temporal module to capture dynamic changes in acetic acid sequences, which can boost the model performance without sacrificing inference speed. Compared with several existing methods, CerviFusionNet demonstrated excellent accuracy and efficiency.

Abstract Image

查看原文本刊更多论文

CerviFusionNet：用于增强宫颈病变多分类的多模态混合 CNN 变换器-GRU 模型

宫颈病变对全世界妇女的健康构成重大威胁。阴道镜检查对筛查和治疗宫颈病变至关重要，但其有效性取决于医生的经验。基于人工智能的阴道镜图像解决方案在宫颈病变筛查方面显示出巨大潜力。然而，一些挑战仍有待解决，如算法性能低下和缺乏高质量的多模态数据集。在这里，我们建立了一个包含 2273 名 HPV+ 患者的多模态阴道镜数据集，其中包括原始阴道镜图像、60 秒和 120 秒的醋酸反应、碘染色、诊断报告和病理结果。利用该数据集，我们开发了 CerviFusionNet，这是一种混合架构，它融合了卷积神经网络和视觉转换器，以学习稳健的表征。我们设计了一个时间模块来捕捉醋酸序列的动态变化，这可以在不牺牲推理速度的情况下提高模型性能。与现有的几种方法相比，CerviFusionNet 表现出了卓越的准确性和效率。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

iScience Multidisciplinary-Multidisciplinary

CiteScore

7.20

自引率

1.70%

发文量

1972

审稿时长

6 weeks

期刊介绍： Science has many big remaining questions. To address them, we will need to work collaboratively and across disciplines. The goal of iScience is to help fuel that type of interdisciplinary thinking. iScience is a new open-access journal from Cell Press that provides a platform for original research in the life, physical, and earth sciences. The primary criterion for publication in iScience is a significant contribution to a relevant field combined with robust results and underlying methodology. The advances appearing in iScience include both fundamental and applied investigations across this interdisciplinary range of topic areas. To support transparency in scientific investigation, we are happy to consider replication studies and papers that describe negative results. We know you want your work to be published quickly and to be widely visible within your community and beyond. With the strong international reputation of Cell Press behind it, publication in iScience will help your work garner the attention and recognition it merits. Like all Cell Press journals, iScience prioritizes rapid publication. Our editorial team pays special attention to high-quality author service and to efficient, clear-cut decisions based on the information available within the manuscript. iScience taps into the expertise across Cell Press journals and selected partners to inform our editorial decisions and help publish your science in a timely and seamless way.