混合模型的高效知识提炼:用于遥感图像分类的视觉转换器-卷积神经网络-卷积神经网络方法

IF 1.5 Q3 AUTOMATION & CONTROL SYSTEMS
Huaxiang Song, Yuxuan Yuan, Zhiwei Ouyang, Yu Yang, Hui Xiang
{"title":"混合模型的高效知识提炼:用于遥感图像分类的视觉转换器-卷积神经网络-卷积神经网络方法","authors":"Huaxiang Song,&nbsp;Yuxuan Yuan,&nbsp;Zhiwei Ouyang,&nbsp;Yu Yang,&nbsp;Hui Xiang","doi":"10.1049/csy2.12120","DOIUrl":null,"url":null,"abstract":"<p>In various fields, knowledge distillation (KD) techniques that combine vision transformers (ViTs) and convolutional neural networks (CNNs) as a hybrid teacher have shown remarkable results in classification. However, in the realm of remote sensing images (RSIs), existing KD research studies are not only scarce but also lack competitiveness. This issue significantly impedes the deployment of the notable advantages of ViTs and CNNs. To tackle this, the authors introduce a novel hybrid-model KD approach named HMKD-Net, which comprises a CNN-ViT ensemble teacher and a CNN student. Contrary to popular opinion, the authors posit that the sparsity in RSI data distribution limits the effectiveness and efficiency of hybrid-model knowledge transfer. As a solution, a simple yet innovative method to handle variances during the KD phase is suggested, leading to substantial enhancements in the effectiveness and efficiency of hybrid knowledge transfer. The authors assessed the performance of HMKD-Net on three RSI datasets. The findings indicate that HMKD-Net significantly outperforms other cutting-edge methods while maintaining a significantly smaller size. Specifically, HMKD-Net exceeds other KD-based methods with a maximum accuracy improvement of 22.8% across various datasets. As ablation experiments indicated, HMKD-Net has cut down on time expenses by about 80% in the KD process. This research study validates that the hybrid-model KD technique can be more effective and efficient if the data distribution sparsity in RSIs is well handled.</p>","PeriodicalId":34110,"journal":{"name":"IET Cybersystems and Robotics","volume":null,"pages":null},"PeriodicalIF":1.5000,"publicationDate":"2024-07-10","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://onlinelibrary.wiley.com/doi/epdf/10.1049/csy2.12120","citationCount":"0","resultStr":"{\"title\":\"Efficient knowledge distillation for hybrid models: A vision transformer-convolutional neural network to convolutional neural network approach for classifying remote sensing images\",\"authors\":\"Huaxiang Song,&nbsp;Yuxuan Yuan,&nbsp;Zhiwei Ouyang,&nbsp;Yu Yang,&nbsp;Hui Xiang\",\"doi\":\"10.1049/csy2.12120\",\"DOIUrl\":null,\"url\":null,\"abstract\":\"<p>In various fields, knowledge distillation (KD) techniques that combine vision transformers (ViTs) and convolutional neural networks (CNNs) as a hybrid teacher have shown remarkable results in classification. However, in the realm of remote sensing images (RSIs), existing KD research studies are not only scarce but also lack competitiveness. This issue significantly impedes the deployment of the notable advantages of ViTs and CNNs. To tackle this, the authors introduce a novel hybrid-model KD approach named HMKD-Net, which comprises a CNN-ViT ensemble teacher and a CNN student. Contrary to popular opinion, the authors posit that the sparsity in RSI data distribution limits the effectiveness and efficiency of hybrid-model knowledge transfer. As a solution, a simple yet innovative method to handle variances during the KD phase is suggested, leading to substantial enhancements in the effectiveness and efficiency of hybrid knowledge transfer. The authors assessed the performance of HMKD-Net on three RSI datasets. The findings indicate that HMKD-Net significantly outperforms other cutting-edge methods while maintaining a significantly smaller size. Specifically, HMKD-Net exceeds other KD-based methods with a maximum accuracy improvement of 22.8% across various datasets. As ablation experiments indicated, HMKD-Net has cut down on time expenses by about 80% in the KD process. This research study validates that the hybrid-model KD technique can be more effective and efficient if the data distribution sparsity in RSIs is well handled.</p>\",\"PeriodicalId\":34110,\"journal\":{\"name\":\"IET Cybersystems and Robotics\",\"volume\":null,\"pages\":null},\"PeriodicalIF\":1.5000,\"publicationDate\":\"2024-07-10\",\"publicationTypes\":\"Journal Article\",\"fieldsOfStudy\":null,\"isOpenAccess\":false,\"openAccessPdf\":\"https://onlinelibrary.wiley.com/doi/epdf/10.1049/csy2.12120\",\"citationCount\":\"0\",\"resultStr\":null,\"platform\":\"Semanticscholar\",\"paperid\":null,\"PeriodicalName\":\"IET Cybersystems and Robotics\",\"FirstCategoryId\":\"1085\",\"ListUrlMain\":\"https://onlinelibrary.wiley.com/doi/10.1049/csy2.12120\",\"RegionNum\":0,\"RegionCategory\":null,\"ArticlePicture\":[],\"TitleCN\":null,\"AbstractTextCN\":null,\"PMCID\":null,\"EPubDate\":\"\",\"PubModel\":\"\",\"JCR\":\"Q3\",\"JCRName\":\"AUTOMATION & CONTROL SYSTEMS\",\"Score\":null,\"Total\":0}","platform":"Semanticscholar","paperid":null,"PeriodicalName":"IET Cybersystems and Robotics","FirstCategoryId":"1085","ListUrlMain":"https://onlinelibrary.wiley.com/doi/10.1049/csy2.12120","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q3","JCRName":"AUTOMATION & CONTROL SYSTEMS","Score":null,"Total":0}
引用次数: 0

摘要

在各个领域,结合视觉转换器(ViT)和卷积神经网络(CNN)作为混合教师的知识提炼(KD)技术在分类方面取得了显著效果。然而,在遥感图像(RSI)领域,现有的知识提炼研究不仅数量稀少,而且缺乏竞争力。这一问题严重阻碍了 ViT 和 CNN 显著优势的发挥。为解决这一问题,作者引入了一种名为 HMKD-Net 的新型混合模型 KD 方法,该方法由 CNN-ViT 组合教师和 CNN 学生组成。与流行观点相反,作者认为 RSI 数据分布的稀疏性限制了混合模型知识转移的效果和效率。作为解决方案,作者提出了一种简单而创新的方法来处理 KD 阶段的差异,从而大大提高了混合知识转移的效果和效率。作者在三个 RSI 数据集上评估了 HMKD-Net 的性能。研究结果表明,HMKD-Net 的性能明显优于其他前沿方法,同时体积明显缩小。具体来说,HMKD-Net 超越了其他基于 KD 的方法,在各种数据集上的准确率最高提高了 22.8%。消融实验表明,HMKD-Net 在 KD 过程中减少了约 80% 的时间支出。这项研究验证了,如果能很好地处理 RSI 中的数据分布稀疏性,混合模型 KD 技术将更加有效和高效。
本文章由计算机程序翻译,如有差异,请以英文原文为准。

Efficient knowledge distillation for hybrid models: A vision transformer-convolutional neural network to convolutional neural network approach for classifying remote sensing images

Efficient knowledge distillation for hybrid models: A vision transformer-convolutional neural network to convolutional neural network approach for classifying remote sensing images

In various fields, knowledge distillation (KD) techniques that combine vision transformers (ViTs) and convolutional neural networks (CNNs) as a hybrid teacher have shown remarkable results in classification. However, in the realm of remote sensing images (RSIs), existing KD research studies are not only scarce but also lack competitiveness. This issue significantly impedes the deployment of the notable advantages of ViTs and CNNs. To tackle this, the authors introduce a novel hybrid-model KD approach named HMKD-Net, which comprises a CNN-ViT ensemble teacher and a CNN student. Contrary to popular opinion, the authors posit that the sparsity in RSI data distribution limits the effectiveness and efficiency of hybrid-model knowledge transfer. As a solution, a simple yet innovative method to handle variances during the KD phase is suggested, leading to substantial enhancements in the effectiveness and efficiency of hybrid knowledge transfer. The authors assessed the performance of HMKD-Net on three RSI datasets. The findings indicate that HMKD-Net significantly outperforms other cutting-edge methods while maintaining a significantly smaller size. Specifically, HMKD-Net exceeds other KD-based methods with a maximum accuracy improvement of 22.8% across various datasets. As ablation experiments indicated, HMKD-Net has cut down on time expenses by about 80% in the KD process. This research study validates that the hybrid-model KD technique can be more effective and efficient if the data distribution sparsity in RSIs is well handled.

求助全文
通过发布文献求助,成功后即可免费获取论文全文。 去求助
来源期刊
IET Cybersystems and Robotics
IET Cybersystems and Robotics Computer Science-Information Systems
CiteScore
3.70
自引率
0.00%
发文量
31
审稿时长
34 weeks
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
确定
请完成安全验证×
copy
已复制链接
快去分享给好友吧!
我知道了
右上角分享
点击右上角分享
0
联系我们:info@booksci.cn Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。 Copyright © 2023 布克学术 All rights reserved.
京ICP备2023020795号-1
ghs 京公网安备 11010802042870号
Book学术文献互助
Book学术文献互助群
群 号:481959085
Book学术官方微信