Xuesong Wang, Bin Zhou, Jian Peng, Feng Huang, Xianyu Wu
{"title":"利用改进型 DenseNet 加强红外偏振和可见光图像的三源跨模态图像融合","authors":"Xuesong Wang, Bin Zhou, Jian Peng, Feng Huang, Xianyu Wu","doi":"10.1016/j.infrared.2024.105493","DOIUrl":null,"url":null,"abstract":"<div><p>The fusion of multi-modal images to create an image that preserves the unique features of each modality as well as the features shared across modalities is a challenging task, particularly in the context of infrared (IR)-visible image fusion. In addition, the presence of polarization and IR radiation information in images obtained from IR polarization sensors further complicates the multi-modal image-fusion process. This study proposes a fusion network designed to overcome the challenges associated with the integration of low-resolution IR, IR polarization, and high-resolution visible (VIS) images. By introducing cross attention modules and a multi-stage fusion approach, the network can effectively extract and fuse features from different modalities, fully expressing the diversity of the images. This network learns end-to-end mapping from sourced to fused images using a loss function, eliminating the need for ground-truth images for fusion. Experimental results using public datasets and remote-sensing field-test data demonstrate that the proposed methodology achieves commendable results in qualitative and quantitative evaluations, with gradient based fusion performance <span><math><mrow><msup><mrow><mi>Q</mi></mrow><mrow><mi>AB</mi><mo>/</mo><mi>F</mi></mrow></msup></mrow></math></span>, mutual information (MI), and <span><math><mrow><msub><mi>Q</mi><mrow><mi>CB</mi></mrow></msub></mrow></math></span> values higher than the second-best values by 0.20, 0.94, and 0.04, respectively. This study provides a comprehensive representation of target scene information that results in enhanced image quality and improved object identification capabilities. In addition, outdoor and VIS image datasets are produced, providing a data foundation and reference for future research in related fields.</p></div>","PeriodicalId":13549,"journal":{"name":"Infrared Physics & Technology","volume":null,"pages":null},"PeriodicalIF":3.1000,"publicationDate":"2024-08-19","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":"{\"title\":\"Enhancing three-source cross-modality image fusion with improved DenseNet for infrared polarization and visible light images\",\"authors\":\"Xuesong Wang, Bin Zhou, Jian Peng, Feng Huang, Xianyu Wu\",\"doi\":\"10.1016/j.infrared.2024.105493\",\"DOIUrl\":null,\"url\":null,\"abstract\":\"<div><p>The fusion of multi-modal images to create an image that preserves the unique features of each modality as well as the features shared across modalities is a challenging task, particularly in the context of infrared (IR)-visible image fusion. In addition, the presence of polarization and IR radiation information in images obtained from IR polarization sensors further complicates the multi-modal image-fusion process. This study proposes a fusion network designed to overcome the challenges associated with the integration of low-resolution IR, IR polarization, and high-resolution visible (VIS) images. By introducing cross attention modules and a multi-stage fusion approach, the network can effectively extract and fuse features from different modalities, fully expressing the diversity of the images. This network learns end-to-end mapping from sourced to fused images using a loss function, eliminating the need for ground-truth images for fusion. Experimental results using public datasets and remote-sensing field-test data demonstrate that the proposed methodology achieves commendable results in qualitative and quantitative evaluations, with gradient based fusion performance <span><math><mrow><msup><mrow><mi>Q</mi></mrow><mrow><mi>AB</mi><mo>/</mo><mi>F</mi></mrow></msup></mrow></math></span>, mutual information (MI), and <span><math><mrow><msub><mi>Q</mi><mrow><mi>CB</mi></mrow></msub></mrow></math></span> values higher than the second-best values by 0.20, 0.94, and 0.04, respectively. This study provides a comprehensive representation of target scene information that results in enhanced image quality and improved object identification capabilities. In addition, outdoor and VIS image datasets are produced, providing a data foundation and reference for future research in related fields.</p></div>\",\"PeriodicalId\":13549,\"journal\":{\"name\":\"Infrared Physics & Technology\",\"volume\":null,\"pages\":null},\"PeriodicalIF\":3.1000,\"publicationDate\":\"2024-08-19\",\"publicationTypes\":\"Journal Article\",\"fieldsOfStudy\":null,\"isOpenAccess\":false,\"openAccessPdf\":\"\",\"citationCount\":\"0\",\"resultStr\":null,\"platform\":\"Semanticscholar\",\"paperid\":null,\"PeriodicalName\":\"Infrared Physics & Technology\",\"FirstCategoryId\":\"101\",\"ListUrlMain\":\"https://www.sciencedirect.com/science/article/pii/S1350449524003773\",\"RegionNum\":3,\"RegionCategory\":\"物理与天体物理\",\"ArticlePicture\":[],\"TitleCN\":null,\"AbstractTextCN\":null,\"PMCID\":null,\"EPubDate\":\"\",\"PubModel\":\"\",\"JCR\":\"Q2\",\"JCRName\":\"INSTRUMENTS & INSTRUMENTATION\",\"Score\":null,\"Total\":0}","platform":"Semanticscholar","paperid":null,"PeriodicalName":"Infrared Physics & Technology","FirstCategoryId":"101","ListUrlMain":"https://www.sciencedirect.com/science/article/pii/S1350449524003773","RegionNum":3,"RegionCategory":"物理与天体物理","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q2","JCRName":"INSTRUMENTS & INSTRUMENTATION","Score":null,"Total":0}
引用次数: 0
摘要
多模态图像的融合是一项极具挑战性的任务,尤其是在红外(IR)-可见光图像融合的背景下,如何融合多模态图像以创建既能保留各模态独特特征又能保留各模态共享特征的图像是一项极具挑战性的任务。此外,从红外偏振传感器获取的图像中存在偏振和红外辐射信息,这使得多模态图像融合过程更加复杂。本研究提出了一种融合网络,旨在克服与低分辨率红外、红外偏振和高分辨率可见光(VIS)图像融合相关的挑战。通过引入交叉注意模块和多级融合方法,该网络可以有效地提取和融合不同模态的特征,充分表达图像的多样性。该网络利用损失函数学习从源图像到融合图像的端到端映射,从而消除了融合时对地面实况图像的需求。使用公共数据集和遥感现场测试数据得出的实验结果表明,所提出的方法在定性和定量评估方面都取得了值得称赞的结果,基于梯度的融合性能 QAB/F、互信息(MI)和 QCB 值分别比次佳值高出 0.20、0.94 和 0.04。这项研究提供了目标场景信息的综合表征,从而提高了图像质量和物体识别能力。此外,还生成了室外和 VIS 图像数据集,为今后相关领域的研究提供了数据基础和参考。
Enhancing three-source cross-modality image fusion with improved DenseNet for infrared polarization and visible light images
The fusion of multi-modal images to create an image that preserves the unique features of each modality as well as the features shared across modalities is a challenging task, particularly in the context of infrared (IR)-visible image fusion. In addition, the presence of polarization and IR radiation information in images obtained from IR polarization sensors further complicates the multi-modal image-fusion process. This study proposes a fusion network designed to overcome the challenges associated with the integration of low-resolution IR, IR polarization, and high-resolution visible (VIS) images. By introducing cross attention modules and a multi-stage fusion approach, the network can effectively extract and fuse features from different modalities, fully expressing the diversity of the images. This network learns end-to-end mapping from sourced to fused images using a loss function, eliminating the need for ground-truth images for fusion. Experimental results using public datasets and remote-sensing field-test data demonstrate that the proposed methodology achieves commendable results in qualitative and quantitative evaluations, with gradient based fusion performance , mutual information (MI), and values higher than the second-best values by 0.20, 0.94, and 0.04, respectively. This study provides a comprehensive representation of target scene information that results in enhanced image quality and improved object identification capabilities. In addition, outdoor and VIS image datasets are produced, providing a data foundation and reference for future research in related fields.
期刊介绍:
The Journal covers the entire field of infrared physics and technology: theory, experiment, application, devices and instrumentation. Infrared'' is defined as covering the near, mid and far infrared (terahertz) regions from 0.75um (750nm) to 1mm (300GHz.) Submissions in the 300GHz to 100GHz region may be accepted at the editors discretion if their content is relevant to shorter wavelengths. Submissions must be primarily concerned with and directly relevant to this spectral region.
Its core topics can be summarized as the generation, propagation and detection, of infrared radiation; the associated optics, materials and devices; and its use in all fields of science, industry, engineering and medicine.
Infrared techniques occur in many different fields, notably spectroscopy and interferometry; material characterization and processing; atmospheric physics, astronomy and space research. Scientific aspects include lasers, quantum optics, quantum electronics, image processing and semiconductor physics. Some important applications are medical diagnostics and treatment, industrial inspection and environmental monitoring.