Visible-to-Infrared Image Translation for Matching Tasks

IF 4.7 2区地球科学 Q1 ENGINEERING, ELECTRICAL & ELECTRONIC

IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing Pub Date : 2024-09-25 DOI:10.1109/JSTARS.2024.3468456

Decao Ma;Shaopeng Li;Juan Su;Yong Xian;Tao Zhang

{"title":"Visible-to-Infrared Image Translation for Matching Tasks","authors":"Decao Ma;Shaopeng Li;Juan Su;Yong Xian;Tao Zhang","doi":"10.1109/JSTARS.2024.3468456","DOIUrl":null,"url":null,"abstract":"Visible-to-infrared image translation is an important way to enrich infrared data. However, the reliability of the data generated by image translation in downstream tasks has always been controversial. This article proposes a method that integrates visible-to-infrared image translation tasks and multimodal template matching tasks. The image generation network is based on a generative adversarial networks (GANs), and network training is supervised by L1 loss, GANs loss, and match loss, where the matching loss includes normalized cross-correlation (NCC) loss and match patch (MP) loss. NCC loss is constructed based on the NCC matching algorithm. MP loss is calculated by modeling template matching as a contrastive learning problem. In experiments on the KAIST, VEDAI, and AVIID datasets, this method outperforms state-of-the-art methods in terms of image generation quality and template matching accuracy. Our method incorporates the image matching process into image-to-image translation, demonstrating the usability of GANs-based image generation for critical downstream tasks. This research resolves the practical controversy of generating images based on GANs and provides a theoretical reference for image generation for tasks, such as multisource image object detection and data association.","PeriodicalId":13116,"journal":{"name":"IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing","volume":null,"pages":null},"PeriodicalIF":4.7000,"publicationDate":"2024-09-25","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=10694789","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing","FirstCategoryId":"5","ListUrlMain":"https://ieeexplore.ieee.org/document/10694789/","RegionNum":2,"RegionCategory":"地球科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"ENGINEERING, ELECTRICAL & ELECTRONIC","Score":null,"Total":0}

引用次数: 0

Abstract

Visible-to-infrared image translation is an important way to enrich infrared data. However, the reliability of the data generated by image translation in downstream tasks has always been controversial. This article proposes a method that integrates visible-to-infrared image translation tasks and multimodal template matching tasks. The image generation network is based on a generative adversarial networks (GANs), and network training is supervised by L1 loss, GANs loss, and match loss, where the matching loss includes normalized cross-correlation (NCC) loss and match patch (MP) loss. NCC loss is constructed based on the NCC matching algorithm. MP loss is calculated by modeling template matching as a contrastive learning problem. In experiments on the KAIST, VEDAI, and AVIID datasets, this method outperforms state-of-the-art methods in terms of image generation quality and template matching accuracy. Our method incorporates the image matching process into image-to-image translation, demonstrating the usability of GANs-based image generation for critical downstream tasks. This research resolves the practical controversy of generating images based on GANs and provides a theoretical reference for image generation for tasks, such as multisource image object detection and data association.

查看原文本刊更多论文

用于匹配任务的可见光到红外图像转换

可见光到红外图像转换是丰富红外数据的重要方法。然而，图像翻译生成的数据在下游任务中的可靠性一直存在争议。本文提出了一种集成可见光到红外图像翻译任务和多模态模板匹配任务的方法。图像生成网络基于生成式对抗网络（GANs），网络训练由 L1 损失、GANs 损失和匹配损失监督，其中匹配损失包括归一化交叉相关（NCC）损失和匹配补丁（MP）损失。NCC 损失是基于 NCC 匹配算法构建的。MP 损失是通过将模板匹配建模为对比学习问题来计算的。在 KAIST、VEDAI 和 AVIID 数据集的实验中，该方法在图像生成质量和模板匹配准确性方面都优于最先进的方法。我们的方法将图像匹配过程纳入了图像到图像的转换，证明了基于 GANs 的图像生成在关键下游任务中的可用性。这项研究解决了基于 GANs 生成图像的实际争议，并为多源图像对象检测和数据关联等任务的图像生成提供了理论参考。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing 地学-成像科学与照相技术

CiteScore

9.30

自引率

10.90%

发文量

563

审稿时长

4.7 months

期刊介绍： The IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing addresses the growing field of applications in Earth observations and remote sensing, and also provides a venue for the rapidly expanding special issues that are being sponsored by the IEEE Geosciences and Remote Sensing Society. The journal draws upon the experience of the highly successful “IEEE Transactions on Geoscience and Remote Sensing” and provide a complementary medium for the wide range of topics in applied earth observations. The ‘Applications’ areas encompasses the societal benefit areas of the Global Earth Observations Systems of Systems (GEOSS) program. Through deliberations over two years, ministers from 50 countries agreed to identify nine areas where Earth observation could positively impact the quality of life and health of their respective countries. Some of these are areas not traditionally addressed in the IEEE context. These include biodiversity, health and climate. Yet it is the skill sets of IEEE members, in areas such as observations, communications, computers, signal processing, standards and ocean engineering, that form the technical underpinnings of GEOSS. Thus, the Journal attracts a broad range of interests that serves both present members in new ways and expands the IEEE visibility into new areas.