Spatial-Frequency Guided Pixel Transformer for NIR-to-RGB translation

Abstract

Near-Infrared (NIR) imaging is extensively employed in various applications, providing enhanced sensitivity and contrast over traditional visible-spectrum imaging. However, NIR images inherently lack the rich spatial and texture details that visible-color images possess (i.e., images with red, green, and blue bands, or RGB images). NIR-to-RGB translation – the process of converting NIR images to RGB images – has been extensively studied for this purpose. However, existing methods face two inherent challenges in NIR-to-RGB translation: spectral mapping ambiguity and statistical weak correlation. We propose a novel deep-learning architecture, termed Spatial-Frequency Guided Pixel Transformer (SF-GPT), for NIR-to-RGB Translation. Specifically, SF-GPT introduces a Spatial-Frequency Guided Multi-head Self-Attention (SFG-MSA) mechanism, enabling effective feature transfer between spatial-frequency and pixel features. Additionally, by reducing the spatial dimensions of the input images in both feature domains, we mitigate the complexity of the learning process without compromising information integrity. As a result, our method achieves state-of-the-art performance, surpassing existing techniques by over 1.5 dB in PSNR. Code will be available at https://github.com/jianghongcheng/SF-GPT.