Edge Feature Enhanced Transformer Network for RGB and infrared image fusion based object detection

Abstract

Compared to RGB imaging, synchronous RGB and infrared (IR) imaging enables to offer more comprehensive information to improve object detection performance in diverse settings. However, current associated fusion approaches have limitation in local edge detail learning, resulting in challenges like missed detection of edge small, and occluded objects in the fused images. To fully utilize information from different modalities, an efficient Edge Feature Enhanced Transformer Network (EFETN) is developed in this study for target detection. The proposed network is designed as a dual-stream backbone transformer-based framework. Edge feature enhancement is addressed in this dual-stream network structure by embedding two proposed modules. One is constructed for feature extraction from each individual modality image through simultaneous context information and low-frequency filtering enhancement. The other utilizes the self-attention mechanism of the Transformer to merge features from various modalities in a residual form and mutually enhance each other to improve object detection performance. Comprehensive experiments and ablation studies conducted on multiple public datasets validate the efficacy of the proposed approach, showcasing state-of-the-art detection performance.