EEF: Energy score-guided feature enhancement fusion method for RGB and thermal infrared images object detection

Abstract

The full exploitation of the complementarity between different modalities is crucial for RGB and Thermal infrared images (RGB-T) object detection. However, most existing methods utilizing a traditional backbone to extract features often struggle to enhance the discriminability of features from different modalities, thereby restricting the representational capacity of fused features. We propose an energy score-guided feature enhancement fusion method (EEF) for RGB-T object detection. Firstly, we design an energy-based feature enhancement module (EFEM) that leverages the proposed channel energy score to assess the importance and reliability of feature channels to enhance the discriminability of features and make them more focused on the region of the object. Then, we introduce an Efficient Cross-modal Fusion Module (ECFM) to capture complementary information between modalities by utilizing the global feature interaction capability of attention mechanisms. Finally, we incorporate an adaptive feedback module (AFM), which utilizes the fused features as guidance information to obtain the corresponding learning weights for different modalities to enhance the representational capacity of original features. We thoroughly evaluate our approach on the LLVIP and FLIR datasets, achieving preferable results of 64.9% and 41.1% mAP. The promising results adequately demonstrate the effectiveness of EEF in RGB-T object detection tasks.