MCFusion: Frequency Domain Characteristics Enhancement and Feature Compensation Fusion Network for RGB-T Object Detection

The multimodal object detection technology based on visible-thermal vision sensors has drawn significant attention as it is capable of achieving reliable object detection in complex scenes with challenging lighting conditions such as low light or backlight. However, there has been a lack of focus on the frequency domain feature information of the visible and thermal modalities themselves, as well as the complementarity of cross-modal features. Furthermore, current visible and thermal feature fusion methods only utilize feature information from the current layer, neglecting context information. Therefore, this article proposes a novel network framework for enhancing frequency domain characteristics and fusing cross-layer and cross-modal features to compensate for these limitations. This framework introduces two key modules: the frequency domain characteristics enhancement (FCE) module and the cross-layer and cross-modal feature compensation fusion (CCF) module. The FCE module consists of two sub-modules. Reduce high-frequency information loss (FCE-RHFL) module operates on the visible modality to reduce high-frequency information loss, utilizing methods such as high-frequency masking and frequency recombination. Meanwhile, enhance high-frequency information representation (FCE-EHFR) module enhances high-frequency information representation for the thermal modality through convolutions with different kernels and frequency domain enhancement techniques. In the CCF module, cross-modal and cross-layer feature compensation methods are employed to compensate for differences in modalities, followed by capture complementary information across modalities using a query-guided cross-attention mechanism. Finally, we conduct experimental comparisons on the KAIST and FLIR datasets, and the experimental results demonstrate that our method has excellent performance and robust detection results.