Multimodal object detection method based on bidirectional dynamic sampling and adaptive Cross-Modal fusion

Abstract

The integration of visible (RGB) and infrared (IR) images have garnered extensive attention in fields such as object detection, object tracking, and scene segmentation. The complementarity of the perceptual information of these two modalities enables robust detection in all-weather and complex environments. Existing RGB-IR fusion methods have made notable progress in enhancing detection accuracy but still face problems such as modality alignment errors and insufficient feature fusion, which severely undermine detection performance. To address these challenges, we propose a model named Dynamic Communication Transformer (DynaComFormer), which can simultaneously resolve the problems of modality alignment errors and insufficient fusion precision. Within DynaComFormer, we have designed two modules: the Bidirectional Adaptive Sampling Module (BASM) and the Cross-Complementary Fusion Module (C²Fusion module). The BASM enhances the feature alignment accuracy between modalities through a dynamically guided sampling strategy while effectively reducing computational complexity. The C²Fusion module leverages self-attention mechanisms to establish efficient information interaction channels between two modalities, achieving complementary fusion of deep semantic features. We conducted experimental analyses in complex environments, such as rainy weather, strong light, and smoke. The results demonstrate that our detection performance surpasses other fusion algorithms and non-fusion algorithms by 2% to 10%.The code is available at https://github.com/alei147258/DynaComFormer-main.