Adaptive cross-modal fusion for robust multi-modal object detection in infrared–visible imaging

Given the challenges faced by object detection methods that rely on visible light in complex environments, many researchers have begun to explore the combination of infrared and visible imaging for multi-modal detection. Existing results show that multi-modal fusion has proven effective for improving object detection outcomes. However, most current multi-modal detection methods rely on fixed-parameter feature fusion techniques, failing to account for the imaging differences across diverse environments and the complementary information between different modalities. In this paper, we propose a multi-modal object detection method based on adaptive weight fusion, utilizing the dual-stream framework to extract features from both modalities separately. We design a Cross-Modal Feature Interaction (CMFI) module to integrate global information across modalities and capture long-range dependencies. In addition, we introduce an Adaptive Modal Weight Calculation (AMWC) module, which fully accounts for the characteristics of different modalities in various environments and the complementarity among the modalities. This module dynamically adjusts the fusion weights within the CMFI module based on the input from different modalities. Moreover, a novel loss function is introduced to regulate the internal parameter adjustments of the AMWC module. We conduct extensive experiments on three representative datasets, using mAP@0.5 and mAP@0.5:0.95 as evaluation metrics. Our model achieved 79.1% and 40.6% on the FLIR dataset, 81.9% and 52.1% on M3FD, and 73.5% and 32.5% on KAIST.