A method for noise-suppressed multimodal feature integration in urban scene detection

The complementary imaging properties of visible and thermal infrared make them play a crucial role in multimodal object detection. Multimodal fusion methods that do not effectively deal with intra-modal and inter-modal noise interference can lead to degraded detection performance. To address this problem, we propose a generic multimodal object detection architecture. The noise within the input feature modality is first weakened by the Noise Suppression and Score-guided Fusion module (NSSFuse), while the intra-modal and inter-modal feature representations are enriched, thus facilitating the global interaction of multimodal features. Then the multimodal low-frequency features and high-frequency features are efficiently fused by the Multimodal Frequency Fusion module (MutiFreqFuse), which retains the key information while suppressing the inter-modal irrelevant noise to further enhance the multimodal feature fusion. Numerous experimental results validate the superiority of the model on the benchmark datasets, Multi-Modal Multi-Feature for Traffic Detection (M3FD) and Forward-Looking InfraRed (FLIR). The mean Average Precision (mAP) improves by 4.4–6.8% over the baseline models and is up to 6.3% higher than that of the most recent multimodal models.