EMCFormer: Equalized Multimodal Cues Fusion Transformer for Remote Sensing Visible-Infrared Object Detection Under Long-Tailed Distribution

Abstract

Visible-infrared object detection has been widely applied in multimodal remote sensing image perception tasks due to the strong complementarity between the two modalities. However, visible-infrared remote sensing data often exhibits long-tail distribution characteristics, where some categories have sparse samples, resulting in insufficient training and poor detection performance for tail categories. To address this issue, this paper proposes an “Equalized Multi-modal Cues Fusion Transformer” (EMCFormer), incorporating an innovative “Multi-modal Heterogeneous Cues Aggregation” (MHCA) module and “Equalized-Adaptive Focal Loss” (EAFL). Specifically, MHCA leverages a cross-modal self-attention mechanism with Gumbel Softmax to generate fused features and enhance the learning of tail category features. By introducing Gumbel random noise, MHCA effectively increases attention on sparse data from tail categories, thereby producing robust fused features that enhance detection performance for these categories. In addition, EAFL dynamically amplifies the contribution of tail categories by using a dynamic focusing factor, improving performance for tail category detection. Extensive experiments on well-recognized datasets demonstrate that EMCFormer effectively improves detection accuracy for tail categories and mitigates the challenges posed by long-tail data distribution.