Multi-modal mamba framework for RGB-T crowd counting with linear complexity

Abstract

Existing RGB-T crowd counting methods enhance counting accuracy by integrating RGB images with thermal imaging features. However, attention-based fusion methods have a computational complexity of $O\left(N^2\right)$, which significantly increases computational costs. Moreover, current approaches fail to sufficiently retain the detailed information of the original modalities during feature fusion, leading to the loss of critical information. To address these issues, this paper proposes a cross-modal fusion network based on Mamba, named VMMNet. Specifically, a Dynamic State Space (DSS) block is designed using the selective scan mechanism, reducing the computational complexity of attention mechanisms from $O\left(N^2\right)$ to linear, thereby significantly improving network efficiency and inference speed. Furthermore, to tackle the issue of information loss during multimodal feature fusion, two innovative modules, the Cross-Mamba Enhancement Block (CMEB) and the Merge-Mamba Fusion Block (MMFB), are introduced. The CMEB enhances inter-modal information interaction through a cross-selective scan mechanism, while the MMFB further integrates the features output by CMEB to ensure information integrity. Finally, a Channel Aware Mamba Decoder (CMD) is designed to enhance the network’s modeling capability in the channel dimension. On existing RGB-T crowd counting datasets, VMMNet reduces FLOPs by 94.3 % compared to the state-of-the-art methods and achieves performance improvements of 18.7 % and 23.3 % in GAME(0) and RMSE, respectively.