FMFNet: A Faster Multimodal Fusion Network for action recognition via efficient modality compensation

Abstract

Skeleton-based methods have achieved remarkable success in human action recognition. However, their performance remains limited due to insufficient diversity of visual features inherently present in skeleton data. While existing multimodal approaches have demonstrated improved recognition accuracy by integrating RGB and skeleton modalities, these methods often rely on multi-stream architectures that involve numerous parameters, resulting in low inference speed. Therefore, efficient fusion strategies for RGB and skeleton modalities in multimodal frameworks deserve further exploration. In this paper, we propose a Faster Multimodal Fusion Network (FMFNet) for action recognition, which effectively integrates RGB and skeleton modalities using a modality compensation strategy. Compared to existing multi-stream multimodal methods, our approach significantly reduces model parameters while maintaining competitive recognition performance. Specifically, we first construct two types of mid-level features based on skeleton and RGB data. These mid-level features are then fused within the proposed modality compensation module to produce comprehensive representations, which are subsequently processed by a lightweight main stream to generate final predictions. Extensive experiments demonstrate that our FMFNet achieves a superior trade-off between recognition accuracy and computational efficiency on two large-scale data sets (NTU RGB+D 60 and 120). For instance, FMFNet obtains an accuracy of 95.5% on the cross-subject benchmark of NTU RGB+D 60, while being approximately 7.4$\times$ smaller and 2.3$\times$ faster than HM-CTNet, one of the current state-of-the-art methods.