Motion Saliency Guided Fine-Grained Multimodal Interactive Learning for RGB-D Action Recognition

Abstract

RGB-D action recognition remains challenging due to the difficulty of effectively integrating complementary RGB and depth information. Most existing methods rely on feature-level fusion after independent encoding, limiting fine-grained cross-modal interaction. CLIP has recently been adapted for video tasks thanks to its powerful vision-language pretraining, but it typically requires additional temporal modules and has not been applied to RGB-D multimodal fusion for action recognition. In this paper, we innovatively extend CLIP for RGB-D action recognition, and propose a novel motion saliency guided fine-grained multimodal interactive learning (MS-FMIL) framework. Specifically, the RGB image and its corresponding depth residual dynamic image at each time step are jointly fed into CLIP image encoder, enabling patch-level fine-grained interaction between static appearance and dynamic motion cues. We further propose a motion-salient patch selection mechanism, where patch-wise motion saliency scores are computed from depth residual dynamic image to select informative regions across modalities. This enhances interaction efficiency and discriminability while reducing computation redundancy. Our method also enables temporal-aware representation learning without explicit temporal modeling modules, by injecting local spatiotemporal context from depth residual dynamic images directly into CLIP. Moreover, it supports weak temporal alignment, allowing depth dynamic information to complement static RGB features without strict frame-level synchronization. Extensive experiments on benchmark datasets demonstrate the effectiveness of our approach, offering a new perspective on leveraging CLIP for multimodal action recognition.