SAM-Net: Semantic-assisted multimodal network for action recognition in RGB-D videos

Abstract

The advent of affordable depth sensors has driven extensive research on human action recognition (HAR) in RGB-D videos. Existing unimodal approaches, such as skeleton-based or RGB video-based methods, have inherent limitations. For instance, the skeleton modality lacks spatial interaction, while the RGB video modality is highly susceptible to environmental noise. Additionally, multimodal action recognition often faces issues like insufficient data fusion and a substantial computational burden for temporal modeling. In this paper, we present an innovative Semantic-Assisted Multimodal Network (SAM-Net) for HAR in RGB-D videos. Firstly, we skillfully generate a SpatioTemporal Dynamic Region (STDR) image to instead of the RGB video modality by leveraging skeleton modality, thereby significantly reducing the video volume. Subsequently, we explore semantic information from large-scale VLMs, which effectively facilitates multimodal adaptation learning. Moreover, we implement an intramodal and intermodal multi-level fusion process for HAR. Finally, through extensive testing on three challenging datasets, our proposed SAM-Net showcases consistent state-of-the-art performance across various experimental configurations. Our codes will be released at https://github.com/2233950316/code.