MDSI: Pluggable Multi-strategy Decoupling with Semantic Integration for RGB-D Gesture Recognition

Abstract

Gestures encompass intricate visual representations, containing both task-relevant cues such as hand shapes and task-irrelevant elements like backgrounds and performer appearances. Despite progress in RGB-D-based gesture recognition, two primary challenges persist: (i) Information Redundancy (IR), which hinders the task-relevant feature extraction in the entangled space and misleads the recognition; (ii) Information Absence (IA), which exacerbates the difficulty of identifying visually similar instances. To alleviate these drawbacks, we propose a pluggable Multi-strategy Decoupling with Semantic Integration methodology, termed MDSI, for RGB-D gesture recognition. For IR, we introduce a Multi-strategy Decoupling Network (MDN) to precisely segregate pose-motion and spatial-temporal-channel features across modalities, thus effectively mitigating redundant information. For IA, we introduce the Semantic Integration Network (SIN), which integrates natural language modeling through semantic filtering and semantic label smoothing, markedly enhancing the model’s semantic understanding and knowledge integration. MDSI’s pluggable architecture allows for seamless integration into various RGB-D-based gesture recognition methods with minimal computational overhead. Experiments conducted on two public datasets demonstrate that our approach provides better feature representation and achieves better performance than state-of-the-art methods.