Learning Human-Object Interactions in Videos by State Space Models

Abstract

Video-based human-object interaction (HOI) recognition aims at labeling human and object sequences with multiple human-object interaction classes. The efficiency of existing methods still requires improvement in terms of parameter and computational complexity, which restricts the application of video-based human-object interaction recognition. In this letter, we present HOI-Mamba, a novel approach for efficient video-based human-object interaction recognition with the state space model. HOI-Mamba transforms the spatial-temporal graph to the sequence and captures the human-object interaction features with bidirectional Mamba, which leads to superior performance with higher efficiency. Experimental results on two public human-object interaction video benchmarks demonstrate that HOI-Mamba achieves significant improvements over existing methods, e.g., achieving higher F1 Score for sub-activity recognition with fewer parameters and FLOPs than existing methods both on the CAD-120 dataset and the Something-Else dataset.