Exploring interaction: Inner-outer spatial–temporal transformer for skeleton-based mutual action recognition

Abstract

Transformer-based methods have achieved significant results in the field of skeleton-based action recognition. However, when dealing with two-person interaction, existing approaches normally embed the skeleton of each person separately and then introduce an additional module to learn their interactions. This risks losing the spatial and semantic connection information between the two entities, which is crucial for interaction identification. To address this issue, a unified interactive spatial–temporal transformer is proposed in this paper. First, a Two-Person Embedding (TPE) is performed to provide a holistic interactive relationship representation, which can effectively avoid the information gap caused by the division of interacting entities. Second, an innovative Inner-Outer Transformer (IOformer) combining with a new spatio-temporal partition strategy is proposed to simultaneously learn the interactions between intra-partition joints and inter-partition skeletal parts. By comprehensively capturing the key spatio-temporal interactive feature, the accuracy and robustness of interaction recognition can be significantly improved. Extensive experiments on three challenging benchmark datasets validate that our method achieves better performance in comprehensive evaluation methods.