Frequency-Aware Self-Supervised Group Activity Recognition with skeleton sequences

Abstract

Self-supervised, skeleton-based techniques have recently demonstrated great potential for group activity recognition via contrastive learning. However, these methods have difficulty accommodating the dynamic and complex nature of spatio-temporal data, weakening the ability to conduct effective modeling and extract crucial features. To this end, we propose a novel Frequency-Aware Group Activity Recognition (FAGAR) network, which offers a comprehensive solution by addressing three key subproblems. First, the challenge of extracting discriminative features is further exacerbated by pose estimation algorithms’ limitations under random spatio-temporal data augmentation. To mitigate this, a frequency domain passing augmentation method that emphasizes individual collaborative changes is introduced, effectively filtering out noise interference. Second, the fixed connections in traditional relation modeling networks fail to adapt to dynamic scene changes. To address this, we design an adaptive frequency domain compression network, which dynamically adjusts to scene variations. Third, the temporal modeling process often leads to a loss of focus on key features, reducing the model’s ability to assess individual contributions within a group. To resolve this, we propose an amplitude-aware loss function that guides the network in learning the relative importance of individuals, ensuring it maintains the correct learning direction. Our FAGAR achieves state-of-the-art performance on several datasets for self-supervised skeleton-based group activity recognition. Code is available at https://github.com/WGQ109/FAGAR.