Multi-scale pyramid-former network with multiple consistency constraints for semi-supervised video action detection

Abstract

Current semi-supervised video action detection methods predominantly emphasize consistency regularization across data augmentations, while overlooking cross-scale consistency modeling in unlabeled video data. To address this limitation, this paper proposes the Multi-Scale Pyramid-Former Network with multiple consistency constraints, termed MSPF Net. Specifically, MSPF Net employs a novel Pyramid Fusion Strategy to integrate action representations at the current scale with those from other scales through weighted fusion. This fusion strategy is embedded in each layer of MSPF Net, with each layer representing a different scale. Then, MSPF Net aggregates representations from different layers to maximize the extraction of scale information from action descriptors in unlabeled videos. Moreover, this paper employs a multiple consistency strategy to impose constraints on multi-scale information in MSPF Net, thereby further enhancing model performance. Experiments were conducted on the JHMDB-21 and UCF101-24 datasets, and the results demonstrated that MSPF Net achieved a 3.1 % and a 0.9 % improvement over the state-of-the-art methods in terms of [email protected] on the two datasets, respectively. Furthermore, the visualization results provide additional evidence that MSPF Net can accurately focus on action instances even in the absence of labels.