Adaptive pooling with dual-stage fusion for skeleton-based action recognition

Pooling is essential in computer vision; however, for skeleton-based action recognition, (1) the unique structure of the skeleton limits the applicability of existing pooling strategies, and (2) the high compactness and low redundancy of the skeleton make information loss after pooling more likely to degrade accuracy. Considering these factors, in this paper, we propose an Improved Graph Pooling Network, referred to as IGPN. First, our method incorporates a region-awareness pooling strategy based on structural partitioning. Specifically, we use the correlation matrix of the original features to adaptively adjust the information weights across different regions of the newly generated feature, allowing for more flexible and effective processing. To prevent the irreversible loss of discriminative information caused by pooling, we introduce dual-stage fusion strategy that includes cross fusion module and information supplement module, which respectively complement feature-level and data-level information. As a plug-and-play structure, the proposed operation can be seamlessly integrated with existing graph conventional networks. Based on our innovations, we develop IGPN-Light, optimised for efficiency, and IGPN-Heavy, optimised for accuracy. Extensive evaluations on several challenging benchmarks demonstrate the effectiveness of our solution. For instance, in cross-subject evaluation on the NTU-RGB+D 60 dataset, IGPN-Light achieves significant accuracy improvements over the baseline while reducing FLOPs (floating-point operations per second) by 60$\sim$70%. Meanwhile, IGPN-Heavy further boosts performance by prioritising accuracy over efficiency.