RAAG:Redundancy-adaptive and attention-guided token pruning for efficient video action detection

Abstract

Video action detection faces significant computational challenges, especially with high-resolution and long video sequences. Existing fixed-rate pruning methods are often suboptimal, risking crucial information loss or retaining excessive redundancy. This paper introduces Redundancy-Adaptive and Attention-Guided Token Pruning (RAAG), a novel, adaptive framework for efficient end-to-end video action detection. RAAG integrates Information Redundancy-Adaptive Token Pruning (IRTP), which dynamically adjusts token keep rate based on inter-frame information redundancy, and a Hierarchical Attention-Guided (HAG) strategy, which refines pruning by allocating distinct layer-specific rates to preserve essential features in early layers and aggressively prune in actor-focused middle layers. Comprehensive experiments on AVA 2.2, JHMDB, and UCF101-24 demonstrate RAAG’s superior performance. Notably, RAAG (ViT-L) achieves 40.5 mAP on AVA 2.2, and robustly performs on JHMDB (90.7 mAP) and UCF101-24 (86.5 mAP). These results validate RAAG’s ability to intelligently balance computational efficiency with detection accuracy across diverse video contents.