Inter and Intra-snippet Multi-head Attention With Position Offset for Action Localization and Recognition

Abstract

Numerous studies have focused on action localization and recognition; however, their performance suffers when applied to weakly supervised scenarios, leading to poor or rapidly declining results. This article introduces an efficient deep learning architecture based on multi-head attention to enhance action localization in untrimmed videos. Our proposed algorithm comprises three stages. Initially, a short-snippet enhancement (SSE) sampling module captures intrinsic details in video frames, adeptly balancing short-term and long-term action contributions for improved localization. The second stage employs inter-snippet and intra-snippet multi-head attention, incorporating positional offset, to capture spatio-temporal dependencies among videos and within individual video snippets, precisely identifying action boundaries. The third stage integrates an action localization network with uncertainty-guided pseudoinstance-level and video-level losses to enhance performance, mitigating the impact of noisy labels. A multistep updating process progressively refines action proposals, augmenting localization precision. To demonstrate the effectiveness of our proposed scheme, we evaluate the performance of the proposed scheme using mean average precision (mAP) over the different thresholds of intersection over union (IoU) as the evaluation measure on the “THUMOS14” and “ActivityNet-v1.3” datasets. Our algorithm achieves an mAP value of 45.20% on “THUMOS14” and an mAP value of 25.24% on “ActivityNet-v1.3.” Furthermore, we compare our technique with 24 state-of-the-art (SOTA) techniques on “THUMOS14” and eleven SOTA techniques on “ActivityNet-v1.3,” confirming the superiority of the proposed scheme.