Combining spatio-temporal attention and multi-level feature fusion for video saliency prediction

Abstract

Recently, 3D convolution-based video saliency prediction models have adopted a fully convolutional encoder-decoder architecture to extract multi-level spatio-temporal features and achieved impressive performance. Deep level features encompass semantic information reflecting salient regions, shallow level features contain detailed information. However, these models have two issues: they fail to capture global information, and the equally weighted fusion mechanism they employ ignores the differences between deep and shallow features. To address these issues, we propose a novel model that combines spatio-temporal attention and multi-level feature fusion, with two main component, the global spatio-temporal correlation (GSC) structure and the attention-guided fusion (AGF) module. The GSC structure employs the Video Swin Transformer to capture global spatio-temporal correlations based on the deepest local spatio-temporal features through the multi-head attention mechanism. Rather than the equally weighted fusion mechanism, the proposed AGF module adaptively compute an attention map with only deep level features through spatio-temporal attention and channel attention branches, which guides the features to focus on salient regions and fuse. Extensive experiments over four datasets demonstrate the proposed model achieves comparable performance against state-of-the-art models and the effectiveness of each component of our model.