Hierarchical spatiotemporal Feature Interaction Network for video saliency prediction

Transformer can build effective long-range dependency relationships and has been effectively utilized for video saliency prediction. However, fewer works have been devoted to the design of Transformer-based models for video saliency prediction. Furthermore, the existing Transformer-based models do not sufficiently explore multi-level Transformer features. To address this limitation, we present a novel Hierarchical Spatiotemporal Feature Interaction Network (i.e., HSFI-Net), which involves three crucial steps, namely multi-scale feature integration, hierarchical feature enhancement, and semantic-guided saliency prediction. Firstly, the multi-level Transformer-based spatiotemporal features are merged step by step using the multi-scale feature integration (MFI) units. Particularly, each MFI unit successively splits and cross-concatenation of features, promoting the interaction of different-level features. Furthermore, it endows features with multi-scale temporal receptive fields via different time-size kernel-based 3D convolutions. Secondly, the temporal-extended feature enhancement (TFE) unit and channel-correlated feature enhancement (CFE) unit are deployed to conduct hierarchical feature enhancement. Here, the TFE unit and the CFE unit learn rich contextual information from the temporal and channel dimensions respectively, providing powerful representations for visual attention regions in videos. Lastly, we design the semantic-guided saliency prediction (SSP) module to consolidate multi-level spatiotemporal features into the final saliency map, where the semantic information serves as a filter for purifying the fused spatiotemporal feature. We conduct extensive experiments on four challenging video saliency datasets, including DHF1K, Hollywood-2, UCF, and DIEM. The experimental results clearly demonstrate that our saliency model outperforms state-of-the-art methods. The code is available at https://github.com/JYJPush/HSFI-Net.