PredVSD: Video saliency prediction based on conditional diffusion model

Abstract

Mainstream deep learning methods for video saliency prediction often use 3D CNNs or Vision Transformers as encoder–decoders, relying on task-specific loss functions to implicitly map input frames to saliency maps. However, these methods are limited by their capacity for salient feature expression. In this study, inspired by the recent advances of diffusion models in video processing tasks, we propose a Conditional Diffusion Model for Video Saliency Prediction (PredVSD), which leverages semantic video features and saliency-specific encodings as conditions to capture more representative saliency features from the target data distribution. To effectively integrate multi-scale visual features and saliency priors, we design an auxiliary network, Saliency-PyramidU-Net, allowing the denoising process to focus more on salient regions across the spatial–temporal plane. Extensive experiments confirm PredVSD’s strong performance across visual and audio-visual datasets.