Scanpath prediction in panoramic videos through multimodal fusion

Abstract

Predicting scanpaths for panoramic visual stimuli presents a significant challenge due to the extensive field of view, the high resolution of panoramic content, and the complexity of human cognitive behavior. Accurate scanpath prediction holds substantial promise for applications such as quality adaptation strategies in the capture, processing, storage, and streaming of omnidirectional media. Despite its importance, limited studies have explored scanpath prediction in panoramic video stimuli that integrate both visual and auditory modalities. To address this gap, we perform the scanpath prediction in panoramic videos through multi-modality modeling using long short-term memory (LSTM) and Transformer based deep-learning networks. With the rapid advancement of DNNs, LSTM and Transformer based architectures have become pivotal in sequence-to-sequence tasks, significantly enhancing scanpath prediction capabilities. We propose two multi-modal prediction schemes. The first model, LSSCAN, employs a LSTM-based model to generate incrementally refined prediction outputs. The second model, TRSCAN, employs a transformer-based architecture, integrating contextual information through self-attention and cross-attention mechanisms to enhance predictive accuracy. Experimental results demonstrate that LSSCAN excels at capturing and modeling inertial patterns in scanpath prediction, while TRSCAN achieves superior performance in leveraging visual contextual information and making long-term predictions.