Semantic-based temporal attention network for Arabic Video Captioning

Abstract

In recent years, there has been a surge in active research aiming to bridge the gap between computer vision and natural language. In a linguistically diverse region like the Arab world, it is essential to establish a mechanism that facilitates the understanding of visual aspects in native languages. Presents an Arabic video captioning method using an encoder–decoder paradigm based on CNN and LSTM. We employ a temporal attention mechanism along with semantic features to align keyframes with relevant semantic tags. Due to the lack of an Arabic captioning dataset, we use Google’s machine translation system to generate Arabic captions for the MSVD and MSR-VTT datasets, which can be used to train end-to-end Arabic video captioning models. The semantic features are extracted from a neural semantic representation network, which has been specifically trained on Arabic tags for better understanding. Semitic languages like Arabic are heavily attributed to complex morphology, which poses challenges for video captioning. We alleviate these difficulties by employing the AraBERT model as a preprocessing tool. Comprehensive experimental results demonstrate the superior performance of the proposed method compared to state-of-the-art models on two widely-used benchmarks: achieving a CIDEr score of 72.1% on MSVD and 38.0% on MSR-VTT.