Multimodal video summarization based on graph contrastive learning with fine-grained graph interaction

Abstract

Video summarization aims to extract key information from videos and select concise and representative clips to form a summary. However, unimodal video summarization methods have difficulty capturing the rich semantic in videos, while existing multimodal methods often face interference from redundant frames and noise during modal fusion, resulting in insufficient cross-modal interaction. Therefore, we introduce a multimodal video summarization model based on graph contrastive learning and fine-grained graph interaction. The model first constructs the video and text as graph structures, and uses a spatial–temporal graph network to collaboratively model spatial–temporal dependencies. Second, the node features of the video and text graph are optimized using graph contrastive learning to eliminate redundant frames and noise. In the cross-modal graph matching, the similarity between the video and text graph is modeled in parallel by introducing multiple semantic perspectives to achieve fine-grained cross-modal interaction. In addition, this paper introduces a graph alignment loss to further constrain the consistency of cross-modal semantic alignment. Finally, extensive experiments on two benchmark datasets, TVSum and SumMe, verify the effectiveness of the DCGM model, which outperforms the current state-of-the-art methods in terms of performance.