S2CA: Shared Concept Prototypes and Concept-level Alignment for text–video retrieval

Abstract

Text–video retrieval, as a fundamental task of cross-modal learning, relies on effectively establishing the semantic association between text and video. At present, mainstream semantic alignment methods for text–video adopt instance-level alignment strategies, ignoring the fine-grained concept association and the “concept-level alignment” characteristics of text–video. In this regard, we propose Shared Concept Prototypes and Concept-level Alignment (S2CA) to achieve concept-level alignment. Specifically, we utilize the text–video Shared Concept Prototypes mechanism to bridge the correspondence between text and video. On this basis, we use cross-attention and Gumbel-softmax to obtain Discrete Concept Allocation Matrices and then assign text and video tokens to corresponding concept prototypes. In this way, texts and videos are decoupled into multiple Conceptual Aggregated Features, thereby achieving Concept-level Alignment. In addition, we use CLIP as the teacher model and adopt the Align-Transform-Reconstruct distillation framework to strengthen the multimodal semantic learning ability. The extensive experiments on MSR-VTT, DiDeMo, ActivityNet and MSVD prove the effectiveness of our method.