Clip4Vis: Parameter-free fusion for multimodal video recognition

Abstract

Multimodal video recognition has emerged as a central focus due to its ability to effectively integrate information from diverse modalities, such as video and text. However, traditional fusion methods typically rely on trainable parameters, resulting in increased model computational costs. To address these challenges, this paper presents Clip4Vis, a zero-parameter progressive fusion framework that combines video and text features using a shallow-to-deep approach. The shallow and deep fusion steps are implemented through two key modules: (i) Cross-Model Attention, a module that enhances video embeddings with textual information, enabling adaptive focus on keyframes to improve action representation in the video. (ii) Joint Temporal-Textual Aggregation, a module that integrates video embeddings and word embeddings by jointly utilizing temporal and textual information, enabling global information aggregation. Extensive evaluations on five widely used video datasets demonstrate that our method achieves competitive performance in general, zero-shot, and few-shot video recognition. Our best model, using the released CLIP model, achieves a state-of-the-art accuracy of 87.4 % for general recognition on Kinetics-400 and 75.3 % for zero-shot recognition on Kinetics-600. The code will be released later.