JSS-CLIP: Boosting image-to-video transfer learning with JigSaw side network

Large pre-trained vision-language models, such as CLIP, have achieved remarkable success in computer vision. However, the challenge of extending image-based models to video understanding through effective temporal modeling remains an open problem. Although recent studies have shifted their focus towards image-to-video transfer learning, the majority of existing methods overlook algorithm efficiency when adapting large models to the video domain. In this paper, we propose an innovative JigSaw Side network, JSS-CLIP, aiming to balance the algorithm efficiency and spatiotemporal modeling performance for video action recognition. Specifically, we introduce lightweight side networks attached to the frozen vision model, which avoids the backpropagation through the computationally intensive pre-trained model, thereby significantly reducing computational costs. Additionally, we design an implicit alignment module to guide the generation of hierarchical spatiotemporal JigSaw feature maps. These feature maps encapsulate rich motion information and action cues within videos, facilitating a comprehensive understanding of dynamic content. We conduct extensive experiments on three large-scale action datasets, whose results consistently demonstrate the competitiveness of JSS-CLIP in terms of efficiency and performance. The source code will be released at https://github.com/liarshen/JSS-CLIP.