Cache-Adapter: Efficient video action recognition using adapter fine-tuning and cache memorization technique

Traditional video action recognition tasks face significant computational challenges and require extensive computational resources. Recently, several studies have focused on efficient image-to-video transfer learning to address this problem. In this paper, we introduce a novel cache memory-based fine-tuning model called Cache Adapter, which efficiently fine-tunes large image pre-trained models for the video action recognition. Specifically, we freeze the entire pre-trained network and train only the parameters of the Cache Adapter block we designed to fuse spatio-temporal information. We also employ gated recurrent unit (GRU) to update cache information. By freezing most of the network parameters, we only need to train the adapters, significantly reducing the computational cost while achieving excellent performance. Furthermore, extensive experiments on two video action recognition benchmarks demonstrate that our approach can learn high-quality spatio-temporal representations of videos and achieve performance comparable to or even better than previous methods.