Multi-TuneV: Fine-tuning the fusion of multiple modules for video action recognition

Abstract

The current pre-trained models have achieved remarkable success, but they usually have complex structures and hundreds of millions of parameters, resulting in a huge computational resource requirement to train or fully fine-tune a pre-trained model, which limits its transfer learning on different tasks. In order to migrate pre-trained models to the field of Video Action Recognition (VAR), recent research uses parametric efficient transfer learning (PETL) approaches, while most of them are studied on a single fine-tuning module. For a complex task like VAR, a single fine-tuning method may not achieve optimal results. To address this challenge, we want to study the effect of joint fine-tuning with multiple modules, so we propose a method that merges multiple fine-tuning modules, namely Multi-TuneV. It combines five fine-tuning methods, including ST-Adapter, AdaptFormer, BitFit, VPT and LoRA. We design a particular architecture for Multi-TuneV and integrate it organically into the Video ViT model so that it can coordinate the multiple fine-tuning modules to extract features. Multi-TuneV enables pre-trained models to migrate to video classification tasks while maintaining improved accuracy and effectively limiting the number of tunable parameters, because it combines the advantages of five fine-tuning methods. We conduct extensive experiments with Multi-TuneV on three common video datasets, and show that it surpasses both full fine-tuning and other single fine-tuning methods. When only 18.7 % (16.09 M) of the full fine-tuning parameters are updated, the accuracy of Multi-TuneV on SSv2 and HMDB51 improve by 23.43 % and 16.46 % compared with the full fine-tuning strategy, and improve to 67.43 % and 75.84 %. This proves the effectiveness of joint multi-module fine-tuning. Multi-TuneV provides a new idea for PETL and a new perspective to address the challenge in video understanding tasks. Code is available at https://github.com/hhh123-1/Multi-TuneV.