Few-shot action recognition with temporal feature enhancement in fine-tuning

In the few-shot video action recognition task, parameter-efficient fine-tuning methods can mitigate the risk of overfitting caused by the limited training samples. However, existing frameworks that utilize these methods directly on pre-trained models still need to be improved to enhance the temporal feature extraction capabilities of the model. Additionally, the global property of the attention mechanism may diminish high-frequency information critical for describing motion characteristics of actions. To address these issues, we propose a novel few-shot video action recognition model. The model mainly perform more comprehensive parameter-efficient fine-tuning of the pre-trained image model through the STSKLoRA and SWDST-Adapter modules. The STSKLoRA module employs a multi-path aggregation architecture. By mapping the fitted 3D weight changes to the 2D level to act as the weight changes of the model, this module can enhance the ability of the model to reason about temporal information. The SWDST-Adapter module fuses the stationary wavelet transform in the up- and down-sampling structures of the adapter so that this module can use information from different frequency bands to represent video actions accurately. We conduct experiments on four public video action recognition datasets and achieve the accuracy of 79.1% (HMDB51), 97.5% (UCF101), 88.2% (Kinetics), and 75.2% (SSv2) for 5-way 5-shot, respectively, which demonstrate the superiority of our method compared to state-of-the-art methods.