Multi-level semantic-assisted prototype learning for Few-Shot Action Recognition

Abstract

The Few-Shot Action Recognition (FSAR) task involves recognizing new categories with limited labeled data. The conventional fine-tuning-based adaptation approach is often prone to overfitting and lacks temporal modeling for video data. Moreover, the discrepancy in distribution between meta-training and meta-test sets can also lead to suboptimal performance in few-shot scenarios. This paper introduces a simple yet effective multi-level semantic-assisted prototype learning framework to tackle these challenges. Initially, we leverage CLIP to achieve multimodal adaptation learning and present a multi-level semantic-assisted learning module to enhance the prototypes of different action classes based on semantic information. Additionally, we integrate the lightweight adapters into the CLIP visual encoder to support parameter-efficient transfer learning and improve temporal modeling in videos. Especially, a bias compensation block is employed for feature rectification to mitigate the distribution bias in FSAR stemming from data scarcity. Extensive experiments conducted on five standard benchmark datasets demonstrate the effectiveness of the proposed method.