TBSN: Sparse-Transformer Based Siamese Network for Few-Shot Action Recognition

Abstract

Few-shot learning(FSL) problem is challenging task, which aims to recognize novel categories with only a few labeled samples. It has aroused significant attentions in both industry and academia. Most existing few-shot learning methods focus on image classification, only few works focus on few-shot video classification. For few-shot video classification problem, obtaining temporal features and designing a good distance measurement are two main challenges. In this work, we address these challenges by proposing a Sparse-Transformer Based Siamese Network termed as TBSN for few-shot action recognition which can lever-age the relative relationship and importance of frames to mine temporal characteristics of video. A relation module based on alignment and feedforward network is designed to learn a good distance measurement. In TBSN, we propose two novel modules: (1) an embedding module based on Sparse-Transformer for fusing information from different video clips to effectively capture temporal information of frames, and (2) a relation module based on alignment and feedforward network, which can discover subtle differences between samples. We conduct extensive experiments on two challenging real-world dataset(UCF101 and Kinetics 400) and compared with other state-of-the-art methods, the results demonstrate its superior performance.