Multi-temporal ensemble for few-shot action recognition

Abstract

Few-Shot Action Recognition (FSAR) aims to recognize novel action classes with only a few labeled samples. Due to the scarcity of labeled data, FSAR models suffer from high variance and low confidence. To address this issue, this paper first introduces ensemble learning into the field of FSAR, leveraging the diversity among multiple temporal action representations to generate base models. Specifically, we propose a Multi-Temporal Ensemble (MTE) method for FSAR. By combining sub-sequences of video frames of various lengths (i.e., tuples), MTE creates multiple sets of action representations and generates base models based on these representations. All base models share a single embedding network to learn frame-level features. The proposed method adaptively captures temporal relations with different lengths and speeds while avoiding the computational cost of training multiple deep neural networks. Furthermore, we introduce a Short-term Temporal Modeling Module (STMM) that uses self-attention to highlight frames with high variation, enhancing short-term temporal representation at the frame level. The proposed method has been validated on four benchmark datasets. Extensive experimental results demonstrate that MTE outperforms 26 state-of-the-art FSAR methods. The source code is available at https://github.com/CharmainCahill/MTE.git.