Adversarial Style Mixup and Improved Temporal Alignment for Cross-Domain Few-Shot Action Recognition

Abstract

Cross-Domain Few-Shot Action Recognition (CDFSAR) aims at transferring knowledge from base classes to novel ones with limited labeled data, under distribution shift between base (source domain) and novel (target domain) classes. This paper addresses the issues of insufficient style coverage for the target domain and potential temporal misalignment with chronological order in existing methods. To mitigate distribution shifts across domains, we propose an Adversarial Style Mixup (ASM) module, which enriches the diversity of style distributions covering the target domain. ASM mixes up source and target domain styles through statistical means and variances, with the adversarially learned mixup ratio and style noise. On the other hand, we design an Improved Temporal Alignment (ITA) module to address the issue of temporal misalignment between videos. In the proposed ITA, keyframes are extracted as priors for better temporal alignment with a temporal mixer to reduce the misalignment noise. Extensive experiments on video action recognition datasets demonstrates the superiority of our method compared with the state of the arts for the challenging problem of CDFSAR. Ablation study validates that both the proposed ASM and ITA modules contribute to performance improvement by style distribution expansion and keyframe-based temporal alignment.