mmZeAR: Zero-Effort Cross-Category Action Recognition With mmWave Radar

Abstract

Despite the widespread application of radio frequency (RF) signal-based human action recognition, traditional solutions can only recognize seen categories and the perception scope is restrained by the limited activity classes. When a novel category emerges, the model needs to be optimized again on additionally collected samples at the cost of computation and labor burden. To address this challenge, we develop the mmZeAR system, which learns semantic knowledge from available vision data as class attributes and then transforms the classification into a matching problem. Specifically, we build the attribute space by fusing the coarse-grained video classification features and fine-grained angle change features of 3D joint skeletons. Then we design an efficient feature extraction backbone named TriSqN, which integrates triple radar heatmaps into the final representations by sufficiently exploring the heterogeneous and complementary characteristics. Finally, a projection network is developed between semantic attributes and radar features to construct indirect relationships between samples and labels. By implementing mmZeAR on millimeter wave (mmWave) radar signal datasets, our extensive experiments have demonstrated its remarkable recognition accuracy in novel category recognition with zero effort and achieved state-of-the-art performance.