Aerial video classification with Window Semantic Enhanced Video Transformers

With their exceptional flexibility and cost-effectiveness, unmanned aerial vehicles can capture vast amounts of high-quality aerial videos. Consequently, the research on unmanned aerial vehicle video classification, aiming to analyze the spatio-temporal patterns embedded in these videos automatically, is currently flourishing. Compared to conventional ground videos, aerial videos offer a broader perspective, introducing complex visual patterns of both global scenes and local motions. Although current Transformer-based methods have achieved impressive results in video classification, they struggle to capture small key subject movements from the large backgrounds of aerial videos due to a fixed global receptive field. To address these issues, we propose Window Semantic Enhanced Aerial Video Transformers that explicitly enhance local semantics and learn spatio-temporal features through self-attention design. We introduce a Window Semantic Enhanced Transformer Block, comprising a Window Localization module to identify crucial local regions in aerial videos and then enhance local semantics through Window-based Time Attention. Furthermore, we devise a Video Class Attention Transformer Block that directly learns video-level features by late class embedding of video semantic tokens, preventing intermediate frame-level representation that may lead to information loss. To validate the effectiveness of our approach, we conduct extensive experiments on two aerial video classification datasets, ERA and MOD20, demonstrating superior performance with accuracies of 73.9% and 97.0%, respectively.