Multi-scale spatiotemporal normality learning for unsupervised video anomaly detection

Abstract

Video anomaly detection aims to automatically identify abnormal spatiotemporal patterns in surveillance videos. While unsupervised methods avoid the high cost of collecting abnormal data by learning from regular events, they often struggle to effectively model the inherent multiscale nature of video data. To address this challenge, we propose Multi-Scale Spatiotemporal Normality Learning (MS\(^2\)NL), a unified framework that systematically processes and integrates multiscale features across both spatial and temporal dimensions. Our framework employs an attention-enhanced stepwise fusion module to aggregate spatial features at different resolutions, enabling comprehensive modeling of appearance patterns from local textures to global structures. For temporal information processing, we design a dynamic aggregation module based on one-dimensional dilated convolutions that effectively captures motion dependencies across multi-scale feature maps while maintaining computational efficiency. These multiscale features are processed through dual decoders: a temporal decoder that learns motion normality through RGB-to-optical-flow mapping, and a spatial decoder that models appearance normality via future frame prediction, with multiscale prototype features stored in an external memory network. This sophisticated handling of multiscale information enables MS\(^2\)NL to capture subtle spatial deviations while maintaining sensitivity to temporal anomalies. Extensive experiments on benchmark datasets demonstrate the effectiveness of our approach, achieving state-of-the-art frame-level AUROCs of 98.3%, 91.5%, and 74.9% on the UCSD Ped2, CUHK Avenue, and ShanghaiTech datasets, respectively.