A Unified Approach to Video Anomaly Detection: Advancements in Feature Extraction, Weak Supervision, and Strategies for Class Imbalance

Abstract

This paper explores advancements in Video Anomaly Detection (VAD), combining theoretical insights with practical solutions to address model limitations. Through comprehensive experimental analysis, the study examines the role of feature representations, sampling strategies, and curriculum learning in enhancing VAD performance. Key findings include the impact of class imbalance on the Cross-Modal Awareness-Local Arousal (CMALA) architecture and the effectiveness of techniques like pseudo-curriculum learning in mitigating noisy classes, such as “Car Accident.” Novel strategies like the Sample-Batch Selection (SBS) dynamic segment selection and pre-trained image-text models, including Contrastive Language-Image Pre-training (CLIP) and ViTamin encoder, significantly improve anomaly detection. The research underscores the potential of multimodal VAD, highlighting the integration of audio and visual modalities and the development of multimodal fusion techniques. To support this evolution, the study proposes a Unified WorkStation 4 VAD (UWS4VAD) to streamline research workflows and introduces a new VAD benchmark incorporating multimodal data and textual information. The work envisions enhanced anomaly interpretation and performance by leveraging joint representation learning and Large Language Models (LLMs). The findings set the stage for future advancements, advocating for large-scale pre-training on audio-visual datasets and shifting toward a more integrated, multimodal approach to VADs. Source code of the project available at https://github.com/zuble/uws4vad