A survey on video emotion recognition: Segmentation, classification, and explainable AI techniques

Abstract

Emotion recognition from videos has become a pivotal domain in computer vision and affective computing, contributing to advancements in human–computer interaction, healthcare, security, and multimedia analysis. This survey systematically reviews 137 research papers that span segmentation, classification, and explainable artificial intelligence (XAI) techniques for video-based emotion recognition. The study categorizes works into probabilistic, clustering, deep learning, affective computing, fuzzy logic, genetic algorithms, hybrid, multimodal, and XAI-based approaches. Through a structured evaluation of datasets such as FER2013, CK+, RAVDESS, AffectNet, and EMOTIC, this review highlights how convolutional, recurrent, and transformer architectures, combined with multimodal fusion and attention mechanisms, have enhanced emotion detection accuracy to maximum in certain contexts. It also identifies key challenges including dataset bias, multimodal synchronization, interpretability, and computational complexity. The paper emphasizes the rising importance of XAI in bridging the gap between model transparency and human cognition, proposing that future research focus on explainable, context-aware, and ethically grounded frameworks for robust emotion understanding. By consolidating diverse research trajectories, this survey offers a unified perspective on current advancements, limitations, and future directions in video emotion analysis.