Speech emotion recognition: A systematic mega-review of techniques and pipelines

Abstract

Speech Emotion Recognition (SER) is a rapidly evolving research field that aims to enable machines to automatically identify human emotions from vocal signals. This systematic review presents a comprehensive and structured synthesis of the SER literature, focusing on eleven key research questions that span the theoretical foundations, signal processing pipeline, and methodological advancements in the field. Unlike prior surveys, this review unifies both foundational and state-of-the-art insights across all stages of the SER pipeline through a question-driven structure, offering a clear road-map for both new and experienced researchers in the SER community. We first explore the psychological and computational modeling of emotions, followed by a detailed examination of the different modalities for emotion expression, with a particular emphasis on speech. The review highlights the most widely used emotional speech databases, common pre-processing techniques, and the diverse set of handcrafted and learned features employed in SER. We compare traditional machine learning approaches with recent deep learning models, emphasizing their respective strengths, limitations, and application contexts. Special attention is given to the recent shift toward self-supervised learning (SSL) models such as Wav2Vec2 and HuBERT, which have redefined the state-of-the-art in speech-based representation learning. Special attention is given to evaluation metrics, benchmarking strategies, and real-world deployment challenges, including issues of speaker-independence and environmental variability. The review concludes by identifying key limitations across the literature and articulating future research directions necessary for developing reliable, scalable, and context-aware emotion-aware systems. Overall, this work serves as a central reference for advancing SER research and practical deployment in real-world environments.