A gated leaky integrate-and-fire spiking neural network based on attention mechanism for multi-modal emotion recognition

Abstract

Multi-modal emotion recognition is a key research area in human-computer interaction. It involves processing heterogeneous multi-modal signals, which present challenges in signal alignment while aiming to enhance accuracy and reduce computational complexity. To address these challenges, we apply swarm decomposition to EEG signals to reduce noise and extract Short-Time Fourier Transform features. Heatmap features are then derived from these signals, as well as from other non-physiological signals such as facial expressions, voice, and text. These features from various sources are aligned using Discrete Wavelet Transform. We propose a Gated Leaky Integrate-and-Fire Spiking Convolutional Vision Transformer (GLIFCVT) framework for multimodal emotion recognition. This framework utilizes visual features as the primary modality and incorporates a spiking gated attention mechanism to enhance multimodal fusion and classification. In addition, we propose a novel loss function that integrates Focal and Dice losses to address class imbalance. Experiments demonstrate our proposed model consistently outperform state-of-the-art methods in both accuracy and energy efficiency.