Switch fusion for continuous emotion estimation from multiple physiological signals

Abstract

Physiological signals represent a robust foundation for affective computing, primarily due to their resistance to conscious manipulation by subjects. With the proliferation of applications such as safe driving, mental health treatment, and wearable wellness technologies, emotion recognition based on physiological signals has garnered substantial attention. However, the increasing variety of signals captured by diverse sensors poses a challenge for models to integrate these inputs and accurately predict emotional states efficiently. Determining an optimized fusion strategy becomes increasingly complex as the number of signals grows. To address this, we propose switch fusion, a dynamic allocation fusion algorithm designed to dynamically enable models to learn optimal fusion strategies of multiple modalities. Leveraging the mixture of experts’ frameworks, our approach employs a gating mechanism to route modalities to specialized experts, utilizing these experts as fusion encoder modules. Furthermore, we demonstrate the effectiveness of time series-based models in processing physiological signals for continuous emotion estimation to enhance computational efficiency. Experiments conducted on the continuously annotated signals of emotion dataset highlight the effectiveness of switch fusion, achieving root mean square errors of 1.064 and 1.089 for arousal and valence scores, respectively, surpassing state-of-the-art methods in 3 out of 4 experimental scenarios. This study underscores the critical role of dynamic fusion strategies in continuous emotion estimation from diverse physiological signals, effectively addressing the challenges posed by the increasing complexity of sensor inputs.