Generalized multisensor wearable signal fusion for emotion recognition from noisy and incomplete data

Abstract

Continual real-time monitoring of users’ health via noninvasive wearable devices (e.g., smartwatch, smartphone) demonstrates significant potential to enhance human well-being in everyday life. However, due to respective sampling rates, noise sensitivity, and data types, the inherent heterogeneity of the signals received from multiple sensors make the task of biosignal-based emotion recognition both complex and time-consuming. While how to optimally fuse multimode information (where each sensor produces a unique mode-specific input signal) to ensure a reliable inference performance remains difficult, the particular challenges in this problem setting is primarily threefold: (1) The data availability is limited due to several unique person/device-specific properties and high cost of labeling; (2) The acquired signals from wearable devices are often noisy or may as well be lossy due to users’ personal lifestyle choices or environmental interferences; (3) Due to several intra-individual and inter-individual signal variabilities, enabling model generalizability is always difficult. To this end, we propose a general-purpose multisensor fusion network, GM-FuseNet that can seamlessly integrate and transform multi-sensor signal information for a variety of tasks. Unlike a majority of existing works, which rely on a fundamental assumption that full multi-mode query information is present during inference, GM-FuseNet’s first-level preface multimodal transformer module is explicitly designed to enhance both unimodal and multimodal performance in the presence of partial modality details. We also utilize an effective multimodal temporal correlation loss that aligns the unimode signals pairwise in the temporal domain and encourages the model to learn the temporal correlation across multiple sensor-specific signals. Extensive evaluation using two public datasets WESAD and CASE reports outperformance ($1\text{–}4\text{\%}$) of the proposed GM-FuseNet against state-of-the-art supervised or self-supervised models while delivering a consistently robust generalization all-across. Additionally, by reporting another $2\text{–}4\text{\%}$ improved accuracy and F1-scores, GM-FuseNet also demonstrates a significant promise in handling a variety of test environments including the missing and noisy multisensor query signals.