An Intelligent Camera-Based Contactless Driver Stress State Monitoring Using Multimodality Fusion

Abstract

Driver stress involves complex psychological, physiological, and behavioral responses to stressors across different mobility spaces, which leads to road accidents. Recently, noncontact sensing-derived biosignals have been explored in mental health assessment. However, camera-based biosignals in mobility environments is still challenging. In this study, we aim to classify the driver stress using imaging photoplethysmography (iPPG) signals, facial keypoints, and fusion-based convolutional neural network (CNN). For this, we acquired infrared facial videos from healthy subjects (N=20) in simulated driving. iPPG signals and facial keypoints were extracted using the local group invariance method and CNN, respectively. The iPPG signals were processed with a 1-D CNN, and facial keypoints with a 2-D CNN for feature learning. The proposed approach is able to classify between the drivers' stress states. Experimental results show that the proposed fusion approach achieved an mean classification accuracy (ACC) and F1-score of 87.00% and 86.33%, respectively. The iPPG signals demonstrated a better mean ACC (90.00%) and F1-score (90.33%) among the individual models. Thus, the framework could be extended for driver stress detection in real-time scenarios enabling early stress detection.