Deep Learning Based Real-Time Driver Emotion Monitoring

Abstract

In this paper, we propose a novel, real-time driver emotion monitoring system “in the wild” based on face detection and racial expression analysis. A camera is placed inside the vehicle that continuously looks at the driver's face and monitors the driver's emotional state at regular time intervals. Camera based monitoring of the driver's attentiveness based on the driver's emotional state in naturalistic driving environments is a non-intrusive approach and an important part of an automated driver assistance system (ADAS). Our work employs a face detection model based on mixture of trees with shared pool of parts to robustly detect the drivers face in varied environmental conditions. We also extract racial landmark points, and use them to enhance our emotion recognition system. In our proposed work, we use convolution neural networks. In the first, we use VGG16 to extract appearance features from the detected face image and in the second VGG16 network, to extract geometrical features from the racial landmark points. We then combine these two features using an integration method to accurately recognize the emotions. Based on the recognized emotional state of the driver, the driver can be made aware of his emotional state in case necessary. Experimental results on publicly available driver and face expression datasets show that our system is robust and accurate for driver emotion detection.