Towards Safer Roads: A Deep Learning-Based Multimodal Fatigue Monitoring System

The human factor has been documented as the primary contributor to road accidents bringing outrageous costs, such as property damage, disabling injuries, and loss of life. To mitigate accident-related costs and to enhance driver safety, particularly during unfavorable driving conditions, the transportation industry strives to integrate IoT and Deep Learning technologies. In this work, we propose a holistic IoT-based multimodal technique to monitor driver fatigue by exploiting the facial and physiological information of the driver. A novel deep neural network is designed to classify the eye and mouth states. The results of the classification are fed into the cloud to be fused with other data sources (e.g., health records) in order to assess the corresponding driver risk accurately. Experimental results on various datasets show that the proposed mouth classification and eye state detection solution results in 99.5% and 99.01% accuracy, respectively.