Depth video-based two-stream convolutional neural networks for driver fatigue detection

Recently, much research efforts have been dedicated to the development of computer-vision-based driver fatigue detection systems. Most of them utilize the RGB data, and focus on driver status detection during the day. However, drivers are more likely to be tired and drowsy during night time. In this paper, we present a driver fatigue detection system based on CNN using depth video sequences, which helps to provide alerts properly to fatigue drivers during the night time. Specifically, the two-stream CNN architecture incorporates spatial information of current depth frame and temporal information of neighboring depth frames which is represented by motion vectors. Besides, we propose a background removal system for depth video sequence of driving. Our method is trained and evaluated on our driver behavior dataset. Experiments show that the accuracy of the proposed method achieves 91.57%, which outperforms the baseline system within the recent state-of-the-art.