Integrate memory mechanism in multi-granularity deep framework for driver drowsiness detection

Driver drowsiness detection is a critical task for early warning of safe driving, while existing spatial feature-based methods face the challenges of large variations of head pose. This paper proposes a novel approach to integrate the memory mechanism in a multi-granularity deep framework to detect driver drowsiness, and the temporal dependencies over sequential frames are well integrated with the spatial deep learning framework on the frontal faces. The proposed approach includes two steps. First, the spatial Multi-granularity Convolutional Neural Network is designed to utilize a group of parallel Convolutional Neural Network extractors on well-aligned facial patches of different granularities and extract facial representations effectively for large variations of head pose. Furthermore, it can flexibly fuse detailed appearance clues of the main parts and local-to-global spatial constraints. Second, the memory mechanism is set up using a deep long short-term memory network of facial representations to explore long-term relationships with variable length over sequential frames, which is capable of distinguishing the states with temporal dependencies, such as blinking and closing eyes. The proposed approach achieves 90.05% accuracy and about 37 frames per second (FPS) speed on the evaluation set of the National Tsing Hua University Driver Drowsiness Detection dataset, which is applied to the intelligent vehicle for driver drowsiness detection. A dataset named Forward Instant Driver Drowsiness Detection is also built and will be publicly accessible to speed up the study of driver drowsiness detection.