Two Stream Deep Convolutional Neural Network for Eye State Recognition and Blink Detection

Abstract

Eye state recognition and blink detection has been an important research problem in various fields like driver fatigue and drowsiness measurement, dry eye detection, video spoofing detection, psychological status analysis and many others. Hence an automated eye state classification and blink detection algorithm which is robust to a variety of conditions is required for this purpose. To this end, we propose a novel approach towards detection of eye blinks from a video stream by classifying the eye state of every frame as open or closed. First the eyes are localized from a frame with robust state-of-the-art facial landmark detectors. Then binary masks of the eyes are computed to capture and focus on how much the eyes are open. We propose a novel two stream convolutional neural network model which is jointly trained with the extracted eye patches, their masks as inputs and the corresponding eye state as output. With the eye state predicted by our network for every frame, we model a Finite State Machine to check for blinks by comparing number of consecutive frames with eyes closed against average human blink duration. Extensive experimentation has been done on a various number of popular benchmark datasets both for eye state classification and blink detection. Our proposed eye state classifier achieves a 3.2% and 3.86% improvement over the state-of-the-art in terms of accuracy and equal error rate (EER). The blink detector achieves a 1–2 % improvement over the state-of-the-art in terms of precision and recall. Hence our algorithm outperforms the existing methods for eye state classification and blink detection to the best of our knowledge.