Vision based Lip Reading System using Deep Learning

Abstract

Lip reading is an approach for understanding speech by visually interpreting lip movements. Vision based lip leading system takes a video (without audio) as an input of a person speaking some word or phrase and provides the predicted word or phrase the person is speaking as output. This paper presents the method for Vision based Lip Reading system that uses convolutional neural network (CNN) with attention-based Long Short-Term Memory (LSTM). The dataset includes video clips pronouncing single digits. The pretrained CNN is used for extracting features from pre-processed video frames which then are processed for learning temporal characteristics by LSTM. The SoftMax layer of architecture provides the result of lip reading. In the present work experiments are performed with two pre-trained models namely VGG19 and ResNet50 and the results are compared. To further improve the performance of the system ensembled learning is also used. The system provides 85% accuracy using ResNet50 and ensemble learning.