Lip Reading in Greek words at unconstrained driving scenario

Abstract

This work focuses on the problem of Lip Reading with Greek words in an unconstrained driving scenario. The goal of Lip Reading (LR) is to understand the spoken work using only visual information, a process also known as Visual Speech Recognition (VSR). This method has several advantages over Speech Recognition, as it can work from a distance and is not affected by other sounds like noise in the environment. In this manner, LR can be considered as an alternative method for speech decoding which can be combined with state-of-the-art speech recognition technologies. The contribution of this work is two-fold. Firstly, a novel dataset with image sequences from Greek words is presented. In total, 10 persons spoke 50 words while they were either driving or simply sitting in the passenger’s seat of a car. The image sequences were recorded with a mobile phone mounted on the windshield of the car. Secondly, the recognition pipeline consists of a Convolutional Neural Network followed by a Long-Short Term Memory Network with a plain attention mechanism. This architecture maps the image sequences to words following an end-to-end learning scheme. Experimental results with various protocols indicate that speaker independent Lip Reading is an extremely challenging problem.