Visible speech modelling and hybrid hidden Markov models/neural networks based learning for lipreading

This paper describes a new approach for automatic visible speech recognition based on hybrid hidden Markov models/neural networks. Suitable geometric features extracted from speaker's lip shapes are used to train the speech recognizer with nonsense sentences. First we describe the use of a geometrical-based model for visible speech and we outline a self-organising-map-based approach to determine the visual specific recognition units suitable for our speaker-dependent visible speech recognition task. Then we describe five automatic lipreading systems we developed according to different classification techniques: hidden Markov models, neural networks and hybrid hidden Markov models/neural networks. All these systems are tested on a connected letter recognition task. Finally, the performance comparison underlines that a hybrid hidden Markov models/neural networks based architecture is the most promising for automatic lipreading purposes.