Visual speech recognition by recurrent neural networks

A system for visual speech recognition is described in this paper. In the first phase of the system's operation, time-varying visual speech patterns are obtained from a sequence of images. In the second phase, the system uses recurrent neural networks to classify the spatio-temporal pattern as one of the previously-trained words. By specifying a certain behavior when a recurrent network is presented with exemplar sequences, the network is trained with no more than feed-forward complexity. The network's desired behavior is based on characterizing a given word by well-defined segments. Adaptive segmentation is employed to segment the training sequences of a given word.