Neural network lipreading system for improved speech recognition

Abstract

A modified time-delay neural network (TDNN) has been designed to perform both automatic lipreading (speech reading) in conjunction with acoustic speech recognition in order to improve recognition both in silent environments as well as in the presence of acoustic noise. The system is far more robust to acoustic noise and verbal distractors than is a system not incorporating visual information. Specifically, in the presence of high-amplitude pink noise, the low recognition rate in the acoustic only system (43%) is raised to 75% by the incorporation of visual information. The system responds to (artificial) conflicting cross-modal patterns in a way closely analogous to the McGurk effect in humans. The power of neural techniques is demonstrated in several difficult domains: pattern recognition; sensory integration; and distributed approaches toward 'rule-based' (linguistic-phonological) processing.<>