Spectrum-guided Spatial Feature Enhancement Network for event-based lip-reading

Abstract

The Automatic Lip-reading task aims to recognize spoken words through visual cues from the speaker’s lip movements. This crucial task complements audio-based speech recognition systems and can substitute them when sound is unavailable. Event-based lip-reading methods have gained increasing attention due to the advantages of event cameras, such as high temporal resolution and low power consumption. However, existing methods often fail to fully utilize the spatial information in event data due to its sparsity and the presence of random activations. To address this, we propose a novel Spectral-guided Spatial Enhancement Network (SSE-Net). SSE-Net introduces two core innovations: the Spectrum-guided Spatial Feature Enhance Module (SSEM) and the Multi-Scale Spatial Interaction Module (MS-SIM). SSEM employs frequency domain enhancement and spatial feature enhancement strategies to augment spatial features crucial for event-based lipreading tasks. MS-SIM conducts the fusion and interaction of multi-level semantics, enriching the contextual information of lip representations. We conducted experiments on the event-based lip-reading dataset DVS-Lip with our proposed method and demonstrated its superiority over other state-of-the-art event-based lip-reading methods.