Multimodal speech recognition using EEG and audio signals: A novel approach for enhancing ASR systems

Abstract

Speech recognition using EEG signals captured during covert (imagined) speech has garnered substantial interest in Brain–Computer Interface (BCI) research. While the concept holds promise, current implementations must improve performance compared to established Automatic Speech Recognition (ASR) methods using audio. An area often underestimated in previous studies is the potential of EEG utilization during overt speech. Integrating overt EEG signals with speech data by leveraging advancements in deep learning presents significant potential to enhance the efficacy of these systems. This integration proves particularly advantageous in noisy environments and for individuals with speech impairments—challenges even conventional ASR techniques struggle to address effectively. Our investigation delves into this relationship by introducing a novel multimodal model that merges EEG and speech inputs. Our model achieves a multiclass classification accuracy of 95.39%. When subjected to artificial white noise added to the input audio, our model exhibits a notable level of resilience, surpassing the capabilities of models reliant solely on single EEG or audio modalities. The validation process, leveraging the robust techniques of t-SNE and silhouette coefficient, corroborates and solidifies these advancements.