Deep Learning-Driven Decision Fusion: Spatio-Spectrogram Features for Inner Speech Recognition From Electroencephalogram Signals

Abstract

Inner speech recognition using electroencephalogram (EEG) signals shows strong potential for developing assistive communication technologies. Existing methods often process spatial and temporal features separately, lack interpretability, and are usually tested on a single dataset, limiting their generalization. This study proposes a dual-branch deep learning framework that combines spatial features extracted through common spatial patterns (CSPs) with spectral-temporal features derived from multitaper spectrograms, using convolutional and long short-term memory networks. The model was evaluated on two public datasets, achieving classification accuracies of 89.99% and 92.47% in subject-dependent experiments. Subject-independent evaluation using leave-one-subject-out cross-validation yielded reduced accuracies of 26.20% and 20.47%, reflecting intersubject variability. Interpretability analyses using saliency maps, gradient-weighted class activation mapping, and feature contribution ratios highlighted physiologically meaningful patterns related to model decisions. The proposed method demonstrates strong performance and interpretability for subject-dependent inner speech recognition; while future work will focus on increasing data diversity and improving subject-independent generalization. This study contributes to the development of reliable and explainable EEG-based inner speech decoding for communication applications.