A novel hierarchical attention-guided refinement method with EEG assistance for enhancing target speech in a multi-speaker competing environment

Abstract

Enhancing target speech in noisy, multi-speaker environments is a critical challenge, particularly in engineering contexts, such as construction sites, factories, and transportation systems, where multi-source competing speech scenarios are common and the need for efficient speech enhancement is critical to ensuring safety and operational effectiveness. The latest research is prone to recovering auditory attention with brain activity assistance. However, existing methods emerged with the challenges of multimodal feature extraction bottleneck, and fusion bottleneck. To address these challenges, this paper proposes a hierarchical attention-guided refinement network for enhancing EEG-assisted speech (HierEEG). HierEEG is an end-to-end explainable time-domain model comprising three core modules: a Multi-Scale Feature Modulation Refinement (MFMR) module, a Hierarchical Attention Fusion (HAF) network, and a Lightweight Speech Decoder. The first module learns the different granularities of feature representations and facilitates the interaction between short-term and long-term features through a feature modulator, obtaining multi-scale refined speech embeddings and EEG features. Then, the second module hierarchically guides the model’s attention focusing on high-level semantic features, outputting the generation of clean speech mask embeddings. Finally, a lightweight speech decoder is used to reconstruct the clean speech sample. Our comprehensive experiments on comparison, ablation, subject-dependent, subject-independent, transfer-learning, engineering, and calculation-cost experiments show that our proposed framework, HierEEG, outperforms state-of-the-art methods on mainstream Cocktail Party Datasets, especially achieving relative improvements of 0.21 dB and 0.15 in SI-SDR and PESQ. The proposed HierEEG validates the robustness in engineer simulated experiment, over 10 dB accuracy even with the various noises, artifacts, and poor contact. Furthermore, HierEEG makes great transfer performance for personalized user-specific adjustments, with simply 12 min of fine-tuning samples. HierEEG’s efficient processing and low computational cost, with under 70 % inference utilization on the Jeston Nano embedding device, enhances the potential applications in multi-speaker competing environments. Finally, the brain region experiment demonstrates the explainability of HierEEG, which ensures that the decisions made by the HierEEG can be understood in the context of the brain’s functional organization.