Brain Network Features Differentiate Intentions from Different Emotional Expressions of the Same Text

Abstract

Intent differentiation in speech communication relies not only on linguistic information but also on paralinguistic information. The same textual content, when pronounced with different prosodies and emotions, may express totally different intentions. The true intentions in this condition can be easily grasped by our brain. Therefore, combining text, speech, and electroencephalography (EEG) for intent discrimination on the same text may be an effective approach. Before fusing speech and text modalities, the current study focused on exploring effective EEG-based features for Chinese intent recognition as no previous research has utilized EEG signals for this purpose. To tackle this issue, we first created a Chinese multimodal spoken language intention understanding (CMSLIU) dataset, in which the same texts were pronounced with varying prosodies to express different intents. To identify effective brain features that were most relevant to intent recognition improvement, we compared the event-related spectral perturbation and effective brain connectivity patterns on two intent conditions (praise vs. irony). It was found that the praise expression tended to elicit stronger high-frequency brain activities while the irony expression involved a more suppressive network connection in the right hemisphere. These features were trained on the CMSLIU dataset and achieved an intention classification accuracy of 78.66%, which indicated a great potential of the EEG features in intent discrimination on the same text.