MDCNN: A multimodal dual-CNN recursive model for fake news detection via audio- and text-based speech emotion recognition

Abstract

The increasing complexity and diversity of emotional expression pose challenges when identifying fake news conveyed through text and audio formats. Integrating emotional cues derived from data offers a promising approach for balancing the tradeoff between the volume and quality of data. Leveraging recent advancements in speech emotion recognition (SER), our study proposes a Multimodal Recursive Dual-Convolutional Neural Network Model (MDCNN) for fake news detection, with a focus on sentiment analysis based on audio and text. Our proposed model employs convolutional layers to extract features from both audio and text inputs, facilitating an effective feature fusion process for sentiment classification. Through a deep bidirectional recursive encoder, the model can better understand audio and text features for determining the final emotional category. Experiments conducted on the Interactive Emotional Dyadic Motion Capture (IEMOCAP) dataset, which contains 5531 samples across four emotion types—anger, happiness, neutrality, and sadness—demonstrate the superior performance of the MDCNN. Its weighted average precision (WAP) is 78.8 %, which is 2.5 % higher than that of the best baseline. Compared with the existing sentiment analysis models, our approach exhibits notable enhancements in terms of accurately detecting neutral categories, thereby addressing a common challenge faced by the prior models. These findings underscore the efficacy of the MDCNN in multimodal sentiment analysis tasks and its significant achievements in neutral category classification tasks, offering a robust solution for precisely detecting fake news and conducting nuanced emotional analyses in speech recognition scenarios.