Multimodal stock market emotion recognition model trained with a large language model

Abstract

Stock market emotion recognition models are often trained on public textual datasets. These datasets are well-labelled but may not have the same distribution as real-world data and consequently fail to reflect the real world. Furthermore, stock market emotion is reflected not only in textual comments but also in images; emotion recognition on the basis of textual comments alone is one-sided. Considering these issues, in this paper, we propose a multimodal emotion recognition framework that is trained via imitation learning from a large language model. The proposed framework contains two main innovations. First, by leveraging large language model's compositionality capability, the proposed framework generates pseudo labels from textual comments. Through imitating the patterns of the pseudo labels, the framework can be trained directly on unlabelled real-world data, which addresses current models' distribution drift between public datasets and the real-world. Second, multimodal fusion is equipped with the proposed framework, enabling emotion recognition from textual stock market comments and images simultaneously. Compared with existing methods, the proposed method in this paper significantly improves the performance of stock market emotion recognition by leveraging large language models and multimodal stock market data. The experimental results demonstrate that the emotion recognition framework proposed in this paper outperforms the existing single-mode models and multimodal models, with the accuracy, precision, recall, and F1 score being 82.97 %, 83.03 %, 83.05 %, and 82.88 % respectively. This framework provides a new pathway for multimodal and unsupervised/semi-supervised emotion recognition in the stock market.