A Novel Framework for Agricultural Futures Price Prediction With BERT-Based Topic Identification and Sentiment Analysis

In China's financial and economic system, the agricultural futures market plays an important role in guiding the market to self regulate and providing efficient information transmission for regulators. The effective prediction of futures prices can assist in guiding agricultural production, monitoring operational risks arising from significant price fluctuations, and enhancing the predictability and pertinence of the country's macroeconomic regulation policies. This study investigates the main variety of grain futures—soybean futures, taking into account complex market and non-market influencing factors. Using historical market data and related news headlines of soybean futures as source data and integrating topic identification and sentiment analysis techniques, a novel framework for predicting agricultural futures prices that integrates topic sentiment is constructed. This model uses BERTopic to extract topic information from agricultural news texts, then integrates FinBERT to construct topic-based sentiment features, fuses them with structured market features, and constructs LSTM price prediction model with multi-feature inputs. In order to better model the short-term features and state transfer patterns of the time series, hidden Markov model (HMM) is further used to extract the hidden states, which are deeply fused with the LSTM model. The empirical results show that the model fusing topic and sentiment features significantly improves the forecasting accuracy in all lags, LSTM works best in short-term forecasting, and the combination of HMM and LSTM exhibits significant performance advantages in medium- and long-term forecasting. Compared with the baseline model that relies only on market features, topic sentiment features provide important incremental information for price forecasting, and the contribution of each topic sentiment feature calculated based on the PI metric is close to 50%. In addition, deep learning–based prediction model performs better than baseline machine learning models in dealing with extreme external shocks such as climate disasters, the COVID-19 pandemic, and the Russia–Ukraine conflict.