Predicting the Change on Stock Market Index Using Emotions of Market Participants with Regularization Methods

Abstract

Stock market index as the composite of a series of representative stocks plays a very crucial role in the financial market. Predicting the change of stock market index is vital for investors and stock holders to capture the trend of stocks which they are interested. Recently research from behavioral finance suggests that emotions of market participates can influence stock market index. However, variable selection becomes a major challenge. Normally, lots of key words related to emotions can be extracted from the social media, meaning that the number of predictor variables p for the data mining methods is very large. Traditional variable selection methods require that the number of observations n is sufficient lager and regularization methods could select variables for high dimensional conditions. However, it is common that n is close to p when analyzing the emotions data within a specific time period. Under this condition, both variable selection methods are applicable, but few research has been done on it. In this paper, we compare the traditional variable selection method with the regularization method under the condition that n is close to p. Then we apply typical data mining methods to predict the SSE Composite Index in China with the selected variables. The results show that the regularization methods give much better performance compared with traditional variable infliction factor (VIF) analysis.