Stock market analysis from Twitter and news based on streaming big data infrastructure

Abstract

Due to the rapid development of the web, services of social media and Internet of Things (IoT) are producing a huge volume of data in every second. This data is not only large, but also grows quickly and is difficult to analyze. Most of traditional big data framework can't process such data in real-time. For processing the data in real-time, many companies and researchers have started to develop new big data frameworks. The Apache Spark, Apache Flink and Apache Storm have been introduced for real-time data processing. With the new processing frameworks, it has become more efficient to analyze the streaming data. Stock market analysis is a hot issued domain to analyze the big streaming data. In this paper, we build a real-time processing system to analyze tweets for finding correlation with the stock market. System configuration, performance of our system is explained. With 77% accuracy of Twitter data classification, we got 80% of separation of increase/decrease of stock value.