Efficient Algorithms for Preprocessing and Stemming of Tweets in a Sentiment Analysis System

The preprocessing step approximately consumes 85% of the time and efforts of overall time and efforts of the Knowledge Discovery in Database, (KDD). Sentiments analysis, as a new trend in KDD and data mining, requires many preprocessing steps such as tokenization, stop words removing, and stemming. These steps play, in addition to their preparation role, the data reduction role by excluding worthless data and preserving significant data. This paper presents the design and implementation of a system for English tweets segmentation, cleaning, stop words removing, and stemming. This system implemented as MS-SQL Server stored procedures to be part of a tightly coupled sentiments mining system. Many experiments accomplished to prove the validity and efficiency of the system using different sizes data sets arranged from 250000 to 1000000 tweets and it accomplished the data reduction process to achieve considerable size reduction with preservation of significant data set's attributes. The system exhibited linear behavior according to the data size growth.