Microblogging Hash Tag Recommendation System Based on Semantic TF-IDF: Twitter Use Case

Limitation in the number of characters in microblogging systems, such as Twitter, forces users to use various terms for the same meaning, object, or concept. Sometimes the same term is used in a shorter form (e.g. #friend and #frnd) in a tweet. This problem makes finding similarities between such tags and their corresponding tweets harder. The classical text mining methods cannot be used efficiently in the short tweets. Thus tweets similarity and subsequently tag recommendation, as one of the problems in microblogging social networks, needs a new method with higher efficiency. In this paper we have defined a new semantic based method to find similarities among short messages. We have modeled each short message as a semantic vector which can be used along with any similarity method such as cosine similarity. Then we evaluated the accuracy of the new semantic similarity based tag recommendation system using various semantic based algorithms and compare their results. The semantic based algorithms used are: Shortest Path, Wu & Palmer, Lin, JiangConrath, Resnik, Lesk, LeacockChodorow, and Hirst-StOnge. Results are evaluated using 8396744 real English tweets and show around 6 times improvement in accuracy over normal TF-IDF.