Impact of Stemming on Efficiency of Messages Likelihood Definition in Telegram Newsfeeds

Abstract

The work is dedicated to the development of the system to define the credibility of text messages posted in Telegram newsfeeds. The topicality of the work is stipulated by the concentration of information and its ability to influence shaping of the social opinions on the state relations and political moods via news feeds in messengers and social networks, the number of which is constantly growing and supported by bots and biased authors. The proposed system functions on the basis of coordination of text parsing, text processing, database with messages from the official sources of information, and the client (author) database. The degree of similarity of the generated text messages is determined on the basis of defining Damerau-Levenshtein distance in the Text Processing Module. The work shows it is possible to increase the efficiency (up to 1,44 times for messages of around 1500 symbols) of the given module performance through incoming messages stemming at the preprocessing stage because this enables to reduce the computational complexity of Damerau-Levenshtein method at the expense of word shortening to their stem via neglecting auxiliary parts such as suffixes and endings. Thus, stemming helps to reduce the amount of symbols to be processed at the very stage of Damera u-Levenshtein algorithm application, which proves feasibily of applying stemming in the preprocessing block.