Detection of Fuzzy Duplicate Texts in News Feeds

Abstract

The paper is devoted to the problem of fuzzy duplicate texts detection in news feeds. The signature methods of detecting fuzzy duplicate news are considered. Signatures describe the content of a news as one or a group of numbers. It is proposed to use Description words big signature. This signature consist of set of flags for the presence of description words and vector of names. This vector include names of objects, countries, names of politicians. It allow setting the exact direction of the news on this or that event. In paper the results of testing the signature methods are given. Proposed signature showed good results both in recall and in precision of duplicate news detection.