Identification of Key Sentences in the Task of Text Duplicate Detection

Abstract

The paper considers the problems of detecting duplicates of large text documents. To reduce the verification time, it is proposed to submit the document being verified with a set of key sentences. Key sentences are selected for parts of the text and are used to search for matches over the Internet. As a criterion for choosing key sentences, the largest sum of the weights of the words included in the sentence is calculated, taking into account the global frequency of words. Studies have shown that the use of key offers can significantly reduce the number of queries to search engines. At the same time, good duplicate text detection results are preserved.