Increase the Speed of Search Index in the Duplicate Text Detection Systems

Abstract

The work is devoted to the organization of the search index of the duplicate text detection systems Author.NET. The paper considers the structure of the search index. The search index can be divided into several groups of files - terms, documents, TF*IDF index, signatures, shingles. In the article the ways of organizing the search index are considered - index storage in RAM, compression of index file, tree index, index table of contents. It is given a proposal for organizing a search index for duplicate text detection system. During the experiments, it was found that the most profitable option is to use compressed index files stored on the SSD.