Problems of detecting fuzzy duplicates

Abstract

This article discusses the problem of detecting fuzzy duplicates. Recently, much attention has been paid to the development of methods for reducing the computational complexity of the algorithms being created by choosing various heuristics. With the use of approximate approaches is observed a decrease in the detection rate of duplicates. An important factor affecting the accuracy and completeness of the duplicates definition in comparison problems is the selection of the substantive part. Another key requirement for the quality of detection algorithms for fuzzy duplicates is their resistance to "small" data changes and the ability to process them. One of the first studies in the field of finding fuzzy duplicates is the work of U. Manber and N. Heintze. In these works, sequences of adjacent letters are used to construct the sample. Dactogram includes all text substrings of a fixed length. Completeness, accuracy and F-measure were chosen as the main indicators of the quality of the algorithms. It was supposed to compare the algorithms by these parameters, and also to determine their mutual correlation and joint coverage by different algorithms combinations of the initial set of pairs for the fuzzy duplicates.