Similar Duplicate Record Detection of Big Data Based on Entropy Grouping Clustering

Abstract

At present, the similar duplicate records of massive data cannot be detected effectively by current methods, an algorithm of Property Entropy Grouping Clustering is proposed (EGC). The basic idea constructs an entropy metric based on similarity between objects, the importance of each property can be evaluated and a key property subset can be obtained. According to the key property to split the data sets into small data sets, the similar duplicated records are identified based on the algorithm of Sorted-Neighborhood Method. The theory an alysis and experiments show that identification accuracy and detection efficiency of the method are higher and it can effectively solve the problems of identification in similar duplicate records of the big data set.