Finding Correlated Item Pairs through Efficient Pruning with a Given Threshold

Abstract

Given a minimum threshold in a massive market-basket data set, an item pair whose correlation above the threshold is considered correlated. In this paper, we provide a randomized algorithm SERIT-a Searching-corrElated-pair Randomized algorithm for dIfferent Thresholds- to find all correlated pairs effectively, which adopts the Pearson's correlation coefficient [11] as the measure criterion. In their CIKM'06 paper [2], Zhang et al. address the same problem by taking the relation of Pearson's coefficient and Jaccard distance into account. However, it is inefficient when the threshold is small. We propose a new probability function to prune uncorrelated item pairs based on [2], which can cover the shortage of the former one. Experimental results with synthetic and real data sets reveal that with a given threshold, even if it is small, SERIT algorithm can prune the item pairs unwanted efficiently and save large computational resources.