Implementing BDFS(b) with diff-sets for real-time frequent pattern mining in dense datasets - first findings

Abstract

Finding frequent patterns from databases has been the most researched topic in association-rule mining. Business-intelligence using data mining has felt an increased thrust for real-time frequent pattern mining algorithms finding huge demand from numerous real-time business applications like e-commerce, recommender-systems, group-decision-support-systems, supply-chain-management, to name a few. Last decade has seen development of mind-whelming algorithms, among which, vertical-mining algorithms have been found to be very effective. However, with dense-datasets, the performances of these algorithms significantly degrade. Moreover, these algorithms are not suited to respond to the real-time need. In this paper, we describe BDFS(b)-diff-sets, an algorithm to perform real-time frequent pattern mining using diff-sets and using an intelligent staged search technique, by-passing usual breadth-first and depth-first search-techniques. Empirical evaluations show that our algorithm can make a fair estimation of the probable frequent-patterns reacting to the user-defined time bound and reaches some of the longest frequent patterns much faster than the existing algorithms.