A Parallel Uncertain Frequent Itemset Mining Algorithm with Spark

Abstract

Frequent Itemset Mining (FIM) from large-scale databases has emerged as an important problem in the data mining and knowledge discovery research community. However, FIM suffers from three important limitations with the rapidly expanding of big data in all domains. First, it assumes that all items have the same importance. Second, it ignores the fact that data collected in a real-life environment is often inaccurate. Third, it is also a data-intensive and computation-intensive process which makes the FIM algorithm very time-consuming over large datasets. To address these issues, we propose a Parallel uncertain frequent itemset mining algorithm with spark (Pufim). Pufim firstly expresses item uncertainty by considering both the probability and weight, and calculates the maximum probability weight value of 1-items. Next, a distributed Pufim-tree structure is designed inspiring by FP-Tree for reducing the times of scanning the databases. Each node of Pufim-tree stores an item and its maximum probability weight value. Finally, experiments on publicly available UCI datasets demonstrate that Pufim achieves more prominent results than other related approaches across various metrics. In addition, the empirical study also shows Pufim has a good scalability