Load Balancing Algorithms for Comparative Pattern Mining

Abstract

For addressing the issues of ineffective mining and memory overflow when dealing with high-dimensional and large-scale datasets with traditional comparative pattern mining, and to further lift the limitation of a single machine’s own hardware, the study proposes a parallel comparative pattern mining algorithm based on Spark cluster environment. By constructing an extended data collection project tree, introducing an optimised decision tree for mining, and improving the related load balancing strategy, effective mining of large-scale and high-dimensional datasets is achieved. Experiments show that the algorithm proposed in the study has a maximum value of 1883 and a minimum value of 1549 for the number of contrasting patterns mined in the small-scale and low-dimensional Mushroom dataset, which is slightly higher than the mining method of strong jump revealed patterns with good classification performance. In the large-scale and high-dimensional dataset US census1990, the overall running time of the algorithm of the study is low compared to the cryptogrowth algorithm (T_max 43.2 min, T_min 18.4 min), and finally the failure request rate of the algorithm itself and the improved and weighted polling algorithms are ompared separately, and the results show that the improved algorithm takes the lowest time of 4%. The experiment showcases that the classification effect of the studied algorithm is good, the load balancing strategy of the improved algorithm is effective, and the overall performance of the algorithm is good.