Big data clustering method based on parallel K-means

Abstract

In the era of big data, traditional data clustering algorithms have gradually failed to meet the application requirements, and the optimization of data compression and parallelization methods has become a research hotspot. Based on the analysis of the traditional K-means clustering algorithm, this paper optimizes and improves the parallelized K-means algorithm, and proposes the Spark-Kmeans algorithm, which mainly retains the sample set distribution information by random sampling of large samples, and pre-clusters the samples in the nodes, and reclusters the pre-clustering in the convergence node. And it uses this as the initialization clustering center, so as to eliminate the problem of algorithm convergence instability caused by random initialization of the clustering center. Finally, single-node clustering and Spark-Kmeans clustering experiments are performed on the kdd_cup99 dataset and sklearn randomly generated dataset, and the effectiveness of the algorithm is verified by time-consuming, purity, error squared and indexes.