Implementation of GraphFrames-Based Parallelized Label Propagation Algorithm in Clusters

In the era of big data, the number of network users has exploded, the number of network nodes has increased, and the association relationships between nodes have become more intricate. Ordinary university students who lack a big data experimental environment have been unable to use the traditional label propagation algorithm to deal with large-scale complex network data efficiently. To solve these problems, this paper proposes a parallelized label propagation algorithm based on GraphFrames. Firstly, a multi-node big data cluster environment is built by using the existing computer room resources of universities, and GraphFrames is used to parallelize the label propagation algorithm in the cluster environment. Experiments show that the parallelized label propagation algorithm based on GraphFrames can easily cope with large-scale complex networks with millions of data nodes. The relationship between the running time of the algorithm and the number of nodes in the cluster is explored by varying the number of nodes in the cluster; In terms of the community division effect of the algorithm, the F _Measure value of the large-scale complex network with one million levels can be stably maintained at about 60%, and the F _Measure value of the small-scale real social network is improved by 20% compared with other traditional community discovery algorithm.