A Distributed PCM Clustering Algorithm Based on Spark

Abstract

With the large-scale growth of data, traditional single-machine data processing methods are difficult to deal with massive data, especially iterative clustering algorithms that require frequent reading and writing operations. On the basis of Spark framework, this paper proposes a distributed possibilistic c-means algorithm based on memory computing, called Spark-PCM. The proposed method improves the related processing of distributed matrix operation and is implemented on the Spark platform. Experimental results show that the proposed Spark-PCM algorithm runs in a linear relationship with the number of nodes and has a good scalability, which indicates that it has higher scalability and adaptability to large-scale data.