A 2-Tier Clustering Algorithm with Map-Reduce

Abstract

In the field of data mining, clustering is one of the important methods. K-Means is a typical distance-based clustering algorithm; 2-tier clustering should implement scalable clustering by means of dividing, sampling and knowledge integrating. Among those tools of distributed processing, Map-Reduce has been widely embraced by both academia and industry. Hadoop is an open-source parallel and distributed programming framework for the implementation of Map-Reduce computing model. With the analysis of the Map-Reduce paradigm of computing, we find that Hadoop parallel and distributed computing model is appropriate for the implementation of scalable clustering algorithm. This paper takes advantages of K-Means, 2-tier clustering mechanism and Map-Reduce computing model; proposes a new method for parallel and distributed clustering to explore distributed clustering problem based on Map-Reduce. The method aims to apply the clustering algorithm effectively to the distributed environment. The extensive studies demonstrate that the proposed algorithm is scalable, and the time performance is stable. Meanwhile, adding number of cluster nodes would improve the time performance of clustering.