Efficient Dense Structure Mining Using MapReduce

Abstract

Structure mining plays an important part in the researches in biology, physics, Internet and telecommunications in recently emerging network science. As a main task in this area, the problem of structure mining on graph has attracted much interest and been studied in variant avenues in prior works. However, most of these works mainly rely on single chip computational capacity and have been constrained by local optimization. Thus it is an impossible mission for these methods to process massive graphs. In this paper, we propose an unified distributed method in solving some critical graph mining problems on top of a cluster system with the help of MapReduce. These problems include graph transformation, subgraph partition, maximal clique enumeration, connected component finding and community detection. All of these methods are implemented to fully utilize MapReduce execution mechanism, namely the “map-reduce” process. Moreover, considering how our algorithms can be applied in further “cloud” service, we employ several large scale datasets to demonstrate the efficiency and scalability of our solutions.