Modeling and Verification of MRSCAN Based on MapReduce Framework

Abstract

Network clustering (graph clustering) plays a crucial role in discovering the inherent structures within networks. MapReduce-based structural clustering algorithm for networks (MRSCANs) designed based on MapReduce's parallel computing model, efficiently handles large-scale data. However, MRSCAN can only be tested through experiments, and its correctness cannot be guaranteed. To address this issue, this article has achieved the first implementation of functional MRSCAN modeling and subjected it to rigorous mechanized verification in Isabelle. First, based on Google's MapReduce model type definition and higher-order generic functions, a general MapReduce-based algorithm functional modeling framework is constructed across the fundamental global phases of Map, Shuffle, and Reduce. Moreover, diverse strategies are devised during the Shuffle phases according to user requirements, enhancing the applicability and generality of the MapReduce functional modeling framework. Second, formalizing the definition of MRSCAN, is delineated into four key steps: similarity calculation, core calculation, dimension expansion, and structural clustering. Furthermore, the MapReduce functional modeling framework is applied to these four steps to achieve the functional modeling of MRSCAN, which improves efficiency compared to other structural clustering algorithm for networks (SCANs) algorithms. Lastly, a verification framework for MapReduce-based algorithms is proposed at both the global and shuffle stages. Based on this framework, the correctness and reliability of MRSCAN are ensured. The model framework and verification framework of MapReduce-based algorithms proposed in this article can not only address functional modeling and verification of MRSCAN but also provide a reference for a series of other MapReduce-based functional program designs and proofs.