Computing PageRank Scores of Web Crawl Data Using DGX A100 Clusters

Abstract

PageRank is a widely used graph analytics algorithm to rank vertices using relationship data. Large-scale Page Rank is challenging due to its irregular and communication intensive computational characteristics. We implemented Page Rank on NVIDIA's newly released DGX A100 cluster and compared the performance with two recent notable large-scale Page Rank computations using the Common Crawl dataset. The ShenTu framework computed Page Rank scores using a large number of custom microprocessors connected with an HPC class interconnect. The Hronos framework reported the state-of-the-art performance using 3000 commodity CPU nodes and 10 Gbps Ethernet. The Common Crawl dataset captures link relationships between web pages in a graph with 3.563 billion vertices and 128.736 billion edges. Our implementation demonstrated 13x faster PageRank iteration time than the Hronos framework using a cluster with NVLink connected 32 A100 GPUs.