A Fast Distributed Principal Component Analysis with Variance Reduction

Principal Component Analysis (PCA) and Singular Value Decomposition (SVD) are widely used in dimension reduction, feature extraction, low-rank matrix approximation and so on. In large-scale applications, a common alternative is to use cluster which have multiple nodes and multiple cores to accelerate the time of solving problem. Most of the existing distributed PCA algorithm focus on reducing the communication, and there is little attention to the frequent waiting phenomenon caused by the synchronization mechanism. Meanwhile, most of these works are based on either the distributed memory of the processes-level parallelism or the shared memory of threads-level parallelism. In this paper, we propose a fast distributed PCA algorithm with variance reduced, which based on stochastic sampling and Stale Synchronous Parallel. Our algorithm contains the processes-level and threads-level parallelism. Experiments on the "Tianhe-2" super computer demonstrate that our algorithm has a good performance, speedup, and scalability.