Scalable Hardware Accelerator for Mini-Batch Gradient Descent

Abstract

Iterative first-order methods that use gradient information form the core computation kernels of modern statistical data analytic engines, such as MADLib, Impala, Google Brain, GraphLab, MLlib in Spark, among others. Even the most advanced parallel stochastic gradient descent algorithm, such as Hogwild is not very scalable on conventional chip multiprocessors because of the bottlenecks induced by the memory system when sharing large model vectors. We propose a scalable architecture for large scale parallel gradient descent on a Field Programmable Gate Array (FPGA) by taking advantage of the large amount of embedded memory in modern FPGAs. We propose a novel data layout mechanism that eliminates the need for expensive synchronization and locking of shared data, which makes the architecture scalable. A 32-PE system on the Stratix V FPGA shows about 5x improvement in performance compared to state-of-the-art implementation on a 14 core/28 thread Intel Xeon CPU with 64 GB memory and operating at 2.6 GHz.