BiPS: Hotness-aware Bi-tier Parameter Synchronization for Recommendation Models

While current deep learning frameworks are mainly optimized for dense-accessed models, they show low throughput and poor scalability in training sparse-accessed recommendation models. Our investigation shows that the poor performance is due to the parameter synchronization bottleneck. We therefore propose BiPS, a bi-tier parameter synchronization system that alleviates the parameter update and the sparse-accessed parameters communication bottleneck. BiPS includes a bi-tier parameter server that accelerates the traditional CPU-based parameter update process, a hotness-aware parameter placement and communication policy to balance the workloads between CPU and GPU and optimize the communication of sparse-accessed parameters. BiPS overlaps the worker computation with the synchronization stage to enable parameter updates in advance. We implement BiPS and incorporate it into mainstream DL frameworks including TensorFlow, MXNet, and PyTorch. The experimental results based on various deep learning frameworks show that BiPS greatly speeds up the training of recommenders (5 - 9$\times$) as the model scale increases, without degrading the accuracy.