HSB-GDM: a Hybrid Stochastic-Binary Circuit for Gradient Descent with Momentum in the Training of Neural Networks

Abstract

To enable an energy-efficient training of neural networks, this paper proposes a hybrid stochastic-binary (HSB) computing circuit for implementing the gradient descent with momentum (GDM) algorithm. By accumulating the weight-update values step by step, the proposed design executes the weight optimization of a neural network. At each step, the weight-update value is obtained by a linear combination of its previous value and the current gradient. In this design, it is computed in a hybrid stochastic-binary manner and encoded as a dynamic stochastic sequence consisting of 0, +1 and -1. Then, the weights are updated by accumulating the bits in the dynamic stochastic sequence. With the hybrid stochastic-binary design, this circuit can be readily integrated into a neural network accelerator to support online training with a small footprint. Experimental results show that, with little accuracy loss, the area efficiency of the proposed HSB-GDM is improved by 2.68× and energy efficiency by 4.41× compared to a floating-point design using bfloat16 data format.