Training Deep Neural Networks in Low-Precision with High Accuracy Using FPGAs

Abstract

Low-precision training for Deep Neural Networks(DNN) has recently become a viable alternative to standard full-precision algorithms. Crucially, low-precision computation reduces both memory usage and computational cost, providing more scalability for Field Programmable Gate Arrays (FPGAs) with limited on-chip memory. In this paper, we describe and test a prototype training accelerator for Zynq All Programmable System on Chip (APSoC) devices using predominantly 8-bit integer numbers. Block floating-point quantisation and stochastic weight averaging techniques are applied during training toavoid any degradation in accuracy. Results of an implementation reveal memory savings and 17x speed-ups over processor only systems on several training tasks including the MNIST and CIFAR10 benchmarks, and online radio-frequency anomaly detection. Moreover, we propose modifications to the stochastic weight averaging low-precision (SWALP) algorithm to achieve a0.5% accuracy improvement for the abovementioned benchmarks with results within 0.1% of floating-point. We suggest that both inference and training can be deployed in the same package for stand-alone embedded applications.