Efficient Implementation of Activation Function on FPGA for Accelerating Neural Networks

In this paper, we present the Integer Lightweight Softmax (ILS) algorithm for approximating the Softmax activation function. The accurate implementation of Softmax on FPGA can be huge resource-intensive and memory-hungry. Then, we present the implementation of ILS on a Xilinx XCKU040 FPGA to evaluate the effectiveness of ILS. Evaluations on CIFAR 10, CIFAR 100 and ImageNet show that ILS achieves up to $2.47\times, 40\times$ and $323\times$ speedup over CPU implementation, and $4\times, 63\times$ and $51\times$ speedup over GPU implementation, respectively. In comparison to previous FPGA-based Softmax implementations, ILS strikes a better balance between resource consumption and precision accuracy.