A Dynamic ReLU on Neural Network

Abstract

In this paper we propose a dynamic Rectified Linear Unit (D-ReLU) activation function for a multi-layer perceptron (MLP) learning network. We also implement the forward propagation of 2- and 3-layer multi-layer perceptron (MLP) networks with this D-ReLU function on a Cyclone IVE field programmable gate array (FPGA) using 8-bit precision. When compared to networks that use the approximated Sigmoid activation function, our proposed D-ReLU function uses 18-23% less area with only a 0.7-2.9% loss in accuracy. Moreover, the simplified calculations of the D-ReLU function result in 14% and 57% decreases in software execution time than Sigmoid function. In the FPGA hardware implementation, the D-ReLU function uses two fewer clock cycles per layer than the approximated Sigmoid activation function. Thus, using the D-ReLU activation function in MLP networks results in reduced area on an FPGA and lower execution time in software. In addition, the FPGA implementation runs at a 60× lower clock rate than the software version with a 29× faster execution time, indicating a potential of over 1,700× power savings.