Hardware efficient approximate sigmoid activation function for classifying features around zero

Abstract

The binary classification of features around zero in an RNN-LSTM network requires accurate sigmoid activation. The approximate sigmoid activation function is preferred to reduce the computational complexity and hardware resources. Therefore, an IMDB dataset is considered for the Python-based data analysis, the features are passed through the LSTM layer, the dense layer, and finally the sigmoid activation function for binary classification. From the analysis, an approximate 3-term, 8-segment Taylor series sigmoid (${\sigma }_{T\_3\_8}\left(x\right))$ is proposed with an 11-bit customized floating-point (CFP) and provides sufficient accuracy. The ${\sigma }_{T\_3\_8}\left(x\right)$ is implemented with an efficient range select controller, data scheduler and area-efficient arithmetic processing unit (APU). The APU is implemented with a CFP multiplier (CFP-Mul) and Exponent-aware CFP adder (EACFP-Add). Therefore, the FPGA implementation uses fewer hardware resources (LUT, FF and DSP) and obtained 1658 μm² and 0.3305 mW power at 500 MHz in TSMC 65 nm ASIC implementation. This proposed function ${\sigma }_{T\_3\_8}\left(x\right)$ is used in the LSTM cell and classification layer. With the IMDB and SMS spam detection datasets, it provides near-classification metrics compared to the exact σ(x).