Hardware-Optimized Regression Tree-Based Sigmoid and Tanh Functions for Machine Learning Applications

Abstract

The sigmoid and $hyperbolic\ tangent~(tanh)$ functions are widely recognized as the most commonly employed nonlinear activation functions in artificial neural networks. These functions incorporate exponential terms to introduce nonlinearity, which imposes significant challenges when realized on hardware. This brief presents a novel approach for the hardware implementation of sigmoid and tanh functions, leveraging a regression tree and linear regression. The proposed method divides their nonlinear region into small segments using a regression tree. These segments are further approximated using a linear regression technique, the line of best fit. Experimental results demonstrate the average errors of $4\times 10^{-4}$ and $9\times 10^{-4}$ of sigmoid and tanh functions compared to exact functions. The above functions produce 24.52% and 35.71% less average error than the best contemporary method when implemented on the hardware. Additionally, the hardware implementations of sigmoid and tanh functions are more area, power and delay efficient, showcasing the effectiveness of this method compared to other state-of-the-art designs.