A low-power, high accuracy digital design of batch normalized non-linear neuron models: Synthetic experiments and FPGA evaluation

The Batch-Normalization (BN) technique has gained significant attention in Neural Networks (NNs) due to its ability to mitigate the vanishing gradient problem, leading to slow learning rate in early epochs, complicating Activation Functions (AFs) optimization which significantly affects the NNs performance. This paper ultimately focuses on the Batch-Normalized Non-linear Neuron Models (BN-NLN) like Logistic, Softmax, LeakyReLU, Swish, TanH, ELU, SELU and APL. Simulations and FPGA implementations confirm that the proposed neurons outperform in terms of resource and interconnect utilization, delay at 10 MHz clock frequency. Notably, BNTANH stands out as a highly efficient low power neuron model. Extensive statistical analysis proves that proposed neuron models like BNLEAKYRELU, BNSWISH, BNTANH, BNELU, and BNSELU achieve impressive accuracy rates of 97 %, 98 %, 98 %, 97 %, and 98 % respectively, confirming the effectiveness of the proposed models in optimizing both power efficiency and computational accuracy.