Leveraging Continuously Differentiable Activation for Learning in Analog and Quantized Noisy Environments

Abstract

Real-world analog systems, such as photonic neural networks, intrinsically suffer from noise that can impede model convergence and accuracy for a variety of deep learning models. In the presence of noise, some activation functions behave erratically or even amplify the noise. Specifically, ReLU, an activation function used ubiquitously in digital deep learning systems, not only poses a challenge to implement in analog hardware but has also been shown to perform worse than continuously differentiable activation functions. In this paper, we demonstrate that GELU and SiLU enable robust propagation of gradients in analog hardware because they are continuously differentiable functions. To analyze this cause of activation differences in the presence of noise, we used functional interpolation between ReLU and GELU/SiLU to perform analysis and training of convolutional, linear, and transformer networks on simulated analog hardware with different interpolated activation functions. We find that in ReLU, errors in the gradient due to noise are amplified during backpropagation, leading to a significant reduction in model performance. However, we observe that error amplification decreases as we move toward GELU/SiLU, until it is non-existent at GELU/SiLU demonstrating that continuously differentiable activation functions are $\sim 100\times$ more noise-resistant than conventional rectified activations for inputs near zero. Our findings provide guidance in selecting the appropriate activations to realize reliable and performant photonic and other analog hardware accelerators in several domains of machine learning, such as computer vision, signal processing, and beyond.