Towards Stochastic Neural Network via Feature Distribution Calibration

Stochastic neural network (SNN) has attracted increasing attention in recent years, which benefits several important tasks by modeling samples uncertainly, such as adversarial defense, label noise robustness, and model calibration. The current implementations of existing stochastic neural networks are mainly Gaussian noise injection, e.g., deep Variational Information Bottleneck (VIB) uses fixed Gaussian prior to derive noise injection, simple and effective stochastic neural network (SE-SNN) uses a non-informative Gaussian prior to implement it. However, Gaussian distribution assumption is insufficient to model more complex distributions of data in practical, such as the skewed distribution or multi-modal distribution. In this paper, we relax the strict Gaussian prior assumption, and propose a novel distribution calibrated stochastic neural network (DCSNN) which integrates two successive steps. These two steps are as follows: 1) The trained feature vector is preprocessed to make its feature distribution closer to the Gaussian-like distribution. 2) Gaussian distribution’s mean and variance are used to model the sample’s activation indeterminacy. The experimental results show that, compared with the existing methods, our proposed method can achieve state-of-the-art results in a variety of datasets, backbone architectures and multiple applications.