On the Smoothing of Deep Networks

Abstract

Many popular deep neural networks implement an input-output mapping that is non-smooth with respect to the network parameters. This non-smoothness may have contributed to the difficulty of analyzing deep learning theoretically. Sophisticated approaches have recently been proposed to address this specific difficulty. In this note, we explore a simple approach consisting instead in smoothing the input-output mapping. We show how to perform smoothing automatically within a differentiable programming framework. The impact of the smoothing on the convergence behavior can then be automatically controlled. We illustrate our approach with numerical examples using multilayer perceptrons.