Learning and approximating piecewise smooth functions by deep sigmoid neural networks

Abstract

Constructing neural networks for function approximation is a classical and longstanding topic in approximation theory, so is it in learning theory. In this paper, we are going to construct a deep neural network with three hidden layers using sigmoid function to approximate and learn the piecewise smooth functions, respectively. In particular, we prove that the constructed deep sigmoid nets can reach the optimal approximation rate in approximating the piecewise smooth functions with controllable parameters but without saturation. Similar results can also be obtained in learning theory, that is, the constructed deep sigmoid nets can also realize the optimal learning rates in learning the piecewise smooth functions. The above two obtained results underlie the advantages of deep sigmoid nets and provide theoretical assessment for deep learning.