Rate of convergence of over-parametrized deep neural network regression estimates learned by stochastic gradient descent

Abstract

Nonparametric regression with random design is considered. The \(L_2\) error with integration with respect to the design measure is used as error criterion. Over-parametrized deep neural network estimates are defined with logistic activation function where all parameters are learned by stochastic gradient descent. It is shown that the estimates achieve a nearly optimal rate of convergence in case that the regression function is (p, C)–smooth. In case that the regression function satisfies a projection pursuit model or more generally a hierarchical composition model the estimate achieves a rate of convergence which does not depend on the input dimension.