Learning sparse and smooth functions by deep Sigmoid nets

Abstract

To pursue the outperformance of deep nets in learning, we construct a deep net with three hidden layers and prove that, implementing the empirical risk minimization (ERM) on this deep net, the estimator can theoretically realize the optimal learning rates without the classical saturation problem. In other words, deepening the networks with only three hidden layers can overcome the saturation and not degrade the optimal learning rates. The obtained results underlie the success of deep nets and provide a theoretical guidance for deep learning.