Approximating positive homogeneous functions with scale invariant neural networks

Abstract

We investigate the approximation of positive homogeneous functions, i.e., functions $f$ satisfying $f\left(\lambda x\right)=\lambda f\left(x\right)$ for all $\lambda \ge 0$, with neural networks. Extending previous work, we establish new results explaining under which conditions such functions can be approximated with neural networks. As a key application for this, we analyze to what extent it is possible to solve linear inverse problems with $ReLu$ networks. Due to the scaling invariance arising from the linearity, an optimal reconstruction function for such a problem is positive homogeneous. In a $ReLu$ network, this condition translates to considering networks without bias terms. For the recovery of sparse vectors from few linear measurements, our results imply that $ReLu$ networks with two hidden layers allow approximate recovery with arbitrary precision and arbitrary sparsity level $s$ in a stable way. In contrast, we also show that with only one hidden layer such networks cannot even recover 1-sparse vectors, not even approximately, and regardless of the width of the network. These findings even apply to a wider class of recovery problems including low-rank matrix recovery and phase retrieval. Our results also shed some light on the seeming contradiction between previous works showing that neural networks for inverse problems typically have very large Lipschitz constants, but still perform very well also for adversarial noise. Namely, the error bounds in our expressivity results include a combination of a small constant term and a term that is linear in the noise level, indicating that robustness issues may occur only for very small noise levels.