On the optimal approximation of Sobolev and Besov functions using deep ReLU neural networks

This paper studies the problem of how efficiently functions in the Sobolev spaces $W^{s,q}\left({[0,1]}^d\right)$ and Besov spaces $B_{q,r}^s\left({[0,1]}^d\right)$ can be approximated by deep ReLU neural networks with width W and depth L, when the error is measured in the $L^p\left({[0,1]}^d\right)$ norm. This problem has been studied by several recent works, which obtained the approximation rate $O\left({\left(WL\right)}^{-2s/d}\right)$ up to logarithmic factors when $p=q=\infty$, and the rate $O\left(L^{-2s/d}\right)$ for networks with fixed width when the Sobolev embedding condition $1/q-1/p<s/d$ holds. We generalize these results by showing that the rate $O\left({\left(WL\right)}^{-2s/d}\right)$ indeed holds under the Sobolev embedding condition. It is known that this rate is optimal up to logarithmic factors. The key tool in our proof is a novel encoding of sparse vectors by using deep ReLU neural networks with varied width and depth, which may be of independent interest.