Universal approximation property of ODENet and ResNet with a single activation function

Abstract

We study a universal approximation property of ODENet and ResNet. The ODENet is a map from an initial value to the final value of an ODE system in a finite interval. It is considered a mathematical model of a ResNet-type deep learning system. We consider dynamical systems with vector fields given by a single composition of the activation function and an affine mapping, which is the most common choice of the ODENet or ResNet vector field in actual machine learning systems. We demonstrate that both ODENets and ResNets with the restricted vector field of a single composition of the activation function and an affine mapping can uniformly approximate ODENets within the broader class that utilize a general vector field.