Neural adaptive delay differential equations

Abstract

Continuous-depth neural networks, such as neural ordinary differential equations (NODEs), have garnered significant interest in recent years owing to their ability to bridge deep neural networks with dynamical systems. This study introduced a new type of continuous-depth neural network called neural adaptive delay differential equations (NADDEs). Unlike recently proposed neural delay differential equations (NDDEs) that require a fixed delay, NADDEs utilize a learnable, adaptive delay. Specifically, NADDEs reformulate the learning process as a delay-free optimal control problem and leverage the calculus of variations to derive their learning algorithms. This approach enables the model to autonomously identify suitable delays for given tasks, thereby establishing more flexible temporal dependencies to optimize the utilization of historical representations. The proposed NADDEs can reconstruct dynamical systems with time-delay effects by learning true delays from data, a capability beyond both NODEs and NDDEs, and achieve superior performance on concentric and image-classification datasets, including MNIST, CIFAR-10, and SVHN.