PIDNODEs: Neural ordinary differential equations inspired by a proportional–integral–derivative controller

Abstract

Neural Ordinary Differential Equations (NODEs) are a novel family of infinite-depth neural-net models through solving ODEs and their adjoint equations. In this paper, we present a strategy to enhance the training and inference of NODEs by integrating a Proportional–Integral–Derivative (PID) controller into the framework of Heavy Ball NODE, resulting in the proposed PIDNODEs and its generalized version, GPIDNODEs. By leveraging the advantages of control, PIDNODEs and GPIDNODEs can address the stiff ODE challenges by adjusting the parameters (i.e., $K_p$, $K_i$ and $K_d$) in the PID module. The experiments confirm the superiority of PIDNODEs/GPIDNODEs over other NODE baselines on different computer vision and pattern recognition tasks, including image classification, point cloud separation and learning long-term dependencies from irregular time-series data for a physical dynamic system. These experiments demonstrate that the proposed models have higher accuracy and fewer function evaluations while alleviating the dilemma of exploding and vanishing gradients, particularly when learning long-term dependencies from a large amount of data.