Convergence Rates of Online Critic Value Function Approximation in Native Spaces

Abstract

This letter derives rates of convergence of online critic methods for the estimation of the value function for a class of nonlinear optimal control problems. Assuming that the underlying value function lies in reproducing kernel Hilbert space (RKHS), we derive explicit bounds on the performance of the critic in terms of the kernel functions, the number of basis functions, and the scattered location of centers used to define the RKHS. The performance of the critic is precisely measured in terms of the power function of the scattered bases, and it can be used either in an a priori evaluation of potential bases or in an a posteriori assessments of the value function error for basis enrichment or pruning. The most concise bounds in this letter describe explicitly how the critic performance depends on the placement of centers, as measured by their fill distance in a subset that contains the trajectory of the critic. To the authors’ knowledge, precise error bounds of this form are the first of their kind for online critic formulations used in optimal control problems. In addition to their general and immediate applicability to a wide range of applications, they have the potential to constitute the groundwork for more advanced “basis-adaptive” methods for nonlinear optimal control strategies, ones that address limitations due to the dimensionality of approximations.