Online adaptive critic designs with tensor product B-splines and incremental model techniques

Abstract

As an effective optimal control scheme in the field of reinforcement learning, adaptive dynamic programming (ADP) has attracted extensive attention in recent decades. Neural networks (NNs) are commonly employed in ADP algorithms to realize nonlinear function approximation. However, the learning process with NN approximation typically requires a substantial amount of training data and places a heavy computational burden. To improve learning efficiency and alleviate computational load, this paper develops a novel model-free ADP approach based on multivariate splines and incremental control techniques, aimed at achieving online learning of nonlinear control. By utilizing input and output data, an incremental linear approximation model is identified online without any prior knowledge of system dynamics. To improve nonlinear approximation capabilities and lessen computational demands, tensor product B-splines, instead of NNs, are integrated into the critic module to approximate the value function, and the recursive least squares temporal difference (RLS-TD) algorithm is employed to update the weights of spline bases. Convergence analysis of the proposed control scheme is conducted based on the dual-timescale stochastic approximation theory. To illustrate the effectiveness and feasibility of the proposed control scheme, numerical simulations are performed in solving the online nonlinear control problem within the context of the inverted pendulum system.