Optimal Control Using IsoCost-Based Dynamic Programming

Abstract

In this paper, a novel data-driven optimal control method based on reinforcement learning concepts is introduced. The proposed algorithm performs as a workaround to solving the Hamilton–Jacobi–Bellman equation. The main concept behind the proposed algorithm is the so-called IsoCost hypersurface (ICHS), which is a hypersurface in the state space of the system formed by points from which a specific amount of cost is spent by the control strategy in order to asymptotically stabilize the system. The fact that the control strategy requires to spend equal costs in order to stabilize all points on an ICHS is the reason for the naming of the IsoCost concept. Additional assumptions and definitions are mentioned before providing the theory of ICHS optimality. This theory proves, by contradiction, that the ICHS corresponding to the optimal control policy surrounds the ICHSs corresponding to other non-optimal control solutions. This paves the path to finding the optimal control solution using dynamic programming. The proposed method is implemented on the linear, fixed-base inverted pendulum, cart-pole and torsional pendulum bar system models and the results are compared with that of literature. The performance of this method in terms of cost, settling time and computation time is shown using numeric and illustrative comparisons.