Inverse Q-Learning Optimal Control for Takagi–Sugeno Fuzzy Systems

Abstract

Inverse reinforcement learning optimal control is under the framework of learner–expert, the learner system can learn expert system's trajectory and optimal control policy via a reinforcement learning algorithm and does not need the predefined cost function, so it can solve optimal control problem effectively. This article develops a fuzzy inverse reinforcement learning optimal control scheme with inverse reinforcement learning algorithm for Takagi–Sugeno (T–S) fuzzy systems with disturbances. Since the controlled fuzzy systems (learner systems) desire to learn or imitate expert system's behavior trajectories, a learner–expert structure is established, where the learner only know the expert system's optimal control policy. To reconstruct expert system's cost function, we develop a model-free inverse Q-learning algorithm that consists of two learning stages: an inner Q-learning iteration loop and an outer inverse optimal iteration loop. The inner loop aims to find fuzzy optimal control policy and the worst-case disturbance input via learner system's cost function by employing zero-sum differential game theory. The outer one is to update learner system's state-penalty weight via only observing expert systems' optimal control policy. The model-free algorithm does not require that the controlled system dynamics are known. It is proved that the designed algorithm is convergent and also the developed inverse reinforcement learning optimal control policy can ensure T–S fuzzy learner system to obtain Nash equilibrium solution. Finally, we apply the presented fuzzy inverse Q-learning optimal control method to nonlinear unmanned surface vehicle system and the computer simulation results verified the effectiveness of the developed scheme.