Inverse reinforcement learning for discrete-time linear systems based on inverse optimal control.

Abstract

This paper mainly deals with inverse reinforcement learning (IRL) for discrete-time linear time-invariant systems. Based on input and state measurement data from expert agent, several algorithms are proposed to reconstruct cost function in optimal control problem. The algorithms mainly consist of three steps, namely updating control gain via algebraic Riccati equation (ARE), gradient descent to correct cost matrix, and updating weight matrix based on inverse optimal control (IOC). First, by reformulating gain formula of optimal control in the learner system, we present a model-based IRL algorithm. When the system model is fully known, the cost function can be iteratively computed. Then, we develop a partially model-free IRL framework for reconstructing the cost function by introducing auxiliary control inputs and decomposing the algorithm into outer and inner loop. Therefore, in the case where the input matrix is unknown, weight matrix in the cost function is reconstructed. Moreover, the convergence of the algorithms and the stability of corresponding closed-loop system have been demonstrated. Finally, simulations verify the effectiveness of the proposed IRL algorithms.