Data-Driven Optimization-Based Cost and Optimal Control Inference

Abstract

This letter develops a novel optimization-based inverse reinforcement learning (RL) control algorithm that infers the minimal cost from observed demonstrations via optimization-based policy evaluation and update. The core idea is the simultaneous evaluation of the value function matrix and cost weight during policy evaluation under a given control policy, which simplifies the algorithmic structure and reduces the iterations required for convergence. Based on this idea, we first develop a model-based algorithm with detailed implementation steps, and analyze the monotonicity and convergence properties of the cost weight. Then, based on Willems’ lemma, we develop a data-driven algorithm to learn an equivalent weight matrix from persistently excited (PE) data. We also prove the convergence of the data-driven algorithm and show that the converged results learned from PE data are unbiased. Finally, simulations on a power system are carried out to demonstrate the effectiveness of the proposed inverse RL algorithm.