A Reinforcement Learning Look at Risk-Sensitive Linear Quadratic Gaussian Control

Abstract

This paper proposes a novel robust reinforcement learning framework for discrete-time systems with model mismatch that may arise from the sim2real gap. A key strategy is to invoke advanced techniques from control theory. Using the formulation of the classical risk-sensitive linear quadratic Gaussian control, a dual-loop policy iteration algorithm is proposed to generate a robust optimal controller. The dual-loop policy iteration algorithm is shown to be globally exponentially and uniformly convergent, and robust against disturbance during the learning process. This robustness property is called small-disturbance input-to-state stability and guarantees that the proposed policy iteration algorithm converges to a small neighborhood of the optimal controller as long as the disturbance at each learning step is small. In addition, when the system dynamics is unknown, a novel model-free off-policy policy iteration algorithm is proposed for the same class of dynamical system with additive Gaussian noise. Finally, numerical examples are provided for the demonstration of the proposed algorithm.