Model-free optimal control for discrete-time Markovian jump linear systems: A Q-learning approach

This paper presents a model-free optimal control strategy for discrete-time Markovian Jump Linear Systems (MJLS) using a Q-learning-based reinforcement learning (RL) framework. Conventional model-based control techniques for MJLS rely on full knowledge of system dynamics and the solution of coupled algebraic Riccati equations (CARE), which may not be feasible in many practical scenarios. To overcome this limitation, we propose a novel Q-function formulation that explicitly incorporates the Markovian switching behavior of the system. An off-policy Q-learning algorithm is developed to estimate the kernel matrix of the Q-function directly from raw input-state data, enabling the computation of optimal controller gains without requiring system models. We rigorously prove that the learned controller gains converge to those of the model-based optimal controller, thereby ensuring mean-square stability. Simulation results on a networked control system with Markovian packet losses demonstrate the convergence, stability, and practical effectiveness of the proposed model-free controller.