A tutorial review of policy iteration methods in reinforcement learning for nonlinear optimal control

Abstract

Reinforcement learning (RL) has been a powerful framework for designing optimal controllers for nonlinear systems. This tutorial review provides a comprehensive exploration of RL techniques, with a particular focus on policy iteration methods for the development of optimal controllers. We discuss key theoretical aspects, including closed-loop stability and convergence analysis of learning algorithms. Additionally, the review addresses practical challenges encountered in real-world applications, such as the development of accurate process models, incorporating safety guarantees during learning, leveraging physics-informed machine learning and transfer learning techniques to overcome learning difficulties, managing model uncertainties, and enabling scalability through distributed RL. To demonstrate the effectiveness of these approaches, a simulation example of a chemical reactor is presented, with open-source code made available on GitHub. The review concludes with a discussion of open research questions and future directions in RL-based control of nonlinear systems.