A deep reinforcement learning-based controller design framework for Lipschitz continuous nonlinear systems

Abstract

Due to the complex dynamics and uncertainty of the nonlinear systems, designing controllers for such systems poses significant challenges. To address this dilemma, deep reinforcement learning (DRL) indicates a promising method. However, most designs of reward/value functions in DRL rely on experience, which takes much trial and error. In order to decrease the trial cost, this paper proposes a novel DRL method founded on actor-critic (AC) architecture for nonlinear system controller design, which is called actor-Lyapunov (AL). Diverging from conventional AC architecture, AL eliminates the necessity of the critic network. The actor network can be trained by utilizing a kind of Lyapunov function as the value function. Firstly, we provide a perspective of normed linear space to clarify the controller design. The controller generated by the actor network is regarded as a proper mapping within the state space. Based on this concept, the convergence of this approach under gradient descent is briefly analyzed. Next, a refined value function related to the exponent is introduced to promote the training effect of the actor network. Finally, simulations are conducted to validate the efficacy of our approach and illustrate the advantages of the refined value function in improving system performance.