Nonlinear Disturbance Compensation and Reference Tracking via Reinforcement Learning with Fuzzy Approximators

Abstract Reinforcement Learning (RL) algorithms can learn optimal control laws for nonlinear dynamic systems without relying on a mathematical model of the system to be controlled. While RL can in principle discover control laws from scratch, by solely interacting with the process, in practice this does not yield any significant advantages. Learning control laws from scratch is lengthy and may lead to system damage due to the trial and error nature of the learning process. In this paper, we adopt a different and largely unexplored approach: a nominal control law is used to achieve reasonable, yet suboptimal, performance and a RL agent is trained to act as a nonlinear compensator whose task is to improve upon the performance of the nominal controller. The RL agent learns by means of an actor-critic algorithm using a plant model acquired on-line, alongside the critic and actor. Fuzzy approximators are employed to represent all the adjustable components of the learning scheme. One advantage of fuzzy approximators is the straightforward way in which they allow for the inclusion of prior knowledge. The proposed control scheme is applied to a reference tracking problem of 1-DOF robot arm influenced by an unknown payload disturbance due to gravity. The nominal controller is a PD controller, which is unable to properly compensate the effect of the disturbance considered. Simulation results indicate that the novel method is able to learn to compensate the disturbance for any reference angle varying throughout the experiment.