Reinforcement learning-based trajectory tracking optimal control for underactuated unmanned surface vehicles under asymmetric input saturation

Abstract

For underactuated unmanned surface vehicles (USVs) under asymmetric input saturation caused by thrust-limit characteristics, as well as unknown dynamics and ocean environmental disturbances, a trajectory tracking optimal control (TTOC) scheme is proposed using the reinforcement learning (RL) method. Through coordinate transformations and mathematical derivation, an underactuated USV motion model is transformed into the standard affine nonlinear form. To address the asymmetric input saturation of underactuated USVs, a new inverse hyperbolic tangent-type penalty function is designed for control inputs, relaxing the assumption of input saturation limits being symmetric. Based on RL methods and adaptive neural networks (NNs), an actor-critic NN framework is developed, with weight update laws designed for NNs. This framework learns the TTOC law for underactuated USVs through the online interaction of actor and critic NNs while adapting to unknown dynamics and disturbances. In particular, a robustifying term is designed and added to the output of an actor NN to compensate for the adverse effects of a lumped residual term, which enhances the robustness of the TTOC law and thereby achieves asymptotic regulation of trajectory tracking errors. Theoretical analyses and simulation results indicate that the proposed TTOC scheme enables underactuated USVs to asymptotically track the desired trajectory.