Model-Free Linear Noncausal Optimal Control of Wave Energy Converters via Reinforcement Learning

Abstract

This article introduces a novel reinforcement learning (RL) method for wave energy converters (WECs), which directly generates linear noncausal optimal control (LNOC) policies on continuous action space. Unlike other existing WEC RL algorithms looking at the problem mainly from a learning perspective, the proposed RL approach adopts a control-theoretic approach by delving into the underlying WEC energy maximization (EM) optimal control problem (OCP). This leads to control-informed decisions on choosing the RL state, as well as developing the RL structure. The proposed model-free LNOC (MF-LNOC) offers substantial advantages, including significantly improved performance due to the use of noncausal information, a simplified RL with linear actor and quadratic critic structures, and remarkable fast convergence speeds, achieved using less than 150 s of data points, for a benchmarked point absorber, which can be further shortened using the replay technique. This reduction in training time allows for controller reconfiguration in pace with sea changes. Demonstrative numerical simulations are presented to verify the efficacy of the proposed methods. The proposed MF-LNOC also shows robustness against wave prediction inaccuracies and changing sea conditions. The MF-LNOC methodology can be highly attractive for WEC developers who want to design an efficient and reliable controller for WECs but also hope to avoid the challenge of establishing a control-oriented model that can preserve high fidelity over a wide range of sea conditions.