Exploring the performance impact of soft constraint integration on reinforcement learning-based autonomous vessel navigation: Experimental insights

Abstract

Reinforcement learning has shown promise in enabling autonomous ship navigation, allowing vessels to adapt and make informed decisions in complex marine environments. However, the integration of soft constraints and their impact on performance in RL-based autonomous vessel navigation research remain understudied. This research addresses this gap by investigating the implications of soft constraints in the context of the risk-averse ship navigation problem. Four distinct soft constraint functions are proposed, which are integrated with two widely used RL algorithms, resulting in the creation of eight risk-averse autonomous vessel navigation models. To ensure a comprehensive evaluation of their performance, comparative analyses are conducted across seven virtual digital channel environments. Additionally, a novel metric, known as Large Helm Momentum (LHM), is introduced to quantify the smoothness of autonomous vessel navigation. Through thorough experimentation, key considerations for the design of soft constraint functions in the domain of autonomous ship navigation are identified. A comprehensive understanding of how different soft constraint functions influence autonomous driving behavior has been achieved. Key considerations for designing soft constraint functions in the domain of autonomous ship navigation have also been identified. Five principles, namely the constraint association principle, dominance of hard constraints, reward-balance principle, mapping requirement principle, and iterative improvement principle, are proposed to optimize the design of soft constraint functions for autonomous ship navigation, providing valuable guidance and insights.