Swim4Real: Deep Reinforcement Learning-Based Energy-Efficient and Agile 6-DOF Control for Underwater Vehicles

Abstract

Uncrewed underwater vehicles (UUVs) require precise and energy-efficient six-degrees-of-freedom (6-DOF) control to operate in complex underwater environments for long periods of time. Traditional controllers, like Proportional-Integral-Derivative (PID), struggle with nonlinear dynamics, while Model Predictive Control depends on accurate models, which are often complex or unavailable. Deep Reinforcement Learning (DRL), on the other hand, enables controllers to learn control strategies through environmental interactions, using neural networks capable of capturing nonlinear relationships. In this work, we introduce an end-to-end DRL-based controller, the Robust and Energy Efficient Framework (REEF) DRL, designed for precise 6-DOF control of UUVs while minimizing energy consumption. Furthermore, to improve robustness and adaptability, we propose REEF-DR DRL, which incorporates domain randomization. Through a comprehensive simulation-based evaluation, we demonstrate that our approach outperforms state-of-the-art DRL-based 6-DOF controllers for UUVs in terms of accuracy and energy efficiency. Furthermore, REEF DRL and REEF-DR DRL achieve position and orientation accuracy comparable to a well-tuned PID controller while reducing energy consumption by at least 30%. In-water experiments show that our controllers maintain high performance comparable to the well-tuned PID but reduce energy consumption by at least 39%. This work represents a significant advancement in applying DRL to underwater robotics, offering a promising solution to extend UUV operational autonomy.