AUV decision-making in bistatic sonar target tracking via deep reinforcement learning

Abstract

The maneuvering decisions of an Autonomous Underwater Vehicle (AUV) equipped with a sonar receiver are critical for acquiring informative measurements, thereby enhancing target tracking accuracy and maintaining persistent track continuity. This study investigates a bistatic sonar configuration comprising a stationary acoustic source and a mobile AUV carrying a linear sonar array, with a focus on real-time generation of optimized heading commands for the AUV. To address the practical limitations of conventional model-based strategies under stringent decision-time constraints, a deep reinforcement learning (DRL) framework is employed to derive an efficient decision policy. Implemented using the Proximal Policy Optimization algorithm, the DRL policy integrates with a target tracker by utilizing estimated target states and associated uncertainties as inputs. A reward function based on the Optimal Sub-Pattern Assignment metric is designed to guide policy learning. To train the policy, a comprehensive simulation environment is developed, incorporating target properties, acoustic propagation characteristics, and bistatic measurement models. Once trained, the DRL-based policy effectively guides the AUV in real-time to establish favorable bistatic configurations and enhance tracking performance. Comparative Monte Carlo simulations against a state-of-the-art model-based receding horizon control method demonstrate that the DRL-based approach yields a 14.5 % average improvement in tracking accuracy. Detailed analyses of AUV behaviors under the learned policy, along with robustness and generalization evaluations, underscore the practical potential of the proposed approach for real-world deployment in bistatic sonar target tracking scenarios.