Offline reinforcement learning strategies guided meta-heuristics for scheduling bi-objective unmanned surface vessel problems with multiple constraints

Abstract

This study proposes a reinforcement learning-guided meta-heuristics framework for bi-objective unmanned surface vessel (USV) scheduling problems under complex marine constraints, aiming to minimize the maximum completion time and total collision risk index, simultaneously. First, to specify the problems, a bi-objective mathematical model is developed considering three constraints, battery capacity, marine obstacles, and uncertain task executing time. Second, four meta-heuristics are used and improved to solve the focused problems. Based on the problem features, seven local search operators are designed to enhance the algorithms’ performances. Third, two state-reward strategies are designed and integrated into Q-learning and SARSA, respectively, to form four reinforcement learning (RL) algorithms. The four RL algorithms are off-line trained and employed to select the optimal local search operator during the iteration of meta-heuristics for improving the search efficiency. Finally, the study evaluates the performances of the proposed algorithms on 10 cases with different scales. The experimental results and statistical tests verify the efficiency of the local search operators. It is demonstrated that the four proposed RL algorithms can further improve algorithms’ performances. The particle swarm optimization (PSO) integrating Q-learning with the second state-reward strategy (PSO_QL2) exhibits the best competitiveness among all compared algorithms.