A reinforcement learning method for two-layer shipboard real-time energy management considering battery state estimation

Abstract

Increasing global environmental concerns encourage a continuous reduction in carbon emissions from the shipping industry. It has become an irreversible trend to replace traditional fossil fuels with advanced energy storage technology. However, an improper energy management leads to not only energy waste but also undesired costs and emissions. Accordingly, the authors develop a two-layer shipboard energy management framework. In the initial stage, a shipboard navigation planning problem is formulated that considers battery state estimation and is subsequently solved using particle swarm optimisation to obtain an optimal speed trajectory. To track the scheduled speed, a reinforcement learning method based on a deep Q-Network is proposed in the second stage to realise real-time energy management of the diesel generator and energy storage system. This approach ensures that the state of charge remains within a safe range and that the performance is improved, avoiding excessive discharge from the energy storage systems and further enhancing the efficiency. The numerical results demonstrate the necessity and effectiveness of the proposed method.