Reinforcement learning-assisted particle swarm algorithm for effluent scheduling problem with an influent estimation of WWTP

Abstract

Effluent scheduling of wastewater treatment process (WWTP) is essential to ensure compliance with regulatory standards regarding effluent quality. Through the integration of pipe and plant systems, the influent can be estimated prior to entering the treatment process, providing additional information for scheduling. However, the traditional evolutionary computation methods face challenges in utilizing information from inflow estimation, resulting in decisions that do not account for long-term returns. For solving effluent scheduling problems with influent estimation, reinforcement learning can facilitate decision-making based on long-term environmental factors to improve the optimization ability of evolutionary computations. Thus, a framework of reinforcement learning-assisted particle swarm optimization algorithm (RLA-PSO) is proposed, using reinforcement learning part to generate solutions and guide optimization by learning from the influent estimation on a long-time scale. Meanwhile, it employs the optimization part to find the optimal solutions to intensify the learning effect of the reinforcement learning part. For the reinforcement learning part, a deep Q-network method with appropriate states and rewards is designed to efficiently learn the relationship between state, action and reward for the coming period. For the optimization part, a set-based particle optimization algorithm is employed to search for the optimized solution in a future period. The benchmark simulation model No.1(BSM1) is used to evaluate the performance of the proposed RLA-PSO algorithm for the effluent scheduling problem of WWTP. The computational experiments to the state-of-the-art methods show the proposed algorithm can achieve superior performance in effluent quality and process efficiency.