Balance of exploration and exploitation: Non-cooperative game-driven evolutionary reinforcement learning

Abstract

In a complex and dynamic environment, it becomes difficult to solve problems with a single policy updating mode. Although evolutionary reinforcement learning partially addresses this issue, it fails to consider the real-time status of the agent, resulting in inflexible policy updating modes that can negatively affect algorithm performance. To address this issue, we propose an evolutionary reinforcement learning based on non-cooperative games that leverages the benefits of various algorithms in cooperative and non-cooperative settings. Firstly, a competition framework is established between the evolutionary algorithm and reinforcement learning using a non-cooperative game. The policy updating mode is dynamically selected based on the game’s outcome, ensuring diverse algorithms through differentiated population evolution guided by Nash equilibrium. Secondly, the evolutionary algorithm collaborates with the non-cooperative game to establish a cooperative framework that balances exploration and convergence. Since the exploration of the evolutionary algorithm does not rely on the environment, it is used to guide the non-cooperative game, which helps the algorithm successfully overcome the local optimum. This synergy significantly enhances algorithm performance. Experimental results demonstrate that the proposed algorithm outperforms individual algorithms.