A distributed multi-agent deep reinforcement learning approach for dynamic beam hopping optimization in LEO mega-constellations

With the rapid advancement of low Earth orbit (LEO) satellite networks, conventional static beam allocation methods have become insufficient in addressing the challenges posed by dynamic traffic demands and uneven user distribution. To overcome these limitations, we propose a novel beam hopping scheduling approach specifically designed for dealing with uncertain channel conditions and time-varying traffic requirements in LEO satellite systems. We first develop a multi-objective optimization model that effectively balances the critical performance metrics of throughput and delay in LEO satellite networks. Building upon this foundation, we formulate a locally interacting Markov game model and rigorously prove the existence of at least one Nash equilibrium, thereby establishing a theoretical basis for our approach. To implement this model effectively, we introduce the Multi-Agent Deep Q-Network with Local Cooperative Rewards (MDQN-LCR) algorithm, which enables satellites to make intelligent decisions through a distributed Q-learning framework enhanced by a cooperative reward mechanism. Through extensive simulation experiments in diverse scenarios, results demonstrate that MDQN-LCR outperforms existing centralized methods by achieving 2.6% higher throughput in large-scale deployments and 13.5% lower transmission delay in short time slots. Our approach demonstrates superior stability with a confidence interval 42% smaller than that of centralized QMIX, while significantly reducing communication overhead through its distributed architecture. This makes our solution particularly suitable for large constellation scenarios , thus offering a practical and scalable alternative for next-generation satellite communication systems.