Coordinated Q-learning based Multi-hop Routing for UAV-assisted communication

Unmanned Aerial Vehicle (UAV) assisted communication is gaining prominence as a vital solution for establishing effective emergency communication during disaster management operations. UAVs are essential for enhancing and expanding communication systems, acting as relays to boost data transmission to ground stations, extend network coverage, and provide connectivity. However, the dynamic and resource-limited nature of aerial networks necessitates robust routing mechanisms to facilitate seamless data dissemination. While existing Q-learning-based routing protocols are adaptive to changing network conditions and resilient to failures, they often lead to suboptimal network-wide decisions due to UAVs operating independently, each maximizing its gains. This paper proposes a novel Coordinated Q-learning-based Multi-hop Routing (CQMR) algorithm for multi-UAV networks. To the best of our knowledge, this is the first time a routing algorithm introduces UAV coordination for data routing through utility function approximation with a message-passing scheme, enabling the selection of globally optimal joint actions. This novel approach meticulously considers a comprehensive set of parameters for data routing, including minimizing the expected number of hops to the destination, monitoring energy usage, maintaining network connectivity, preventing UAV collisions, and supporting adaptive network reorganization. This integrated consideration of multiple factors positions the proposed solution as superior to existing work, offering a uniquely robust and highly effective strategy for UAV-assisted communication in dynamic, resource-constrained environments, such as emergency scenarios. CQMR builds upon and extends the Improved Q-learning-based Multi-hop Routing (IQMR) algorithm, demonstrating a 12.47% increase in energy efficiency and a 13.34% higher success rate in data transmission compared to IQMR while requiring 40% fewer hops to reach the destination.