Multi-Agent Proximal Policy Optimization based efficient user association and resource allocation in UAV-assisted Heterogeneous Cellular Networks

The rapid growth in wireless communication demands has placed unprecedented pressure on modern networks, particularly concerning capacity enhancement and coverage expansion. Heterogeneous Cellular Networks (HCNs), with their flexible multi-layered architecture, present a promising solution to these challenges. However, these networks face considerable complexity and resource constraints. Consequently, developing efficient User Association and Resource Allocation (UARA) schemes is essential for establishing sustainable, high-performance wireless communication systems. In this work, we establish a paradigm for an Unmanned Aerial Vehicle (UAV)-assisted HCN with multiple User Equipment (UEs). An optimization scheme for UAV deployment and UARA policy is proposed to maximize the long-term system utility while ensuring quality of service for UEs. Specifically, we employ an efficient greedy deployment algorithm to dynamically update UAV locations and maximize the coverage utility. For UARA decision-making, we introduce a dynamic non-convex mixed-integer nonlinear programming problem and model it as a partially observable Markov decision process. Subsequently, an Independent Reward-based Hybrid action space Multi-Agent Proximal Policy Optimization algorithm (IRH-MAPPO) is proposed. Additionally, we utilize a centralized training and distributed execution framework and incorporate value normalization and action masking to improve its efficiency. Experimental results demonstrate that our algorithm significantly outperforms the baselines in terms of system utility, particularly in Spectral Efficiency (SE) and Energy Efficiency (EE), which confirms its effectiveness, superiority, and scalability.