Generative AI-driven reinforcement learning for beamforming and scheduling in multi-cell MIMO-NOMA systems

Abstract

This article introduces a novel generative artificial intelligence-enhanced primal–dual proximal policy optimization (GAI-PDPPO) framework for joint user scheduling and beamforming in downlink multi-cell multiple-input and multiple-output non-orthogonal multiple access (MC-MIMO-NOMA) networks. Designed to address the challenges of interference-laden environments typical in beyond the fifth generation (B5G)/sixth generation (6G) systems, the proposed method formulates a complex mixed-integer nonlinear programming problem to minimize transmit power under stringent Quality-of-Service (QoS) constraints. Unlike conventional approaches, GAI-PDPPO incorporates an invertible transformer-based actor-critic architecture capable of modeling high-dimensional channel state information and unknown-source interference. Through the integration of generative pretraining and prioritized experience replay, the framework accelerates convergence and enhances policy generalization. Extensive simulations demonstrate that GAI-PDPPO consistently outperforms standard primal–dual PPO and benchmark solutions, achieving lower power consumption and higher spectral efficiency under varying signal-to-interference-plus-noise ratio (SINR) thresholds and interference conditions.