A pipelining task offloading strategy via delay-aware multi-agent reinforcement learning in Cybertwin-enabled 6G network

Cybertwin-enabled 6th Generation (6G) network is envisioned to support artificial intelligence-native management to meet changing demands of 6G applications. Multi-Agent Deep Reinforcement Learning (MADRL) technologies driven by Cybertwins have been proposed for adaptive task offloading strategies. However, the existence of random transmission delay between Cybertwin-driven agents and underlying networks is not considered in related works, which destroys the standard Markov property and increases the decision reaction time to reduce the task offloading strategy performance. In order to address this problem, we propose a pipelining task offloading method to lower the decision reaction time and model it as a delay-aware Markov Decision Process (MDP). Then, we design a delay-aware MADRL algorithm to minimize the weighted sum of task execution latency and energy consumption. Firstly, the state space is augmented using the lastly-received state and historical actions to rebuild the Markov property. Secondly, Gate Transformer-XL is introduced to capture historical actions' importance and maintain the consistent input dimension dynamically changed due to random transmission delays. Thirdly, a sampling method and a new loss function with the difference between the current and target state value and the difference between real state-action value and augmented state-action value are designed to obtain state transition trajectories close to the real ones. Numerical results demonstrate that the proposed methods are effective in reducing reaction time and improving the task offloading performance in the random-delay Cybertwin-enabled 6G networks.