Optimal bipartite consensus for multi-agent systems using twin Q-learning deterministic policy gradient algorithm with adaptive learning rate

Abstract

We investigate the optimal bipartite consensus control (OBCC) problem for multi-agent systems (MASs) over a signed network. Due to the improper cooperation-competition strength (CCS) among agents, the system may be unstable or even non-convergent. Recognizing the close relationship between CCS and the training of the critic network, we propose a twin Q-learning deterministic policy gradient algorithm with an adaptive learning rate (ALR-TQDPG). First, an adaptive learning rate formula is established based on the CCS and historical temporal difference (TD) error variations. The weights of two factors are dynamically adjusted using the weight equation as training progresses, then dynamically adjusting the update magnitude (i.e., learning rate) of critic network weights. Second, to solve the underestimation problem of Q-value, a twin Q-learning algorithm is adopted to improve system performance. The addition of experience replay and target network methods enhances algorithm stability. Lyapunov stability theory and functional analysis are utilized to ensure the ALR-TQDPG algorithm’s convergence. Finally, numerical simulations confirm that the suggested approach is effective.