A Distributed Primal–Dual Method for Constrained Multiagent Reinforcement Learning With General Parameterization

Abstract

This article proposes a novel distributed approach for solving a cooperative constrained multiagent reinforcement learning problem, where agents seek to minimize a global objective function subject to shared constraints. Unlike existing methods that rely on centralized training or coordination, our approach enables fully decentralized online learning, with each agent maintaining local estimates of both primal and dual variables. Specifically, we develop a distributed primal–dual algorithm based on actor–critic methods, leveraging local information to estimate Lagrangian multipliers. We establish consensus among the Lagrangian multipliers across agents and prove the convergence of our algorithm to an equilibrium point, analyzing the suboptimality of this equilibrium compared to the exact solution of the unparameterized problem. Furthermore, we introduce a constrained cooperative Cournot game with stochastic dynamics as a test environment to evaluate the algorithm’s performance in complex, real-world scenarios.