Multi-Agent Reinforcement Learning With Deep Networks for Diverse Q $Q$ -Vectors

Abstract

In multi-agent reinforcement learning (MARL) tasks, the state-action value, commonly referred to as the $Q$-value, can vary among agents because of their individual rewards, resulting in a $Q$-vector. Determining an optimal policy is challenging, as it involves more than just maximizing a single $Q$-value. Various optimal policies, such as a Nash equilibrium, have been studied in this context. Algorithms like Nash Q-learning and Nash Actor-Critic have shown effectiveness in these scenarios. This paper extends this research by proposing a deep Q-networks algorithm capable of learning various $Q$-vectors using Max, Nash, and Maximin strategies. We validate the effectiveness of our approach in a dual-arm robotic environment, a representative human cyber-physical systems (HCPS) scenario, where two robotic arms collaborate to lift a pot or hand over a hammer to each other. This setting highlights how incorporating MARL into HCPS can address real-world complexities such as physical constraints, communication overhead, and dynamic interactions among multiple agents.