Hypergraph-Based Model for Modeling Multi-Agent Q-Learning Dynamics in Public Goods Games

Modeling the learning dynamic of multi-agent systems has long been a crucial issue for understanding the emergence of collective behavior. In public goods games, agents interact in multiple larger groups. While previous studies have primarily focused on infinite populations that only allow pairwise interactions, we aim to investigate the learning dynamics of agents in a public goods game with higher-order interactions. With a novel use of hypergraphs for encoding higher-order interactions, we develop a formal model (a Fokker-Planck equation) to describe the temporal evolution of the distribution function of Q-values. Noting that early research focused on replicator models to predict system dynamics failed to accurately capture the impact of hyperdegree in hypergraphs, our model effectively maps its influence. Through experiments, we demonstrate that our theoretical findings are consistent with the agent-based simulation results. We demonstrated that as the number of groups an agent is involved in reaches a certain scale, the learning dynamics of the system evolve to resemble those of a well-mixed population. Furthermore, we demonstrate that our model offers insights into algorithmic parameters, such as the Boltzmann temperature, facilitating parameter tuning.