Q-learning facilitates norm emergence in metanorm game model with topological structures

Abstract

Axelrod’s model and its subsequent studies have become a valuable framework for fostering cooperation norms among self-interested agents. Within this framework, the concepts of “boldness” and “vengefulness” are specifically employed to characterize agents’ behaviors in terms of cooperation and punishment (including metapunishment). Describing behavior solely through the parameters B and V may be overly simplistic and lacks generalizability, making it difficult to apply to other scenarios. Moreover, privacy concerns and the difficulty of evaluating complex states in real-world scenarios limit agents’ access to detailed payoff information from their neighbors. To address these questions, our paper employs self-regarding Q-learning, a well-established method for examining the dynamics of strategy updates and agents’ learning processes, to investigate whether metanorms can naturally emerge through players’ strategy selection. Through extensive experiments, we observe cooperative norms’ successful emergence driven by agents’ strategy selection variations. Over 90% of agents choose to cooperate on average. In subsequent analyses, we explore the underlying reasons for the emergence of cooperative norms from perspectives of changes in Q-values, punishment and metapunishment frequencies. Additionally, we examine the impact of topological structures on players’ strategy selection and assess the emergence of norms across different temptation levels, population sizes, and regulatory intensity levels to validate the model’s sensitivity.