Limiting dynamics for Q-learning with memory one in two-player, two-action games

We develop a computational method to identify all pure strategy equilibrium points in the strategy space of the two-player, two-action repeated games played by Q-learners with one period memory. In order to approximate the dynamics of these Q-learners, we construct a graph of pure strategy mutual best-responses. We apply this method to the iterated prisoner’s dilemma and find that there are exactly three absorbing states. By analyzing the graph for various values of the discount factor, we find that, in addition to the absorbing states, limit cycles become possible. We confirm our results using numerical simulations.