Reinforcement Learning with an Extended Classifier System in Zero-sum Markov Games

A reinforcement learning (RL) agent can learn how to win against an opponent agent in zero-sum Markov Games after episodes of training. However, it is still challenging for the RL agent to acquire the optimal policy if the opponent agent is also able to learn concurrently. In this paper, we propose a new RL algorithm based on the eXtended Classifier System (XCS) that maintains a population of competing rules for action selection and uses the genetic algorithm (GA) to evolve the rules for searching the optimal policy. The RL agent can learn from scratch by observing the behaviors of the opponent agent without making any assumptions about the policy of the RL agent or the opponent agent. In addition, we use eligibility trace to further speed up the learning process. We demonstrate the performance of the proposed algorithm by comparing it with several benchmark algorithms in an adversarial soccer game against the same deterministic policy learner.