A multi-step minimax Q-learning algorithm for two-player zero-sum Markov games

An interesting iterative procedure is proposed to solve two-player zero-sum Markov games. Under suitable assumptions, the boundedness of the proposed iterates is obtained theoretically. Using results from stochastic approximation, the almost sure convergence of the proposed multi-step minimax Q-learning is obtained theoretically. More specifically, the proposed algorithm converges to the game theoretic optimal value with probability one, when the model information is not known. Numerical simulations authenticate that the proposed algorithm is effective and easy to implement.