Computing monotone policies for Markov decision processes by exploiting sparsity

Abstract

This paper considers Markov decision processes whose optimal policy is a randomized mixture of monotone increasing policies. Such monotone policies have an inherent sparsity structure. We present a two-stage convex optimization algorithm for computing the optimal policy that exploits the sparsity. It combines an alternating direction method of multipliers (ADMM) to solve a linear programming problem with respect to the joint action state probabilities, together with a subgradient step that promotes the monotone sparsity pattern in the conditional probabilities of the action given the state. In the second step, sum-of-norms regularization is used to stress the monotone structure of the optimal policy.