A primal–dual policy iteration algorithm for constrained Markov decision processes

Abstract

The solution algorithms of Constrained Markov Decision Process (CMDP), a widely adopted model for sequential decision-making, have been intensively studied in the literature. Despite increasing effort, the Linear Programming (LP) formulation of CMDP remains the dominant exact method that leads to the optimal solution without constraint violations. However, the LP formulation is computationally inefficient due to the curse of dimensionality in CMDP state and action spaces. In this study, we introduce a novel policy iteration method for CMDP, based on decomposition and row-generation techniques. We design a Primal–Dual Policy Iteration (PDPI) algorithm that utilizes state values and Lagrangian multipliers to improve randomized stationary policies in an iterative fashion. We analytically show that upon convergence, PDPI produces the optimal solution for CMDP. An upper bound of the convergence iterations is also given. To validate the algorithm performance, we conduct comprehensive computational experiments on six benchmarking problems curated from the literature. Results show that PDPI outperforms conventional methods considerably, improving the total algorithm runtime by up to 89.19%. The improvement becomes more significant as the problem size grows larger. We further provide insights and discuss the impact of the developed method.