Weighted mesh algorithms for general Markov decision processes: Convergence and tractability

Abstract

We introduce a mesh-type approach for tackling discrete-time, finite-horizon Markov Decision Processes (MDPs) characterized by state and action spaces that are general, encompassing both finite and infinite (yet suitably regular) subsets of Euclidean space. In particular, for bounded state and action spaces, our algorithm achieves a computational complexity that is tractable in the sense of Novak & Woźniakowski [12], and is polynomial in the time horizon. For an unbounded state space the algorithm is “semi-tractable” in the sense that the complexity is proportional to ${\epsilon }^{-c}$ with some dimension independent $c\ge 2$, to achieve precision ε, and polynomial in the time horizon with linear degree in the underlying dimension. As such, the proposed approach has some flavor of the randomization method by Rust [14] which uses uniform sampling in compact state space. However, the present approach is essentially different due to the inhomogeneous finite horizon setting, which involves general transition distributions over a possibly non-compact state space. To demonstrate the effectiveness of our algorithm, we provide illustrations based on Linear-Quadratic Gaussian (LQG) control problems.