Distributionally Robust Chance-Constrained Markov Decision Processes with Random Payoff

A Markov Decision Process (MDP) is a framework used for decision-making. In an MDP problem, the decision maker’s goal is to maximize the expected discounted value of future rewards while navigating through different states controlled by a Markov chain. In this paper, we focus on the case where the transition probabilities vector is deterministic, while the reward vector is uncertain and follow a partially known distribution. We employ a distributionally robust chance constraints approach to model the MDP. This approach entails the construction of potential distributions of reward vector, characterized by moments or statistical metrics. We explore two situations for these ambiguity sets: one where the reward vector has a real support and another where it is constrained to be nonnegative. In the case of a real support, we demonstrate that solving the distributionally robust chance-constrained Markov decision process is mathematically equivalent to a second-order cone programming problem for moments and \(\phi \)-divergence ambiguity sets. For Wasserstein distance ambiguity sets, it becomes a mixed-integer second-order cone programming problem. In contrast, when dealing with nonnegative reward vector, the equivalent optimization problems are different. Moments-based ambiguity sets lead to a copositive optimization problem, while Wasserstein distance-based ambiguity sets result in a biconvex optimization problem. To illustrate the practical application of these methods, we examine a machine replacement problem and present results conducted on randomly generated instances to showcase the effectiveness of our proposed methods.