Robustness to Modeling Errors in Risk-Sensitive Markov Decision Problems With Markov Risk Measures

Abstract

We consider risk-sensitive Markov decision processes (MDPs), where the MDP model is influenced by a parameter which takes values in a compact metric space. These situations arise when the underlying dynamics of the system depend on parameters that drifts over time. For example, mass of a vehicle depends on the number of passengers in the vehicle, which may change from one trip to another. Similarly, the energy demand of a building depends on the local weather, which changes every hour of the day. We identify sufficient conditions under which small perturbations in the model parameters lead to small changes in the optimal value function and optimal policy. This is achieved by establishing the continuity of the value function with respect to the parameters. A direct consequence of this result is that an optimal policy under a specific parameter remains near-optimal if the parameter is perturbed slightly. Implications of the results for data-driven decision-making, decision-making with preference uncertainty, and systems with changing noise distributions are discussed.