Risk-Sensitive Control With Multiarmed Bandit Adaptive Sampling

Abstract

In this article, we propose a risk-sensitive (RS) control algorithm that optimizes the entropic risk measure (ERM). RS control minimizes the impact of high cost but low probability outcomes in dynamic decision making. ERM is a time-consistent convex measure of risk with tunable sensitivity. Our ERM-sensitive method combines online adaptive multistage sampling, RS upper confidence bound action selection, and sample-based estimation of ERM. Unlike many other RS control approaches, our method is applicable to problems with both discrete and continuous state spaces, and it does not require knowledge of the underlying transition probabilities. This makes our approach particularly useful in a wide variety of RS reinforcement learning domains. We show $O(\log M / M)$ error convergence under mild conditions and demonstrate the numerical performance on two RS benchmark problems with discrete and continuous state spaces, respectively.