Bandit problems with arbitrary side observations

Abstract

A bandit problem with side observations is an extension of the traditional two-armed bandit problem, in which the decision maker has access to side information before deciding which arm to pull. In this paper, the essential properties of the side observations that allow achievability results with respect to the minimal inferior sampling time are extracted and formulated. The sufficient conditions for good side information obtained here contain various kinds of random processes as special cases, including i.i.d sequences, Markov chains, periodic sequences, etc. A necessary condition is also provided, giving more insight into the nature of bandit problems with side observations. A game-theoretic approach simplifies the analysis and justifies the viewpoint that the side observation serves as an index of different sub-bandit machines.