On Restless Linear Bandits

Abstract

A more general formulation of the linear bandit problem is considered to allow for dependencies over time. Specifically, it is assumed that there exists an unknown $\mathbb {R}^{d}$ -valued stationary $\varphi $ -mixing sequence of parameters $(\theta _{t}, \; t \in \mathbb {N})$ which gives rise to payoffs. This instance of the problem can be viewed as a generalization of both the classical linear bandits with iid noise, and the finite-armed restless bandits. In light of the well-known computational hardness of optimal policies for restless bandits, an approximation is proposed whose error is shown to be controlled by the $\varphi $ -dependence between consecutive $\theta _{t}$ . An optimistic algorithm, called LinMix-UCB, is proposed for the case where $\theta _{t}$ has an exponential mixing rate. The proposed algorithm is shown to incur a sub-linear regret of $\mathcal {O}\left ({{\sqrt {d n\mathop {\mathrm {polylog}} (n) }}}\right)$ with respect to an oracle that always plays a multiple of $\mathbb {E}\;\theta _{t}$ . The main challenge in this setting is to ensure that the exploration-exploitation strategy is robust against long-range dependencies. The proposed method relies on Berbee’s coupling lemma to carefully select near-independent samples and construct confidence ellipsoids around empirical estimates of $\mathbb {E}\;\theta _{t}$ .