One-Armed Bandit Problem and the Mirror Descent Algorithm

Abstract

The application of the mirror descent algorithm (MDA) in the one-armed bandit problem in the minimax setting in relation to data processing has been considered. This problem has also been known as a game with nature, in which the payoff function of the player is the mathematical expectation of the total income. The player must determine the most effective method of the two available ones during the control process and ensure its preferential use. In this case, the a priori efficiency of one of the methods is known. In this paper, a modification of the MDA that makes it possible to improve the control efficiency by using additional information has been considered. The proposed strategy preserves the characteristic property of strategies for one-armed bandits: if a known action is applied once, it will be applied until the end of control. Modifications for the algorithm for single processing and for its batch version have been considered. Batch processing is interesting in that the total processing time is determined by the number of packets, and not by the original amount of data, with the possibility of providing parallel processing of data in packets. For the proposed algorithms, the optimal values of the adjustable parameters have been calculated using Monte Carlo simulation and minimax risk estimates have been obtained.