Linear Program-Based Policies for Restless Bandits: Necessary and Sufficient Conditions for (Exponentially Fast) Asymptotic Optimality

IF 1.4 3区数学 Q2 MATHEMATICS, APPLIED

Mathematics of Operations Research Pub Date : 2023-12-01 DOI:10.1287/moor.2022.0101

Nicolas Gast, Bruno Gaujal, Chen Yan

{"title":"Linear Program-Based Policies for Restless Bandits: Necessary and Sufficient Conditions for (Exponentially Fast) Asymptotic Optimality","authors":"Nicolas Gast, Bruno Gaujal, Chen Yan","doi":"10.1287/moor.2022.0101","DOIUrl":null,"url":null,"abstract":"We provide a framework to analyze control policies for the restless Markovian bandit model under both finite and infinite time horizons. We show that when the population of arms goes to infinity, the value of the optimal control policy converges to the solution of a linear program (LP). We provide necessary and sufficient conditions for a generic control policy to be (i) asymptotically optimal, (ii) asymptotically optimal with square root convergence rate, and (iii) asymptotically optimal with exponential rate. We then construct the LP-index policy that is asymptotically optimal with square root convergence rate on all models and with exponential rate if the model is nondegenerate in finite horizon and satisfies a uniform global attractor property in infinite horizon. We next define the LP-update policy, which is essentially a repeated LP-index policy that solves a new LP at each decision epoch. We conclude by providing numerical experiments to compare the efficiency of different LP-based policies.Funding: This work was supported by Agence Nationale de la Recherche [Grant ANR-19-CE23-0015].","PeriodicalId":49852,"journal":{"name":"Mathematics of Operations Research","volume":"232 1","pages":""},"PeriodicalIF":1.4000,"publicationDate":"2023-12-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Mathematics of Operations Research","FirstCategoryId":"100","ListUrlMain":"https://doi.org/10.1287/moor.2022.0101","RegionNum":3,"RegionCategory":"数学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q2","JCRName":"MATHEMATICS, APPLIED","Score":null,"Total":0}

引用次数: 0

Abstract

We provide a framework to analyze control policies for the restless Markovian bandit model under both finite and infinite time horizons. We show that when the population of arms goes to infinity, the value of the optimal control policy converges to the solution of a linear program (LP). We provide necessary and sufficient conditions for a generic control policy to be (i) asymptotically optimal, (ii) asymptotically optimal with square root convergence rate, and (iii) asymptotically optimal with exponential rate. We then construct the LP-index policy that is asymptotically optimal with square root convergence rate on all models and with exponential rate if the model is nondegenerate in finite horizon and satisfies a uniform global attractor property in infinite horizon. We next define the LP-update policy, which is essentially a repeated LP-index policy that solves a new LP at each decision epoch. We conclude by providing numerical experiments to compare the efficiency of different LP-based policies.Funding: This work was supported by Agence Nationale de la Recherche [Grant ANR-19-CE23-0015].

查看原文本刊更多论文

基于线性规划的不宁土匪策略:(指数快速)渐近最优性的充分必要条件

我们提供了一个框架来分析不宁马尔可夫强盗模型在有限和无限时间范围下的控制策略。我们证明了当武器数量趋于无穷时，最优控制策略的值收敛于线性规划(LP)的解。给出了一般控制策略(i)渐近最优，(ii)以平方根收敛速率渐近最优，(iii)以指数速率渐近最优的充要条件。然后构造了在所有模型上具有平方根收敛率的渐近最优的lp -指数策略，当模型在有限视界上非退化且在无限视界上满足一致全局吸引子性质时具有指数收敛率。接下来我们定义LP更新策略，它本质上是一个重复的LP索引策略，在每个决策时期解决一个新的LP。最后，我们通过提供数值实验来比较不同基于lp的策略的效率。本研究由法国国家研究机构(Agence Nationale de la Recherche)资助[Grant ANR-19-CE23-0015]。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

Mathematics of Operations Research 管理科学-应用数学

CiteScore

3.40

自引率

5.90%

发文量

178

审稿时长

15.0 months

期刊介绍： Mathematics of Operations Research is an international journal of the Institute for Operations Research and the Management Sciences (INFORMS). The journal invites articles concerned with the mathematical and computational foundations in the areas of continuous, discrete, and stochastic optimization; mathematical programming; dynamic programming; stochastic processes; stochastic models; simulation methodology; control and adaptation; networks; game theory; and decision theory. Also sought are contributions to learning theory and machine learning that have special relevance to decision making, operations research, and management science. The emphasis is on originality, quality, and importance; correctness alone is not sufficient. Significant developments in operations research and management science not having substantial mathematical interest should be directed to other journals such as Management Science or Operations Research.