{"title":"基于回合制零和马尔可夫博弈的强化学习研究","authors":"D. Shah, Varun Somani, Qiaomin Xie, Zhi Xu","doi":"10.1145/3412815.3416888","DOIUrl":null,"url":null,"abstract":"We consider the problem of finding Nash equilibrium for two-player turn-based zero-sum games. Inspired by the AlphaGo Zero (AGZ) algorithm, we develop a Reinforcement Learning based approach. Specifically, we propose Explore-Improve-Supervise (EIS) method that combines \"exploration\", \"policy improvement\" and \"supervised learning\" to find the value function and policy associated with Nash equilibrium. We identify sufficient conditions for convergence and correctness for such an approach. For a concrete instance of EIS where random policy is used for \"exploration\", Monte-Carlo Tree Search is used for \"policy improvement\" and Nearest Neighbors is used for \"supervised learning\", we establish that this method finds an $\\varepsilon$-approximate value function of Nash equilibrium in $\\widetildeO(\\varepsilon^-(d+4))$ steps when the underlying state-space of the game is continuous and d-dimensional. This is nearly optimal as we establish a lower bound of $\\widetildeØmega (\\varepsilon^-(d+2) )$ for any policy.","PeriodicalId":176130,"journal":{"name":"Proceedings of the 2020 ACM-IMS on Foundations of Data Science Conference","volume":"1 1","pages":"0"},"PeriodicalIF":0.0000,"publicationDate":"2020-02-25","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"8","resultStr":"{\"title\":\"On Reinforcement Learning for Turn-based Zero-sum Markov Games\",\"authors\":\"D. Shah, Varun Somani, Qiaomin Xie, Zhi Xu\",\"doi\":\"10.1145/3412815.3416888\",\"DOIUrl\":null,\"url\":null,\"abstract\":\"We consider the problem of finding Nash equilibrium for two-player turn-based zero-sum games. Inspired by the AlphaGo Zero (AGZ) algorithm, we develop a Reinforcement Learning based approach. Specifically, we propose Explore-Improve-Supervise (EIS) method that combines \\\"exploration\\\", \\\"policy improvement\\\" and \\\"supervised learning\\\" to find the value function and policy associated with Nash equilibrium. We identify sufficient conditions for convergence and correctness for such an approach. For a concrete instance of EIS where random policy is used for \\\"exploration\\\", Monte-Carlo Tree Search is used for \\\"policy improvement\\\" and Nearest Neighbors is used for \\\"supervised learning\\\", we establish that this method finds an $\\\\varepsilon$-approximate value function of Nash equilibrium in $\\\\widetildeO(\\\\varepsilon^-(d+4))$ steps when the underlying state-space of the game is continuous and d-dimensional. This is nearly optimal as we establish a lower bound of $\\\\widetildeØmega (\\\\varepsilon^-(d+2) )$ for any policy.\",\"PeriodicalId\":176130,\"journal\":{\"name\":\"Proceedings of the 2020 ACM-IMS on Foundations of Data Science Conference\",\"volume\":\"1 1\",\"pages\":\"0\"},\"PeriodicalIF\":0.0000,\"publicationDate\":\"2020-02-25\",\"publicationTypes\":\"Journal Article\",\"fieldsOfStudy\":null,\"isOpenAccess\":false,\"openAccessPdf\":\"\",\"citationCount\":\"8\",\"resultStr\":null,\"platform\":\"Semanticscholar\",\"paperid\":null,\"PeriodicalName\":\"Proceedings of the 2020 ACM-IMS on Foundations of Data Science Conference\",\"FirstCategoryId\":\"1085\",\"ListUrlMain\":\"https://doi.org/10.1145/3412815.3416888\",\"RegionNum\":0,\"RegionCategory\":null,\"ArticlePicture\":[],\"TitleCN\":null,\"AbstractTextCN\":null,\"PMCID\":null,\"EPubDate\":\"\",\"PubModel\":\"\",\"JCR\":\"\",\"JCRName\":\"\",\"Score\":null,\"Total\":0}","platform":"Semanticscholar","paperid":null,"PeriodicalName":"Proceedings of the 2020 ACM-IMS on Foundations of Data Science Conference","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/10.1145/3412815.3416888","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"","JCRName":"","Score":null,"Total":0}
引用次数: 8
摘要
我们考虑寻找基于回合制的二人零和博弈的纳什均衡问题。受AlphaGo Zero (AGZ)算法的启发,我们开发了一种基于强化学习的方法。具体而言,我们提出了探索-改进-监督(EIS)方法,将“探索”、“政策改进”和“监督学习”相结合,寻找与纳什均衡相关的价值函数和策略。我们确定了这种方法的收敛性和正确性的充分条件。对于使用随机策略进行“探索”,使用蒙特卡罗树搜索进行“策略改进”,使用最近邻进行“监督学习”的EIS的具体实例,我们建立了该方法在$\ widdetildeo (\varepsilon^-(d+4))$步中找到纳什均衡的$\varepsilon$-近值函数,当博弈的底层状态空间是连续和d维的。这几乎是最优的,因为我们为任何策略建立了$\widetildeØmega (\varepsilon^-(d+2))$的下界。
On Reinforcement Learning for Turn-based Zero-sum Markov Games
We consider the problem of finding Nash equilibrium for two-player turn-based zero-sum games. Inspired by the AlphaGo Zero (AGZ) algorithm, we develop a Reinforcement Learning based approach. Specifically, we propose Explore-Improve-Supervise (EIS) method that combines "exploration", "policy improvement" and "supervised learning" to find the value function and policy associated with Nash equilibrium. We identify sufficient conditions for convergence and correctness for such an approach. For a concrete instance of EIS where random policy is used for "exploration", Monte-Carlo Tree Search is used for "policy improvement" and Nearest Neighbors is used for "supervised learning", we establish that this method finds an $\varepsilon$-approximate value function of Nash equilibrium in $\widetildeO(\varepsilon^-(d+4))$ steps when the underlying state-space of the game is continuous and d-dimensional. This is nearly optimal as we establish a lower bound of $\widetildeØmega (\varepsilon^-(d+2) )$ for any policy.