Approximate linear programming for decentralized policy iteration in cooperative multi-agent Markov decision processes

IF 2.1 3区 计算机科学 Q3 AUTOMATION & CONTROL SYSTEMS
Lakshmi Mandal , Chandrashekar Lakshminarayanan , Shalabh Bhatnagar
{"title":"Approximate linear programming for decentralized policy iteration in cooperative multi-agent Markov decision processes","authors":"Lakshmi Mandal ,&nbsp;Chandrashekar Lakshminarayanan ,&nbsp;Shalabh Bhatnagar","doi":"10.1016/j.sysconle.2024.106003","DOIUrl":null,"url":null,"abstract":"<div><div>In this work, we consider a ‘cooperative’ multi-agent Markov decision process (MDP) involving <span><math><mrow><mi>m</mi><mrow><mo>(</mo><mo>&gt;</mo><mn>1</mn><mo>)</mo></mrow></mrow></math></span> agents. At each decision epoch, all the <span><math><mi>m</mi></math></span> agents independently select actions in order to minimize a common long-term cost objective. In the policy iteration process of multi-agent setup, the number of actions grows exponentially with the number of agents, incurring huge computational costs. Thus, recent works consider decentralized policy improvement, where each agent improves its decisions unilaterally, assuming that the decisions of the other agents are fixed. However, exact value functions are considered in the literature, which is computationally expensive for a large number of agents with high dimensional state–action space. Thus, we propose approximate decentralized policy iteration algorithms, using approximate linear programming with function approximation to compute the approximate value function for decentralized policy improvement. Further, we consider (both) cooperative multi-agent finite and infinite horizon discounted MDPs and propose suitable algorithms in each case. Moreover, we provide theoretical guarantees for our algorithms and also demonstrate their advantages over existing state-of-the-art algorithms in the literature.</div></div>","PeriodicalId":49450,"journal":{"name":"Systems & Control Letters","volume":"196 ","pages":"Article 106003"},"PeriodicalIF":2.1000,"publicationDate":"2025-02-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Systems & Control Letters","FirstCategoryId":"94","ListUrlMain":"https://www.sciencedirect.com/science/article/pii/S0167691124002913","RegionNum":3,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q3","JCRName":"AUTOMATION & CONTROL SYSTEMS","Score":null,"Total":0}
引用次数: 0

Abstract

In this work, we consider a ‘cooperative’ multi-agent Markov decision process (MDP) involving m(>1) agents. At each decision epoch, all the m agents independently select actions in order to minimize a common long-term cost objective. In the policy iteration process of multi-agent setup, the number of actions grows exponentially with the number of agents, incurring huge computational costs. Thus, recent works consider decentralized policy improvement, where each agent improves its decisions unilaterally, assuming that the decisions of the other agents are fixed. However, exact value functions are considered in the literature, which is computationally expensive for a large number of agents with high dimensional state–action space. Thus, we propose approximate decentralized policy iteration algorithms, using approximate linear programming with function approximation to compute the approximate value function for decentralized policy improvement. Further, we consider (both) cooperative multi-agent finite and infinite horizon discounted MDPs and propose suitable algorithms in each case. Moreover, we provide theoretical guarantees for our algorithms and also demonstrate their advantages over existing state-of-the-art algorithms in the literature.
协同多智能体马尔可夫决策过程中分散策略迭代的近似线性规划
在这项工作中,我们考虑了一个涉及m(>1)个智能体的“合作”多智能体马尔可夫决策过程(MDP)。在每个决策时期,所有m个智能体独立地选择行动,以最小化共同的长期成本目标。在多智能体设置策略迭代过程中,动作数量随着智能体数量呈指数增长,产生巨大的计算成本。因此,最近的研究考虑了分散的政策改进,其中每个代理单方面改进其决策,假设其他代理的决策是固定的。然而,文献中考虑的是精确值函数,这对于具有高维状态-动作空间的大量智能体来说,计算成本很高。因此,我们提出了近似分散策略迭代算法,使用近似线性规划和函数逼近来计算分散策略改进的近似值函数。进一步,我们考虑了合作多智能体有限和无限视界贴现mdp,并在每种情况下提出了合适的算法。此外,我们为我们的算法提供了理论保证,并在文献中展示了它们相对于现有最先进算法的优势。
本文章由计算机程序翻译,如有差异,请以英文原文为准。
求助全文
约1分钟内获得全文 求助全文
来源期刊
Systems & Control Letters
Systems & Control Letters 工程技术-运筹学与管理科学
CiteScore
4.60
自引率
3.80%
发文量
144
审稿时长
6 months
期刊介绍: Founded in 1981 by two of the pre-eminent control theorists, Roger Brockett and Jan Willems, Systems & Control Letters is one of the leading journals in the field of control theory. The aim of the journal is to allow dissemination of relatively concise but highly original contributions whose high initial quality enables a relatively rapid review process. All aspects of the fields of systems and control are covered, especially mathematically-oriented and theoretical papers that have a clear relevance to engineering, physical and biological sciences, and even economics. Application-oriented papers with sophisticated and rigorous mathematical elements are also welcome.
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
确定
请完成安全验证×
copy
已复制链接
快去分享给好友吧!
我知道了
右上角分享
点击右上角分享
0
联系我们:info@booksci.cn Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。 Copyright © 2023 布克学术 All rights reserved.
京ICP备2023020795号-1
ghs 京公网安备 11010802042870号
Book学术文献互助
Book学术文献互助群
群 号:481959085
Book学术官方微信