Wenhao Ma;Yu-Chen Chang;Jie Yang;Yu-Kai Wang;Chin-Teng Lin
{"title":"Contrastive Learning-Based Agent Modeling for Deep Reinforcement Learning","authors":"Wenhao Ma;Yu-Chen Chang;Jie Yang;Yu-Kai Wang;Chin-Teng Lin","doi":"10.1109/TETCI.2025.3595684","DOIUrl":null,"url":null,"abstract":"Multi-agent systems often require agents to collaborate with or compete against other agents with diverse goals, behaviors, or strategies. Agent modeling is essential when designing adaptive policies for intelligent machine agents in multi-agent systems, as this is the means by which the controlled agent (ego agent) understands other agents' (modeled agents) behavior and extracts their meaningful policy representations. These representations can be used to enhance the ego agent's adaptive policy which is trained by reinforcement learning. However, existing agent modeling approaches typically assume the availability of local observations from modeled agents during training or a long observation trajectory for policy adaption. To remove these constrictive assumptions and improve agent modeling performance, we devised a <bold>C</b>ontrastive <bold>L</b>earning-based <bold>A</b>gent <bold>M</b>odeling (<bold>CLAM</b>) method that relies only on the local observations from the ego agent during training and execution. With these observations, CLAM is capable of generating consistent high-quality policy representations in real time right from the beginning of each episode. We evaluated the efficacy of our approach in both cooperative and competitive multi-agent environments. The experiment results demonstrate that our approach improves reinforcement learning performance by at least 28% on cooperative and competitive tasks, which exceeds the state-of-the-art.","PeriodicalId":13135,"journal":{"name":"IEEE Transactions on Emerging Topics in Computational Intelligence","volume":"9 5","pages":"3719-3726"},"PeriodicalIF":5.3000,"publicationDate":"2025-08-13","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"IEEE Transactions on Emerging Topics in Computational Intelligence","FirstCategoryId":"94","ListUrlMain":"https://ieeexplore.ieee.org/document/11123723/","RegionNum":3,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"COMPUTER SCIENCE, ARTIFICIAL INTELLIGENCE","Score":null,"Total":0}
引用次数: 0
Abstract
Multi-agent systems often require agents to collaborate with or compete against other agents with diverse goals, behaviors, or strategies. Agent modeling is essential when designing adaptive policies for intelligent machine agents in multi-agent systems, as this is the means by which the controlled agent (ego agent) understands other agents' (modeled agents) behavior and extracts their meaningful policy representations. These representations can be used to enhance the ego agent's adaptive policy which is trained by reinforcement learning. However, existing agent modeling approaches typically assume the availability of local observations from modeled agents during training or a long observation trajectory for policy adaption. To remove these constrictive assumptions and improve agent modeling performance, we devised a Contrastive Learning-based Agent Modeling (CLAM) method that relies only on the local observations from the ego agent during training and execution. With these observations, CLAM is capable of generating consistent high-quality policy representations in real time right from the beginning of each episode. We evaluated the efficacy of our approach in both cooperative and competitive multi-agent environments. The experiment results demonstrate that our approach improves reinforcement learning performance by at least 28% on cooperative and competitive tasks, which exceeds the state-of-the-art.
期刊介绍:
The IEEE Transactions on Emerging Topics in Computational Intelligence (TETCI) publishes original articles on emerging aspects of computational intelligence, including theory, applications, and surveys.
TETCI is an electronics only publication. TETCI publishes six issues per year.
Authors are encouraged to submit manuscripts in any emerging topic in computational intelligence, especially nature-inspired computing topics not covered by other IEEE Computational Intelligence Society journals. A few such illustrative examples are glial cell networks, computational neuroscience, Brain Computer Interface, ambient intelligence, non-fuzzy computing with words, artificial life, cultural learning, artificial endocrine networks, social reasoning, artificial hormone networks, computational intelligence for the IoT and Smart-X technologies.