Parallel Control for Nonzero-Sum Games With Completely Unknown Nonlinear Dynamics via Reinforcement Learning

IF 8.6 1区计算机科学 Q1 AUTOMATION & CONTROL SYSTEMS

IEEE Transactions on Systems Man Cybernetics-Systems Pub Date : 2025-01-22 DOI:10.1109/TSMC.2025.3526357

Jingwei Lu;Qinglai Wei;Fei-Yue Wang

{"title":"Parallel Control for Nonzero-Sum Games With Completely Unknown Nonlinear Dynamics via Reinforcement Learning","authors":"Jingwei Lu;Qinglai Wei;Fei-Yue Wang","doi":"10.1109/TSMC.2025.3526357","DOIUrl":null,"url":null,"abstract":"This article utilizes parallel control to investigate the problem of continuous-time (CT) nonzero-sum games (NZSGs) for completely unknown nonlinear systems via reinforcement learning (RL), and a parallel control-based NZSG (PNZSG) method is developed without reconstructing unknown dynamics or employing off-policy integral RL (IRL). First, novel dynamic control policies (DCPs) are developed for NZSGs by introducing controls into feedback, and an augmented system with augmented performance indices is constructed to derive the DCPs. Then, we theoretically analyze the effect of the DCPs on the control stability and performance indices, and the optimality of PNZSG is proven to be equivalent to the optimality of the original NZSGs. Subsequently, an IRL technique is employed to achieve the developed PNZSG method, and we show that no prior knowledge of the dynamics of NZSGs is needed to deploy the developed PNZSG method because of the augmented system and performance indices. Finally, numerical examples, including cooperative adaptive cruise control (CACC) of a vehicular platoon, demonstrate the correctness of the developed PNZSG method. The associated code is available at: <uri>https://github.com/lujingweihh/Adaptive-dynamic-programming-algorithms/tree/main/model_free_nonzero_sum_games</uri>.","PeriodicalId":48915,"journal":{"name":"IEEE Transactions on Systems Man Cybernetics-Systems","volume":"55 4","pages":"2884-2896"},"PeriodicalIF":8.6000,"publicationDate":"2025-01-22","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"IEEE Transactions on Systems Man Cybernetics-Systems","FirstCategoryId":"94","ListUrlMain":"https://ieeexplore.ieee.org/document/10849990/","RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"AUTOMATION & CONTROL SYSTEMS","Score":null,"Total":0}

引用次数: 0

Abstract

This article utilizes parallel control to investigate the problem of continuous-time (CT) nonzero-sum games (NZSGs) for completely unknown nonlinear systems via reinforcement learning (RL), and a parallel control-based NZSG (PNZSG) method is developed without reconstructing unknown dynamics or employing off-policy integral RL (IRL). First, novel dynamic control policies (DCPs) are developed for NZSGs by introducing controls into feedback, and an augmented system with augmented performance indices is constructed to derive the DCPs. Then, we theoretically analyze the effect of the DCPs on the control stability and performance indices, and the optimality of PNZSG is proven to be equivalent to the optimality of the original NZSGs. Subsequently, an IRL technique is employed to achieve the developed PNZSG method, and we show that no prior knowledge of the dynamics of NZSGs is needed to deploy the developed PNZSG method because of the augmented system and performance indices. Finally, numerical examples, including cooperative adaptive cruise control (CACC) of a vehicular platoon, demonstrate the correctness of the developed PNZSG method. The associated code is available at: https://github.com/lujingweihh/Adaptive-dynamic-programming-algorithms/tree/main/model_free_nonzero_sum_games.

查看原文本刊更多论文

本文利用并行控制技术，通过强化学习（RL）研究了完全未知非线性系统的连续时间（CT）非零和博弈（NZSG）问题，并开发了一种基于并行控制的 NZSG（PNZSG）方法，而无需重建未知动力学或采用非策略积分 RL（IRL）。首先，通过在反馈中引入控制，为 NZSG 开发了新颖的动态控制策略（DCP），并构建了一个具有增强性能指标的增强系统来推导 DCP。然后，我们从理论上分析了 DCP 对控制稳定性和性能指标的影响，并证明 PNZSG 的最优性等同于原始 NZSG 的最优性。随后，我们利用 IRL 技术实现了所开发的 PNZSG 方法，并证明由于系统和性能指标的增强，部署所开发的 PNZSG 方法无需事先了解 NZSG 的动力学知识。最后，包括车辆排的合作自适应巡航控制（CACC）在内的数值示例证明了所开发的 PNZSG 方法的正确性。相关代码请访问：https://github.com/lujingweihh/Adaptive-dynamic-programming-algorithms/tree/main/model_free_nonzero_sum_games。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

IEEE Transactions on Systems Man Cybernetics-Systems AUTOMATION & CONTROL SYSTEMS-COMPUTER SCIENCE, CYBERNETICS

CiteScore

18.50

自引率

11.50%

发文量

812

审稿时长

6 months

期刊介绍： The IEEE Transactions on Systems, Man, and Cybernetics: Systems encompasses the fields of systems engineering, covering issue formulation, analysis, and modeling throughout the systems engineering lifecycle phases. It addresses decision-making, issue interpretation, systems management, processes, and various methods such as optimization, modeling, and simulation in the development and deployment of large systems.