Multi-Agent DRL-Controlled Connected and Automated Vehicles in Mixed Traffic With Time Delays

IF 7.9 1区工程技术 Q1 ENGINEERING, CIVIL

IEEE Transactions on Intelligent Transportation Systems Pub Date : 2024-09-18 DOI:10.1109/TITS.2024.3435036

Zhuwei Wang;Yi Xue;Lihan Liu;Haijun Zhang;Chunhui Qu;Chao Fang

{"title":"Multi-Agent DRL-Controlled Connected and Automated Vehicles in Mixed Traffic With Time Delays","authors":"Zhuwei Wang;Yi Xue;Lihan Liu;Haijun Zhang;Chunhui Qu;Chao Fang","doi":"10.1109/TITS.2024.3435036","DOIUrl":null,"url":null,"abstract":"The development of intelligent transportation systems (ITS) has attracted significant attention to connected and autonomous vehicles (CAVs). It is urgent to investigate multi-CAV intelligent cruise control solutions in mixed traffic environments. In addition, the impact of platoon dynamics and time delays, induced by shared wireless communications, data processing, and actuation cannot be ignored. This article investigates the development of a multi-agent deep reinforcement learning (MADRL) controller tailored for CAVs operating within mixed and dynamic traffic scenarios that involve time delays. Firstly, the error dynamics in the discrete-time domain for each subplatoon is derived by considering the time-varying delays and leading vehicle states, and then the optimal CAV cruise control problem is formulated. Subsequently, the partially observable Markov game (POMG) is used to construct the multi-agent environment, and then a centralized training decentralized execution (CTDE) algorithm framework is proposed based on the multi-agent deep deterministic policy gradient (MADDPG) method. Finally, the computational complexity and the influence of delay are analyzed. The simulation results illustrate the effectiveness of the proposed intelligent algorithm.","PeriodicalId":13416,"journal":{"name":"IEEE Transactions on Intelligent Transportation Systems","volume":"25 11","pages":"17676-17688"},"PeriodicalIF":7.9000,"publicationDate":"2024-09-18","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"IEEE Transactions on Intelligent Transportation Systems","FirstCategoryId":"5","ListUrlMain":"https://ieeexplore.ieee.org/document/10682601/","RegionNum":1,"RegionCategory":"工程技术","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"ENGINEERING, CIVIL","Score":null,"Total":0}

引用次数: 0

Abstract

The development of intelligent transportation systems (ITS) has attracted significant attention to connected and autonomous vehicles (CAVs). It is urgent to investigate multi-CAV intelligent cruise control solutions in mixed traffic environments. In addition, the impact of platoon dynamics and time delays, induced by shared wireless communications, data processing, and actuation cannot be ignored. This article investigates the development of a multi-agent deep reinforcement learning (MADRL) controller tailored for CAVs operating within mixed and dynamic traffic scenarios that involve time delays. Firstly, the error dynamics in the discrete-time domain for each subplatoon is derived by considering the time-varying delays and leading vehicle states, and then the optimal CAV cruise control problem is formulated. Subsequently, the partially observable Markov game (POMG) is used to construct the multi-agent environment, and then a centralized training decentralized execution (CTDE) algorithm framework is proposed based on the multi-agent deep deterministic policy gradient (MADDPG) method. Finally, the computational complexity and the influence of delay are analyzed. The simulation results illustrate the effectiveness of the proposed intelligent algorithm.

查看原文本刊更多论文

有时间延迟的混合交通中由 DRL 控制的多代理互联和自动驾驶车辆

智能交通系统（ITS）的发展吸引了人们对互联和自动驾驶车辆（CAV）的极大关注。当务之急是研究混合交通环境下的多 CAV 智能巡航控制解决方案。此外，由共享无线通信、数据处理和执行引起的排动态和时间延迟的影响也不容忽视。本文研究了多代理深度强化学习（MADRL）控制器的开发，该控制器专为在涉及时间延迟的混合动态交通场景中运行的 CAV 量身定制。首先，通过考虑时变延迟和前导车辆状态，推导出每个子排在离散时域的误差动态，然后提出最优 CAV 巡航控制问题。随后，利用部分可观测马尔可夫博弈（POMG）构建多代理环境，并基于多代理深度确定性策略梯度（MADDPG）方法提出了集中训练分散执行（CTDE）算法框架。最后，分析了计算复杂度和延迟的影响。仿真结果表明了所提智能算法的有效性。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

IEEE Transactions on Intelligent Transportation Systems 工程技术-工程：电子与电气

CiteScore

14.80

自引率

12.90%

发文量

1872

审稿时长

7.5 months

期刊介绍： The theoretical, experimental and operational aspects of electrical and electronics engineering and information technologies as applied to Intelligent Transportation Systems (ITS). Intelligent Transportation Systems are defined as those systems utilizing synergistic technologies and systems engineering concepts to develop and improve transportation systems of all kinds. The scope of this interdisciplinary activity includes the promotion, consolidation and coordination of ITS technical activities among IEEE entities, and providing a focus for cooperative activities, both internally and externally.