Safety-aware Adversarial Inverse Reinforcement Learning (S-AIRL) for Highway Autonomous Driving

Fangjian Li, J. Wagner, Yue Wang
{"title":"Safety-aware Adversarial Inverse Reinforcement Learning (S-AIRL) for Highway Autonomous Driving","authors":"Fangjian Li, J. Wagner, Yue Wang","doi":"10.1115/1.4053427","DOIUrl":null,"url":null,"abstract":"\n Inverse reinforcement learning (IRL) has been successfully applied in many robotics and autonomous driving studies without the need for hand-tuning a reward function. However, it suffers from safety issues. Compared to the reinforcement learning (RL) algorithms, IRL is even more vulnerable to unsafe situations as it can only infer the importance of safety based on expert demonstrations. In this paper, we propose a safety-aware adversarial inverse reinforcement learning algorithm (S-AIRL). First, the control barrier function (CBF) is used to guide the training of a safety critic, which leverages the knowledge of system dynamics in the sampling process without training an additional guiding policy. The trained safety critic is then integrated into the discriminator to help discern the generated data and expert demonstrations from the standpoint of safety. Finally, to further improve the safety awareness, a regulator is introduced in the loss function of the discriminator training to prevent the recovered reward function from assigning high rewards to the risky behaviors. We tested our S-AIRL in the highway autonomous driving scenario. Comparing to the original AIRL algorithm, with the same level of imitation learning (IL) performance, the proposed S-AIRL can reduce the collision rate by 32.6%.","PeriodicalId":164923,"journal":{"name":"Journal of Autonomous Vehicles and Systems","volume":"123 1","pages":"0"},"PeriodicalIF":0.0000,"publicationDate":"2022-01-07","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"1","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Journal of Autonomous Vehicles and Systems","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/10.1115/1.4053427","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"","JCRName":"","Score":null,"Total":0}
引用次数: 1

Abstract

Inverse reinforcement learning (IRL) has been successfully applied in many robotics and autonomous driving studies without the need for hand-tuning a reward function. However, it suffers from safety issues. Compared to the reinforcement learning (RL) algorithms, IRL is even more vulnerable to unsafe situations as it can only infer the importance of safety based on expert demonstrations. In this paper, we propose a safety-aware adversarial inverse reinforcement learning algorithm (S-AIRL). First, the control barrier function (CBF) is used to guide the training of a safety critic, which leverages the knowledge of system dynamics in the sampling process without training an additional guiding policy. The trained safety critic is then integrated into the discriminator to help discern the generated data and expert demonstrations from the standpoint of safety. Finally, to further improve the safety awareness, a regulator is introduced in the loss function of the discriminator training to prevent the recovered reward function from assigning high rewards to the risky behaviors. We tested our S-AIRL in the highway autonomous driving scenario. Comparing to the original AIRL algorithm, with the same level of imitation learning (IL) performance, the proposed S-AIRL can reduce the collision rate by 32.6%.
公路自动驾驶安全感知对抗逆强化学习(S-AIRL)
逆强化学习(IRL)已经成功地应用于许多机器人和自动驾驶研究中,而无需手动调整奖励函数。然而,它存在安全问题。与强化学习(RL)算法相比,IRL算法更容易受到不安全情况的影响,因为它只能根据专家论证来推断安全的重要性。在本文中,我们提出了一种安全感知对抗逆强化学习算法(S-AIRL)。首先,使用控制障碍函数(CBF)来指导安全评论家的培训,该方法利用采样过程中的系统动力学知识,而无需训练额外的指导策略。然后,训练有素的安全评论家被整合到鉴别器中,以帮助从安全的角度辨别生成的数据和专家演示。最后,为了进一步提高安全意识,在鉴别器训练的损失函数中引入了一个调节器,以防止恢复后的奖励函数给风险行为分配高奖励。我们在高速公路自动驾驶场景中测试了S-AIRL。与原始的AIRL算法相比,在相同的模仿学习性能下,本文提出的S-AIRL算法可以将碰撞率降低32.6%。
本文章由计算机程序翻译,如有差异,请以英文原文为准。
求助全文
约1分钟内获得全文 求助全文
来源期刊
自引率
0.00%
发文量
0
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
确定
请完成安全验证×
copy
已复制链接
快去分享给好友吧!
我知道了
右上角分享
点击右上角分享
0
联系我们:info@booksci.cn Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。 Copyright © 2023 布克学术 All rights reserved.
京ICP备2023020795号-1
ghs 京公网安备 11010802042870号
Book学术文献互助
Book学术文献互助群
群 号:481959085
Book学术官方微信