基于强化学习的 DDPG 窄带主动噪声控制技术

IF 1.5 4区 工程技术 Q3 ENGINEERING, MECHANICAL
Seokhoon Ryu, Jihea Lim, Young-Sup Lee
{"title":"基于强化学习的 DDPG 窄带主动噪声控制技术","authors":"Seokhoon Ryu, Jihea Lim, Young-Sup Lee","doi":"10.1007/s12239-024-00102-x","DOIUrl":null,"url":null,"abstract":"<p>This study investigates the use of deep reinforcement learning for active noise control (DRL-ANC) to cancel narrowband noise. The filtered-x least mean square algorithm for ANC, which includes the secondary path model in itself, has been widely used in various applications. If the path model is inaccurate due to the variations of the actual path, control performance and stability of the algorithm can be restricted. To eliminate the effect by the model inaccuracy, it is considered to remove the path model in the novel DRL-ANC strategy. A DRL approach using the deep deterministic policy gradient without any path model is adopted to learn the behavior of a physical environment including the effect of the actual secondary path in real time. However, a temporal credit assignment problem arises due to the time-delayed reward inherent in the secondary path, which means that the current action could not be evaluated by its true. To address this problem, this study proposes a novel definitions of the state and action of the RL agent, specialized in narrowband noise suppression. Additionally, a novel exploration noise is also suggested to enhance effectiveness and practicality of the learning process. Computer simulations and real-time control experiments were conducted, and the results demonstrated that the proposed DRL-ANC algorithm can robustly cope with changes in the secondary path.</p>","PeriodicalId":50338,"journal":{"name":"International Journal of Automotive Technology","volume":"26 1","pages":""},"PeriodicalIF":1.5000,"publicationDate":"2024-08-23","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":"{\"title\":\"Narrowband Active Noise Control with DDPG Based on Reinforcement Learning\",\"authors\":\"Seokhoon Ryu, Jihea Lim, Young-Sup Lee\",\"doi\":\"10.1007/s12239-024-00102-x\",\"DOIUrl\":null,\"url\":null,\"abstract\":\"<p>This study investigates the use of deep reinforcement learning for active noise control (DRL-ANC) to cancel narrowband noise. The filtered-x least mean square algorithm for ANC, which includes the secondary path model in itself, has been widely used in various applications. If the path model is inaccurate due to the variations of the actual path, control performance and stability of the algorithm can be restricted. To eliminate the effect by the model inaccuracy, it is considered to remove the path model in the novel DRL-ANC strategy. A DRL approach using the deep deterministic policy gradient without any path model is adopted to learn the behavior of a physical environment including the effect of the actual secondary path in real time. However, a temporal credit assignment problem arises due to the time-delayed reward inherent in the secondary path, which means that the current action could not be evaluated by its true. To address this problem, this study proposes a novel definitions of the state and action of the RL agent, specialized in narrowband noise suppression. Additionally, a novel exploration noise is also suggested to enhance effectiveness and practicality of the learning process. Computer simulations and real-time control experiments were conducted, and the results demonstrated that the proposed DRL-ANC algorithm can robustly cope with changes in the secondary path.</p>\",\"PeriodicalId\":50338,\"journal\":{\"name\":\"International Journal of Automotive Technology\",\"volume\":\"26 1\",\"pages\":\"\"},\"PeriodicalIF\":1.5000,\"publicationDate\":\"2024-08-23\",\"publicationTypes\":\"Journal Article\",\"fieldsOfStudy\":null,\"isOpenAccess\":false,\"openAccessPdf\":\"\",\"citationCount\":\"0\",\"resultStr\":null,\"platform\":\"Semanticscholar\",\"paperid\":null,\"PeriodicalName\":\"International Journal of Automotive Technology\",\"FirstCategoryId\":\"5\",\"ListUrlMain\":\"https://doi.org/10.1007/s12239-024-00102-x\",\"RegionNum\":4,\"RegionCategory\":\"工程技术\",\"ArticlePicture\":[],\"TitleCN\":null,\"AbstractTextCN\":null,\"PMCID\":null,\"EPubDate\":\"\",\"PubModel\":\"\",\"JCR\":\"Q3\",\"JCRName\":\"ENGINEERING, MECHANICAL\",\"Score\":null,\"Total\":0}","platform":"Semanticscholar","paperid":null,"PeriodicalName":"International Journal of Automotive Technology","FirstCategoryId":"5","ListUrlMain":"https://doi.org/10.1007/s12239-024-00102-x","RegionNum":4,"RegionCategory":"工程技术","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q3","JCRName":"ENGINEERING, MECHANICAL","Score":null,"Total":0}
引用次数: 0

摘要

本研究探讨了利用深度强化学习的主动噪声控制(DRL-ANC)来消除窄带噪声。用于 ANC 的滤波-x 最小均方算法本身包含辅助路径模型,已在各种应用中广泛使用。如果路径模型因实际路径的变化而不准确,算法的控制性能和稳定性就会受到限制。为了消除模型不准确所带来的影响,新型 DRL-ANC 策略考虑去除路径模型。DRL 方法使用深度确定性策略梯度,不使用任何路径模型,以实时学习物理环境的行为,包括实际次要路径的影响。然而,由于次要路径中固有的奖励具有时间延迟性,这意味着当前行动无法通过其真实情况进行评估,从而产生了时间信用分配问题。为了解决这个问题,本研究提出了一种新的 RL 代理状态和行动定义,专门用于窄带噪声抑制。此外,还提出了一种新的探索噪声,以提高学习过程的有效性和实用性。研究人员进行了计算机模拟和实时控制实验,结果表明所提出的 DRL-ANC 算法能够稳健地应对次级路径的变化。
本文章由计算机程序翻译,如有差异,请以英文原文为准。

Narrowband Active Noise Control with DDPG Based on Reinforcement Learning

Narrowband Active Noise Control with DDPG Based on Reinforcement Learning

This study investigates the use of deep reinforcement learning for active noise control (DRL-ANC) to cancel narrowband noise. The filtered-x least mean square algorithm for ANC, which includes the secondary path model in itself, has been widely used in various applications. If the path model is inaccurate due to the variations of the actual path, control performance and stability of the algorithm can be restricted. To eliminate the effect by the model inaccuracy, it is considered to remove the path model in the novel DRL-ANC strategy. A DRL approach using the deep deterministic policy gradient without any path model is adopted to learn the behavior of a physical environment including the effect of the actual secondary path in real time. However, a temporal credit assignment problem arises due to the time-delayed reward inherent in the secondary path, which means that the current action could not be evaluated by its true. To address this problem, this study proposes a novel definitions of the state and action of the RL agent, specialized in narrowband noise suppression. Additionally, a novel exploration noise is also suggested to enhance effectiveness and practicality of the learning process. Computer simulations and real-time control experiments were conducted, and the results demonstrated that the proposed DRL-ANC algorithm can robustly cope with changes in the secondary path.

求助全文
通过发布文献求助,成功后即可免费获取论文全文。 去求助
来源期刊
International Journal of Automotive Technology
International Journal of Automotive Technology 工程技术-工程:机械
CiteScore
3.10
自引率
12.50%
发文量
129
审稿时长
6 months
期刊介绍: The International Journal of Automotive Technology has as its objective the publication and dissemination of original research in all fields of AUTOMOTIVE TECHNOLOGY, SCIENCE and ENGINEERING. It fosters thus the exchange of ideas among researchers in different parts of the world and also among researchers who emphasize different aspects of the foundations and applications of the field. Standing as it does at the cross-roads of Physics, Chemistry, Mechanics, Engineering Design and Materials Sciences, AUTOMOTIVE TECHNOLOGY is experiencing considerable growth as a result of recent technological advances. The Journal, by providing an international medium of communication, is encouraging this growth and is encompassing all aspects of the field from thermal engineering, flow analysis, structural analysis, modal analysis, control, vehicular electronics, mechatronis, electro-mechanical engineering, optimum design methods, ITS, and recycling. Interest extends from the basic science to technology applications with analytical, experimental and numerical studies. The emphasis is placed on contributions that appear to be of permanent interest to research workers and engineers in the field. If furthering knowledge in the area of principal concern of the Journal, papers of primary interest to the innovative disciplines of AUTOMOTIVE TECHNOLOGY, SCIENCE and ENGINEERING may be published. Papers that are merely illustrations of established principles and procedures, even though possibly containing new numerical or experimental data, will generally not be published. When outstanding advances are made in existing areas or when new areas have been developed to a definitive stage, special review articles will be considered by the editors. No length limitations for contributions are set, but only concisely written papers are published. Brief articles are considered on the basis of technical merit.
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
确定
请完成安全验证×
copy
已复制链接
快去分享给好友吧!
我知道了
右上角分享
点击右上角分享
0
联系我们:info@booksci.cn Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。 Copyright © 2023 布克学术 All rights reserved.
京ICP备2023020795号-1
ghs 京公网安备 11010802042870号
Book学术文献互助
Book学术文献互助群
群 号:481959085
Book学术官方微信