ArXiv.org最新文献

筛选

英文中文

ABPT: Amended Backpropagation through Time with Partially Differentiable Rewards 基于部分可微奖励的修正时间反向传播

ArXiv.org Pub Date : 2025-01-24 DOI: 10.48550/arxiv.2501.14513

Fanxing Li, Fangyu Sun, Tingrong Zhang, Danping Zou

{"title":"ABPT: Amended Backpropagation through Time with Partially Differentiable Rewards","authors":"Fanxing Li, Fangyu Sun, Tingrong Zhang, Danping Zou","doi":"10.48550/arxiv.2501.14513","DOIUrl":"https://doi.org/10.48550/arxiv.2501.14513","url":null,"abstract":"Quadrotor control policies can be trained with high performance using the exact gradients of the rewards to directly optimize policy parameters via backpropagation-through-time (BPTT). However, designing a fully differentiable reward architecture is often challenging. Partially differentiable rewards will result in biased gradient propagation that degrades training performance. To overcome this limitation, we propose Amended Backpropagation-through-Time (ABPT), a novel approach that mitigates gradient bias while preserving the training efficiency of BPTT. ABPT combines 0-step and N-step returns, effectively reducing the bias by leveraging value gradients from the learned Q-value function. Additionally, it adopts entropy regularization and state initialization mechanisms to encourage exploration during training. We evaluate ABPT on four representative quadrotor flight tasks li{in both real world and simulation}. Experimental results demonstrate that ABPT converges significantly faster and achieves higher ultimate rewards than existing learning algorithms, particularly in tasks involving partially differentiable rewards. The code will be released at http://github.com/Fanxing-LI/ABPT.","PeriodicalId":522659,"journal":{"name":"ArXiv.org","volume":"1 1","pages":""},"PeriodicalIF":0.0,"publicationDate":"2025-01-24","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://arxiv.org/pdf/2501.14513","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"147901216","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0