未折现不定视界mdp的离线极大极小q函数学习

IF 0.6 4区数学 Q3 STATISTICS & PROBABILITY

Annals of the Institute of Statistical Mathematics Pub Date : 2025-04-21 DOI:10.1007/s10463-025-00924-1

Fengying Li, Yuqiang Li, Xianyi Wu, Wei Bai

{"title":"未折现不定视界mdp的离线极大极小q函数学习","authors":"Fengying Li, Yuqiang Li, Xianyi Wu, Wei Bai","doi":"10.1007/s10463-025-00924-1","DOIUrl":null,"url":null,"abstract":"<div><p>This work considers the offline evaluation problem for indefinite-horizon Markov Decision Processes. A minimax Q-function learning algorithm is proposed, which, instead of i.i.d. tuples <span>\\((s,a,s',r)\\)</span>, evaluates undiscounted expected return based by i.i.d. trajectories truncated at a given time step. The confidence error bounds are developed. Experiments using Open AI’s Cart Pole environment are employed to demonstrate the algorithm.</p></div>","PeriodicalId":55511,"journal":{"name":"Annals of the Institute of Statistical Mathematics","volume":"77 4","pages":"535 - 562"},"PeriodicalIF":0.6000,"publicationDate":"2025-04-21","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":"{\"title\":\"Offline minimax Q-function learning for undiscounted indefinite-horizon MDPs\",\"authors\":\"Fengying Li, Yuqiang Li, Xianyi Wu, Wei Bai\",\"doi\":\"10.1007/s10463-025-00924-1\",\"DOIUrl\":null,\"url\":null,\"abstract\":\"<div><p>This work considers the offline evaluation problem for indefinite-horizon Markov Decision Processes. A minimax Q-function learning algorithm is proposed, which, instead of i.i.d. tuples <span>\\\\((s,a,s',r)\\\\)</span>, evaluates undiscounted expected return based by i.i.d. trajectories truncated at a given time step. The confidence error bounds are developed. Experiments using Open AI’s Cart Pole environment are employed to demonstrate the algorithm.</p></div>\",\"PeriodicalId\":55511,\"journal\":{\"name\":\"Annals of the Institute of Statistical Mathematics\",\"volume\":\"77 4\",\"pages\":\"535 - 562\"},\"PeriodicalIF\":0.6000,\"publicationDate\":\"2025-04-21\",\"publicationTypes\":\"Journal Article\",\"fieldsOfStudy\":null,\"isOpenAccess\":false,\"openAccessPdf\":\"\",\"citationCount\":\"0\",\"resultStr\":null,\"platform\":\"Semanticscholar\",\"paperid\":null,\"PeriodicalName\":\"Annals of the Institute of Statistical Mathematics\",\"FirstCategoryId\":\"100\",\"ListUrlMain\":\"https://link.springer.com/article/10.1007/s10463-025-00924-1\",\"RegionNum\":4,\"RegionCategory\":\"数学\",\"ArticlePicture\":[],\"TitleCN\":null,\"AbstractTextCN\":null,\"PMCID\":null,\"EPubDate\":\"\",\"PubModel\":\"\",\"JCR\":\"Q3\",\"JCRName\":\"STATISTICS & PROBABILITY\",\"Score\":null,\"Total\":0}","platform":"Semanticscholar","paperid":null,"PeriodicalName":"Annals of the Institute of Statistical Mathematics","FirstCategoryId":"100","ListUrlMain":"https://link.springer.com/article/10.1007/s10463-025-00924-1","RegionNum":4,"RegionCategory":"数学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q3","JCRName":"STATISTICS & PROBABILITY","Score":null,"Total":0}

引用次数: 0

摘要

本文研究了不定视界马尔可夫决策过程的离线评价问题。提出了一种极大极小q函数学习算法，该算法不使用i.i.d元组\((s,a,s',r)\)，而是基于在给定时间步长截断的i.i.d轨迹来评估未贴现期望收益。给出了置信误差范围。利用Open AI的Cart Pole环境进行实验，对算法进行了验证。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

查看原文本刊更多论文

Offline minimax Q-function learning for undiscounted indefinite-horizon MDPs

This work considers the offline evaluation problem for indefinite-horizon Markov Decision Processes. A minimax Q-function learning algorithm is proposed, which, instead of i.i.d. tuples \((s,a,s',r)\), evaluates undiscounted expected return based by i.i.d. trajectories truncated at a given time step. The confidence error bounds are developed. Experiments using Open AI’s Cart Pole environment are employed to demonstrate the algorithm.

求助全文

通过发布文献求助，成功后即可免费获取论文全文。去求助

来源期刊

Annals of the Institute of Statistical Mathematics 数学-统计学与概率论

CiteScore

2.00

自引率

0.00%

发文量

审稿时长

6-12 weeks

期刊介绍： Annals of the Institute of Statistical Mathematics (AISM) aims to provide a forum for open communication among statisticians, and to contribute to the advancement of statistics as a science to enable humans to handle information in order to cope with uncertainties. It publishes high-quality papers that shed new light on the theoretical, computational and/or methodological aspects of statistical science. Emphasis is placed on (a) development of new methodologies motivated by real data, (b) development of unifying theories, and (c) analysis and improvement of existing methodologies and theories.