Primal-Dual Regression Approach for Markov Decision Processes with General State and Action Spaces

IF 2.2 2区数学 Q2 AUTOMATION & CONTROL SYSTEMS

SIAM Journal on Control and Optimization Pub Date : 2024-02-12 DOI:10.1137/22m1526010

Denis Belomestny, John Schoenmakers

引用次数: 0

Abstract

SIAM Journal on Control and Optimization, Volume 62, Issue 1, Page 650-679, February 2024.
Abstract. We develop a regression-based primal-dual martingale approach for solving discrete time, finite-horizon MDPs. The state and action spaces may be finite or infinite (but regular enough) subsets of Euclidean space. Consequently, our method allows for the construction of tight upper and lower-biased approximations of the value functions, providing precise estimates of the optimal policy. Importantly, we prove error bounds for the estimated duality gap featuring polynomial dependence on the time horizon. Additionally, we observe sublinear dependence of the stochastic part of the error on the cardinality/dimension of the state and action spaces. From a computational perspective, our proposed method is efficient. Unlike typical duality-based methods for optimal control problems in the literature, the Monte Carlo procedures involved here do not require nested simulations.

查看原文本刊更多论文

具有一般状态和行动空间的马尔可夫决策过程的原始-双重回归方法

SIAM 控制与优化期刊》第 62 卷第 1 期第 650-679 页，2024 年 2 月。摘要我们开发了一种基于回归的原始-双鞅方法，用于求解离散时间、有限视距 MDP。状态空间和行动空间可以是欧几里得空间的有限或无限（但足够规则）子集。因此，我们的方法可以构建严格的上偏和下偏值函数近似值，提供最优策略的精确估计。重要的是，我们证明了估计对偶差距的误差边界，其特点是对时间跨度的多项式依赖。此外，我们还观察到误差的随机部分与状态和行动空间的心率/维度存在亚线性关系。从计算角度来看，我们提出的方法是高效的。与文献中基于二元性的最优控制问题典型方法不同，这里涉及的蒙特卡罗程序不需要嵌套模拟。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

SIAM Journal on Control and Optimization 数学-应用数学

CiteScore

4.00

自引率

4.50%

发文量

143

审稿时长

12 months

期刊介绍： SIAM Journal on Control and Optimization (SICON) publishes original research articles on the mathematics and applications of control theory and certain parts of optimization theory. Papers considered for publication must be significant at both the mathematical level and the level of applications or potential applications. Papers containing mostly routine mathematics or those with no discernible connection to control and systems theory or optimization will not be considered for publication. From time to time, the journal will also publish authoritative surveys of important subject areas in control theory and optimization whose level of maturity permits a clear and unified exposition. The broad areas mentioned above are intended to encompass a wide range of mathematical techniques and scientific, engineering, economic, and industrial applications. These include stochastic and deterministic methods in control, estimation, and identification of systems; modeling and realization of complex control systems; the numerical analysis and related computational methodology of control processes and allied issues; and the development of mathematical theories and techniques that give new insights into old problems or provide the basis for further progress in control theory and optimization. Within the field of optimization, the journal focuses on the parts that are relevant to dynamic and control systems. Contributions to numerical methodology are also welcome in accordance with these aims, especially as related to large-scale problems and decomposition as well as to fundamental questions of convergence and approximation.