Reinforcement learning journal最新文献

Solvable models of learning to pursue a moving target. 学习追逐移动目标的可解模型。

Reinforcement learning journal Pub Date : 2026-01-01

John J Vastola, Kanaka Rajan

{"title":"Solvable models of learning to pursue a moving target.","authors":"John J Vastola, Kanaka Rajan","doi":"","DOIUrl":"","url":null,"abstract":"Agents tasked with intercepting a moving target must learn how to reach it, ideally as efficiently as possible. As observers of animal pursuit behavior have noticed, different strategies are possible: on one extreme, the agent reactively moves toward the target's current location; on the other, the agent predicts the target's future location and moves directly there. Motivated by the desire to understand how such strategies might be learned, we introduce a continuous-time linear-quadratic model of open-loop target pursuit whose optimal control strategies interpolate between these possibilities. Usefully, a small number of interpretable parameters control which type of strategy is optimal, and it is possible to derive closed-form solutions for optimal strategies, policy learning dynamics, and value learning dynamics. Exploiting our model's linear structure, we find that the time scales of learning precisely correspond to the eigenvalues of certain matrices, and that relevant eigenvalue spectra generically have a gap. We show that this gap indicates that agents tend to learn how to reach their target before optimizing their movement. Our results provide a detailed mathematical characterization of pursuit behavior and its learning dynamics, which can both serve as a benchmark for empirical work, and as a foundation for more elaborate theoretical treatments of pursuit.","PeriodicalId":521307,"journal":{"name":"Reinforcement learning journal","volume":"7 ","pages":""},"PeriodicalIF":0.0,"publicationDate":"2026-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC13568698/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"148957824","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Non-Stationary Latent Auto-Regressive Bandits. 非平稳潜伏自回归土匪。

Reinforcement learning journal Pub Date : 2025-08-01

Anna L Trella, Walter Dempsey, Asim H Gazi, Ziping Xu, Finale Doshi-Velez, Susan A Murphy

{"title":"Non-Stationary Latent Auto-Regressive Bandits.","authors":"Anna L Trella, Walter Dempsey, Asim H Gazi, Ziping Xu, Finale Doshi-Velez, Susan A Murphy","doi":"","DOIUrl":"","url":null,"abstract":"For the non-stationary multi-armed bandit (MAB) problem, many existing methods allow a general mechanism for the non-stationarity, but rely on a budget for the non-stationarity that is sub-linear to the total number of time steps <math><mi>T</mi></math> . In many real-world settings, however, the mechanism for the non-stationarity can be modeled, but there is no budget for the non-stationarity. We instead consider the non-stationary bandit problem where the reward means change due to a latent, auto-regressive (AR) state. We develop Latent AR LinUCB (LARL), an online linear contextual bandit algorithm that does not rely on the non-stationary budget, but instead forms good predictions of reward means by implicitly predicting the latent state. The key idea is to reduce the problem to a linear dynamical system which can be solved as a linear contextual bandit. In fact, LARL approximates a steady-state Kalman filter and efficiently learns system parameters online. We provide an interpretable regret bound for LARL with respect to the level of non-stationarity in the environment. LARL achieves sub-linear regret in this setting if the noise variance of the latent state process is sufficiently small with respect to <math><mi>T</mi></math> . Empirically, LARL outperforms various baseline methods in this non-stationary bandit problem.","PeriodicalId":521307,"journal":{"name":"Reinforcement learning journal","volume":"6 ","pages":"765-789"},"PeriodicalIF":0.0,"publicationDate":"2025-08-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC12987683/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"147464725","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Empirical Bound Information-Directed Sampling for Norm-Agnostic Bandits. 规范不可知论强盗的经验界信息导向抽样。

Reinforcement learning journal Pub Date : 2025-08-01

Piotr M Suder, Eric Laber

{"title":"Empirical Bound Information-Directed Sampling for Norm-Agnostic Bandits.","authors":"Piotr M Suder, Eric Laber","doi":"","DOIUrl":"","url":null,"abstract":"Information-directed sampling (IDS) is a powerful framework for solving bandit problems which has shown strong results in both Bayesian and frequentist settings. However, frequentist IDS, like many other bandit algorithms, requires that one have prior knowledge of a (relatively) tight upper bound on the norm of the true parameter vector governing the reward model in order to achieve good performance. Unfortunately, this requirement is rarely satisfied in practice. As we demonstrate, using a poorly calibrated bound can lead to significant regret accumulation. To address this issue, we introduce a novel frequentist IDS algorithm that iteratively refines a high-probability upper bound on the true parameter norm using accumulating data. We focus on the linear bandit setting with heteroskedastic subgaussian noise. Our method leverages a mixture of relevant information gain criteria to balance exploration aimed at tightening the estimated parameter norm bound and directly searching for the optimal action. We establish regret bounds for our algorithm that do not depend on an initially assumed parameter norm bound and demonstrate that our method outperforms state-of-the-art IDS and UCB algorithms.","PeriodicalId":521307,"journal":{"name":"Reinforcement learning journal","volume":"6 ","pages":"2788-2819"},"PeriodicalIF":0.0,"publicationDate":"2025-08-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC13061422/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"147648807","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

When and Why Hyperbolic Discounting Matters for Reinforcement Learning Interventions. 何时以及为什么双曲折现对强化学习干预很重要。

Reinforcement learning journal Pub Date : 2025-08-01

Ian M Moore, Eura Nofshin, Siddharth Swaroop, Susan Murphy, Finale Doshi-Velez, Weiwei Pan

{"title":"When and Why Hyperbolic Discounting Matters for Reinforcement Learning Interventions.","authors":"Ian M Moore, Eura Nofshin, Siddharth Swaroop, Susan Murphy, Finale Doshi-Velez, Weiwei Pan","doi":"","DOIUrl":"","url":null,"abstract":"In settings where an AI agent nudges a human agent toward a goal, the quality of the AI's policy depends on how well it models the human. Despite behavioral evidence that humans hyperbolically discount future rewards, the RL community continues to model humans as Markov Decision Processes (MDPs) with exponential discounting. This is because planning is difficult with non-exponential discounts. In this work, we investigate whether the performance benefits of modeling humans as hyperbolic discounters outweigh the computational costs. We focus on AI interventions that change the human's discounting (i.e. decreases the human's \"nearsightedness\" to help them toward distant goals). We derive a fixed exponential discount factor that can approximate hyperbolic discounting, and prove that this approximation guarantees the AI will never miss a necessary intervention. We also prove that our approximation causes fewer false positives (unnecessary interventions) than the mean hazard rate, another well-known method for approximating hyperbolic MDPs as exponential ones. Surprisingly, our experiments demonstrate that exponential approximations outperform hyperbolic ones in online learning, even when the ground-truth human MDP is hyperbolically discounted.","PeriodicalId":521307,"journal":{"name":"Reinforcement learning journal","volume":"6 ","pages":"690-712"},"PeriodicalIF":0.0,"publicationDate":"2025-08-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC12987681/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"147464717","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0