An approximate policy iteration viewpoint of actor–critic algorithms

Abstract

In this work, we establish sample complexity guarantees for a broad class of policy-space algorithms for reinforcement learning. A policy-space algorithm comprises an actor for policy improvement and a critic for policy evaluation. For the actor, we analyze update rules such as softmax, $ϵ$-greedy, and the celebrated natural policy gradient (NPG). Unlike traditional gradient-based analyses, we view NPG as an approximate policy iteration method. This perspective allows us to leverage the Bellman operator’s properties to show that NPG (without regularization) achieves geometric convergence to a globally optimal policy with increasing stepsizes. For the critic, we study TD-learning with linear function approximation and off-policy sampling. To address the instability of TD-learning in this setting, we propose a stable framework using multi-step returns and generalized importance sampling factors, including two specific algorithms: $\lambda$-averaged $Q$-trace and two-sided $Q$-trace. We also provide a finite-sample analysis for the critic. Combining the geometric convergence of the actor with the finite-sample results of the critic, we establish for the first time an overall sample complexity of $\tilde{O}\left({ϵ}^{-2}\right)$ for finding an optimal policy (up to a function approximation error) using policy-space methods under off-policy sampling and linear function approximation.