Stochastic approximation in non-Markovian environments

Abstract

We analyze a stochastic approximation scheme driven by non-Markovian noise in addition to the standard martingale difference noise. Our main result is that it behaves like a stochastic approximation driven by an ‘equivalent’ Markov noise, i.e., has the same convergence properties, although the convergence rate may degrade. Some implications to reinforcement learning algorithms are discussed.