Distributed policy evaluation over multi-agent network with communication delays

Abstract

This paper investigates the multi-agent policy evaluation problem for distributed reinforcement learning on time-varying directed communication structure with communication delays. In a completely distributed setting, agents jointly learn the value of a given policy through private local evaluation and neighbors’ evaluation. First, we propose the Push-Sum Dual Averaging Algorithm (PS-DAA) to deal with the distributed policy evaluation problem with communication delays. By considering the inevitable communication delays, a more general time-varying directed communication structure, and more realistic state constraints, PS-DAA still achieves sublinear convergence. Further, considering the case where the full update information is unavailable, we extend PS-DAA to the bandit feedback setting, i.e., the values of the sampling points are used instead of the full gradient information. We prove that compared to the full information scheme, the bandit-feedback PS-DAA does not lead to performance degradation. Finally, we verify the effectiveness of the proposed algorithm through two simulation cases.