Temporal Difference Learning of Area Coverage Control with Multi-Agent Systems

Abstract

We formulate an area coverage control problem with multi-agent systems by using Bellman's principle of optimality. The performance index is composed of the additive contributions of a term quadratic in the control effort and of a positive definite term that depends on the coverage metric. In this way, the reward encodes optimality in the sense of the classical Lloyd's algorithm, where the term depending on the coverage metric weights the energy of the state, and the term depending on the control weights the effort energy. Quasi optimality is achieved by an adaptive control policy using an actor-critic neural networks based reinforcement learning strategy, with quadratic function approximations for the value function and the control policy. Optimal configurations for a team of agents correspond to centroidal Voronoi partitions of the workspace, with each agent converging to the centroid of the respective generalized Voronoi cell. The system's dynamics is written in discrete time form, and the temporal difference form of Bellman's equation is used in the policy iteration learning scheme to train critic and actor weights. Remarkably, the obtained class of solutions is consistent with the one obtained with Lloyd's algorithm, with the advantage that the reinforcement learning formulation allows for a model-free implementation based on data measured along a system's trajectory. By storing an appropriate time history of control actions, the gradient of the value function is numerically approximated, allowing one to run the policy approximation without knowledge of the input dynamics. Direct comparisons with Lloyd's algorithm show the expected slower convergence since Lloyd's trajectories are optimal for the continuous time system.