Greedy-DAgger - A Student Rollout Efficient Imitation Learning Algorithm

Abstract

Sampling-based model predictive control algorithms can be computationally expensive and may not be feasible for restricted platforms such as quadcopters. Comparatively speaking, lightweight learned controllers are computationally cheaper and may be more suited for these platforms. Expert control samples provided by a remote model predictive control algorithm could be used to rapidly train a student policy. We present Greedy-DAgger, a hybrid-policy imitation learning approach that leverages expert simulations to improve the student rollout efficiency during the training of a student policy. Our approach builds on the DAgger algorithm by employing a greedy strategy, that selects isolated states from a student trajectory. These states are used to generate expert trajectory samples before supervised learning is performed and the process is repeated. The effectiveness of the Greedy-DAgger algorithm is evaluated on two simulated robotic systems: a cart pole and a quadcopter. For these environments, Greedy-DAgger was shown to be up to ten times more rollout efficient than conventional DAgger. The introduced improvements enable expert-level quadcopter control to be achieved within 8 seconds of wall time. The Crazyflie quadcopter platform was then utilised to validate the simulation results and demonstrate the potential for real-world training with Greedy-DAgger on a constrained platform, leveraging access to a remote GPU-accelerated server.