Weakly supervised multi-modal imitation learning from incompletely labeled demonstrations

Abstract

Multi-modal imitation learning enables the agent to learn demonstrations of multiple modes at the same time. However, as expert demonstrations in practice tend to have incomplete labels for behavior modes, most methods are inefficient. To address this issue, an approach capable of imitation learning from incompletely labeled expert demonstrations, referred to as Weakly Supervised Multi-modal Imitation Learning (WSMIL), is proposed. WSMIL incorporates weakly supervised learning into multi-modal imitation learning by adding a behavior mode classifier to the adversarial network, thus forming adversaries among three players (generator, classifier and discriminator). Both labeled and unlabeled data are fully utilized in this adversarial process where fake state-action-label pairs generated by the generator and the classifier try to deceive the discriminator that tries to identify them and limited labeled expert demonstrations. Additionally, in order to ensure the data distribution of classifier and generator individually to converge to the expert’s real distribution, three extra losses are employed, where simulated annealing behavioral cloning is also added to the generator network to improve the generalization of policy. Experiments show that WSMIL accurately distinguishes modes with incomplete modal labels in demonstrations, learns close to the expert standard for each mode, and is more stable than other multi-modal methods.