Feature-Option-Action: A domain adaption transfer reinforcement learning framework

Abstract

Transfer reinforcement learning (TRL) algorithms have achieved success on alleviating the resource-consumption and sample-insufficiency problem in reinforcement learning (RL). Existing works of cross-domain TRL mainly focus on designing a mapping between the state-action space of source and target domains. We, however, propose a novel TRL framework, Feature-Option-Action (FOA), with novel neural network architecture in this work, to avoid the design of explicit mapping functions between source and target domain. FOA learner is normally trained in the source domain, and the parameters of the option components in the neural network would then be used to initialize the learners in target domain. Empirical evidences have shown that our technique could significantly improve the performance of learners in target domains. Meanwhile, we train FOA models with the model updating methods (in our works, we call it step-update) used in Option-Critic, and illustrate that this method can improve the exploration ability of FOA models by increasing the diversity of options. We also compare step-update with other model updating methods, and the results show that step-update method performs better for FOA model to make transfer training faster and smoother.