Learning Representation with Q-irrelevance Abstraction for Reinforcement Learning

In order to improve the performance of deep reinforcement learning (DRL) algorithm in high-dimensional observation environments, we propose a new auxiliary task to learn representations to aggregate task-relevant information of observations. Inspired by Q-irrelevance abstraction, our auxiliary task trains a deep Q-network (DQN) to predict the true Q value distribution over all discrete actions. Then we use the output of DQN to train the encoder to discriminate states with different Q values. The encoder is used as the representation of proximal policy optimization (PPO). The resulting algorithm is called as Q-irrelevance Abstraction for Reinforcement Learning (QIARL). After training, the encoder can aggregate states with similar Q value distributions together for any policy and any action. Thus the encoder can encode the important information that is relevant to reinforcement learning task. We test QIARL in four Procgen environments compare with PPO, A2C and Rainbow. The experimental results show QIARL outperforms the other three algorithms.