Implicit policy constraint for offline reinforcement learning

Abstract

Offline reinforcement learning (RL) aims to learn policies entirely from passively collected datasets, making it a data-driven decision method. One of the main challenges in offline RL is the distribution shift problem, which causes the algorithm to visit out-of-distribution (OOD) samples. The distribution shift can be mitigated by constraining the divergence between the target policy and the behaviour policy. However, this method can overly constrain the target policy and impair the algorithm's performance, as it does not directly distinguish between in-distribution and OOD samples. In addition, it is difficult to learn and represent multi-modal behaviour policy when the datasets are collected by several different behaviour policies. To overcome these drawbacks, the authors address the distribution shift problem by implicit policy constraints with energy-based models (EBMs) rather than explicitly modelling the behaviour policy. The EBM is powerful for representing complex multi-modal distributions as well as the ability to distinguish in-distribution samples and OODs. Experimental results show that their method significantly outperforms the explicit policy constraint method and other baselines. In addition, the learnt energy model can be used to indicate OOD visits and alert the possible failure.