Enhancing Generalization of Offline RL in Data-Limited Settings With Heuristic Rules

Abstract

With the ability to learn from static datasets, OFFLINE reinforcement learning (RL) emerges as a compelling avenue for real-world applications. However, state-of-the-art offline RL algorithms perform suboptimally when confronted with limited data confined to specific regions within the state space. Performance degradation is attributed to the inability of offline RL algorithms to learn appropriate actions for rare or unseen observations. This article proposes a heuristic rule-based regularization technique and adaptively refines the initial knowledge from heuristics to considerably boost performance in limited data with partially omitted states. The key insight is that the regularization term mitigates erroneous actions for sparse samples and unobserved states covered by domain knowledge. Empirical evaluations on standard offline RL datasets demonstrate a substantial average performance increase compared to ensemble of domain knowledge and existing offline RL algorithms operating on limited data.