Offline-to-online reinforcement learning with efficient unconstrained fine-tuning

Abstract

Offline reinforcement learning provides the capability to learn a policy only from pre-collected datasets, but its performance is often limited by the quality of the offline dataset and the coverage of the state-action space. Offline-to-online reinforcement learning is promising to address these limitations and achieve high sample efficiency by integrating the advantages of both offline and online learning paradigms. However, existing methods typically struggle to adapt to online learning and improve the performance of pre-trained policies due to the distributional shift and conservative training. To address these issues, we propose an efficient unconstrained fine-tuning framework that removes conservative constraints on the policy during fine-tuning, allowing thorough exploration of state-action pairs not covered by the offline data. This framework leverages three key techniques: dynamics representation learning, layer normalization, and increasing the update frequency of the value network to improve sample efficiency and mitigate value function estimation bias caused by the distributional shift. Dynamics representation learning accelerates fine-tuning by capturing meaningful features, layer normalization bounds $Q$-value to suppress catastrophic value function divergence, and increasing the update frequency of the value network enhances the sample efficiency and reduces value function estimation bias. Extensive experiments on the D4RL benchmark demonstrate that our algorithm outperforms state-of-the-art offline-to-online reinforcement learning algorithms across various tasks with minimal online interactions.