LEASE: Offline Preference-Based Reinforcement Learning With High Sample Efficiency

Abstract

Offline preference-based reinforcement learning (PbRL) offers an effective approach to addressing the challenges of designing rewards and mitigating the high costs associated with online interaction. However, since labeling preference needs real-time human feedback, acquiring sufficient preference labels is challenging. To solve this, this article proposes an offline PbRL with a high sample efficiency (LEASE) algorithm, where a learned transition model is leveraged to generate unlabeled preference data. Considering the pretrained reward model may generate incorrect labels for unlabeled data, we design an uncertainty-aware mechanism to ensure the performance of the reward model, where only high-confidence and low-variance data are selected. Moreover, the generalization bound of the reward model is provided to analyze the factors influencing reward accuracy, and the policy learned by LEASEhas a theoretical improvement guarantee. The above developed theory is based on a state-action pair, which can be easily combined with other offline algorithms. The experimental results show that LEASEcan achieve comparable performance to the baseline under fewer preference data without online interaction.