Diversity-Driven Contrastive Value Ensembles with Categorical Constraints for Goal-Conditioned Robotic Control

Abstract

Dear Editor, This letter presents a contrastive reinforcement learning (Contrastive RL)-based framework, addressing challenging goal-conditioned problems in robotic control. While Contrastive RL offers promise in learning state-action-goal relationships, it suffers from a critical limitation: Insufficient discriminability between positive and negative samples attributed to inefficient value exploration and model overfitting. To overcome these challenges, the proposed algorithm extends Contrastive RL by leveraging an ensemble of critic networks to model state-action-goal alignment, alleviating the overfitting problem. Furthermore, the architecture introduces a dual component loss function: 1) A diversity-driven term to mitigate exploration redundancy in value estimation; and 2) A categorical-guidance constraint to ensure the discriminability capacity across contrasting pairs. We term this integrated framework diversity-driven contrastive value ensembles with categorical constraints (DiCE-CC). Experimental validation across three robotic manipulation scenarios demonstrates the effectiveness of the proposed algorithm in solving complex goal-conditioned control problems.