Reinforcement learning from human feedback with fast–slow updates for stable driving strategies

Reinforcement learning (RL) offers a promising framework for decision-making in automated vehicles (AVs), yet its practical application faces major obstacles, including sparse and delayed rewards, unstable policy optimization, and difficulties in designing effective reward functions. To address these challenges, we propose a human-in-the-loop RL framework with a fast–slow update architecture that enables reward-free training while maintaining policy stability. The fast update relies on direct feedback during human takeovers, using binary preference signals to guide the agent in early training. The slow update introduces a similarity constraint by comparing the agent’s actions to those of an auxiliary expert network trained exclusively on human-labeled transitions. This dual-update strategy allows the agent to benefit from efficient exploration while remaining anchored to human-aligned behavior. The method optimizes a combined objective consisting of temporal difference loss, proxy value loss from human preferences, and a similarity loss. Experiments conducted in the CARLA simulator demonstrate that this approach achieves lower takeover rates, faster convergence, and improved driving stability compared to standard RL methods. These results highlight the effectiveness of structured human feedback in reducing training burden and enhancing real-world readiness of autonomous driving policies. The code is available at: https://github.com/BELIV-ASU/aPVP0.9.10.1.git.