A Dual-Aligned SAC Approach With Adaptive Risk Threshold for Safe Motion Planning of Autonomous Driving

Abstract

Safe reinforcement learning (RL) algorithms have attracted significant attention due to safety concerns arising from their deployment in real-world environments. Balancing both safety and driving smoothness remains a critical challenge for autonomous vehicles (AVs). A novel risk-aware RL framework built upon Soft Actor-Critic (SAC) is proposed in this paper. Our primary contribution is a decomposition of optimization objectives: we introduce a dedicated Temporal Convolutional Network (TCN)-based Risk Critic designed to encapsulate heterogeneous risk factors. This network simultaneously models long-term latent collision risks and short-term transient risks, outputting a risk estimate incorporated into the policy optimization via Lagrangian relaxation. Concurrently, the Value Critic focuses solely on maximizing reward based on driving smoothness and destination features. This explicit separation allows the Actor network to learn policies that inherently balance safety and comfort. Secondly, an Exponential Weighted Moving Average Forecast (EWAF)-based adaptive threshold model with a catch-up mechanism is presented to deal with time-varying scenarios. Thirdly, to mitigate safety hazards during the training phase, we employ expert-guided (EG) replay buffer initialization whose samples generated by a robust control-theory-based controller to warm-start the RL training process, significantly reducing early-stage collisions. Extensive simulations conducted in the CARLA simulator demonstrate the effectiveness of our approach, showcasing substantial improvement in comprehensive performance. The source code for our algorithm is available at https://github.com/hover03/Dual-SAC-2AD