Safe Reinforcement Learning for Autonomous Driving by Using Disturbance-Observer-Based Control Barrier Functions

Abstract

Recently, reinforcement learning (RL) has been increasingly used in autonomous driving (AD) navigation control systems. However, most RL-based AD navigation control systems remain in the simulation stage. Its practical application is limited due to growing safety concerns. The safety of these algorithms remains uncertain when confronted with real-world disturbances and vehicle model uncertainties. To enhance the safety of RL, we propose a disturbance observer (DOB) based safe soft actor-critic (SAC) algorithm that combines the SAC algorithm with a safety constraints filter composed of DOB and control barrier function (CBF). When the SAC agent's action output is unsafe, the safety constraints filter will alter it. We employ a DOB to accurately estimate the difference between the nominal model of the vehicle and the actual model, i.e., the lumped disturbances. Then, a more accurate vehicle model can be obtained. To ensure the safety of DOB-SAC under complex and dynamically changing environmental conditions, a further predictive safety constraint is defined based on model predictive control (MPC) ideas. The safe action will be rendered using safety-critical optimal control according to the DOB compensated vehicle model, CBF, and the predictive safety constraints. We discuss the SAC architecture and training details, and investigate the effectiveness of CBF in modeling safety constraints. Joint simulations are conducted in scenarios with static obstacles and intersection scenes with dynamic obstacles.