Enhancing Safety in Model-Based Reinforcement Learning With High-Order Control Barrier Functions

Due to the risk of taking unsafe actions in unknown environment dynamics, reinforcement learning (RL) algorithms with built-in safety guarantees to prevent unexpected accidents has received increasing attention. Introducing the control barrier function is a typical method for imposing safety constraints by constructing the forward invariant set, but this approach generally suffers from the conservativeness of the forward invariant set and difficulties in the training process. To overcome these challenges, this paper proposes a novel algorithm called model-based safe RL with high-order control barrier function (MBSRL-HOCBF). The concepts of generalized feasibility are introduced, including generalized feasible state and generalized feasible region, which can be applied to the modified HOCBF conditions during training, thus reducing the conservativeness of the forward invariant set of HOCBF while ensuring both safety and algorithm performance. Additionally, the safety indicator that explicitly identifies safe states without requiring knowing specific safety criteria is incorporated, and integrated into the common environment model. The integration combines the advantages of traditional model-based RL, including using model-generated data to speed up algorithm training, with the ability to identify the generalized feasibility of each state. Simulation results demonstrate that MBSRL-HOCBF not only achieves high returns but also guarantees safety across multiple control tasks.