A cascaded strategy-based hierarchical reinforcement learning algorithm for lane change decision-making

Abstract

Hierarchical reinforcement learning (HRL) has demonstrated considerable promise in addressing complex driving tasks. However, existing HRL-based autonomous driving decision systems face challenges such as inefficient convergence, lack of interdependence among driving maneuver strategies (including throttle/brake control and steering adjustments), and inadequate risk assessment mechanisms, all of which impede the safety and stability of lane-changing decisions. This study proposes a novel HRL framework for continuous lane-changing decision planning. This framework establishes cascaded relationships between driving maneuvers strategies and integrates a comprehensive risk assessment mechanism to address these challenges. Initially, a hierarchical decision model is developed, where the high-level determines the lane-changing intent, while the low-level manages continuous and precise maneuvers. Subsequently, by integrating a Bayesian network, the cascading between throttle/brake openings and steering angles is achieved, optimizing the system's joint strategy distribution. Furthermore, a comprehensive risk assessment mechanism that evaluates the cooperation level of drivers and the severity of potential collisions is designed to encourage agents to adopt strategies that minimize risk. The effectiveness of the proposed decision-making framework has been validated through comparative experiments in mixed traffic scenarios simulated within the Car Learning to Act (CARLA) environment and corroborated with human driving data from the Next Generation Simulation (NGSIM) database.