Dynamic Option Policy Enabled Hierarchical Deep Reinforcement Learning Model for Autonomous Overtaking Maneuver

Abstract

Driving an Autonomous Vehicle (AV) in dynamic traffic is a critical task, as the overtaking maneuver being considered one of the most complex due to involvement of several sub-maneuvers. Recent advances in Deep Reinforcement Learning (DRL) have resulted in AVs exhibiting exceptional performance in addressing overtaking-related challenges. However, the intricate nature of the overtaking presents difficulties for a RL agent to proficiently handle all its sub-maneuvers that include left lane change, right lane change and straight drive. Furthermore, the dynamic traffic restricts the RL agents to execute the sub-maneuvers at critical checkpoints involved in overtaking. To address this, we propose an approach inspired by semi-Markov options, called Dynamic Option Policy enabled Hierarchical Deep Reinforcement Learning (DOP-HDRL). This innovative approach allows the selection of sub-maneuver agents using a single dynamic option policy, while employing individual DRL agents specifically trained for each sub-maneuver to perform tasks during overtaking in dynamic environments. By breaking down overtaking maneuvers into several sub-maneuvers and controlling them using a single policy, the DOP-HDRL approach reduces training time and computational load compared to classical DRL agents. Moreover, DOP-HDRL easily integrates basic traffic safety rules into overtaking maneuvers to offer more robust solutions. The DOP-HDRL approach is rigorously evaluated through multiple overtaking and non-overtaking scenarios inspired by the National Highway Traffic Safety Administration (NHTSA) pre-crash scenarios in the CARLA simulator. On an average, the DOP-HDRL approach shows 100% completion rate, 14% least collision rate, 25% optimal clearance distance, and 7% more average speed compared to the state-of-the-art methods.